Pipelines

• Process streams, or batches of data.
• Pipline enable us to make an efficient use of I/O and multiple CPUs cores.
• Pipleine is a series of stages, connected by channels.
• Each stage is a represented by a goroutine.
• Stage - take data in, perform an operation on it, and send the data out.

Stages

• Separate the concerns of each stage
• Process individual stage concurrently.
• A stage could consume and return the same type. func square(in <-chan int) <-chan int{//...}
• This enables composability of pipeline. square(square(generator(2,3)))

Execrcises

https://go.dev/play/p/KPKOqa7rMKJ

package main

import "fmt"

// Build a Pipeline
// generator() -> square() -> print

// generator - convertes a list of integers to a channel
func generator(nums ...int) <-chan int {
    out := make(chan int)

    go func() {
        for _, n := range nums {
            out <- n
        }
        close(out)
    }()
    return out
}

// square - receive on inbound channel
// square the number
// output on outbound channel
func square(in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        for n := range in {
            out <- n * n
        }
        close(out)
    }()
    return out
}

func main() {
    // set up the pipeline
    for n := range square(square(generator(2, 3))) {
        fmt.Println(n)
    }

    // run the last stage of pipeline
    // receive the values from square stage
    // print each one, until channel is closed.

}