Mastering Go Concurrency with a Real-World Case Study

Hey there, dev! Who hasn't faced a batch process that grinds the application to a halt or takes hours to run? Whether it's processing payments, generating reports, or, in our case today, validating and registering thousands of bank slips (boletos, a common payment method in Brazil).

In this post, we'll take a real-world problem and turn it into a hands-on lesson on Go's superpower: concurrency. We'll go from slow, predictable code to a solution that flies, using the tools the language provides. Ready to get your hands dirty with goroutines, channels, and sync.WaitGroup? Let's dive in!

1. The Scenario: Processing Bank Slips in a Single File (The Slow Way)

Let's start with the basics. We have a massive list of bank slips to process. Each one needs to be validated, perhaps by consulting an external service, and then saved to the database.

The initial code, likely written in a hurry by someone unfamiliar with concurrency, would be a simple for loop.

func ProcessBankSlipsSequentially(slips []BankSlip) {
    for _, slip := range slips {
        validateSlip(slip)
        saveToDatabase(slip)
        // Imagine each step takes ~100ms
    }
}

The problem? If we have 10,000 slips and each takes 100ms, we're talking about... well, way too long! We can do better.

2. First Attempt: Unleashing Goroutines (Controlled Chaos?)

The first idea that comes to mind is, "I'll just throw everything into a goroutine!"

func ProcessBankSlipsWithGoroutines(slips []BankSlip) {
    for _, slip := range slips {
        go func(s BankSlip) {
            validateSlip(s)
            saveToDatabase(s)
        }(slip)
    }
    // Now what? How do we know when it's finished?
}

This is faster, but it creates two problems:

1. The main function exits immediately without waiting for the goroutines to finish.
2. If we have 1 million slips, are we going to open 1 million database connections at once? That's a recipe for disaster.

We need control. This is where sync.WaitGroup comes in.

3. Getting Organized: Using sync.WaitGroup to Wait for the Crew

A WaitGroup is like a bouncer at a club. It keeps track of how many guests (goroutines) are inside and only turns off the lights when everyone has left.

func ProcessWithWaitGroup(slips []BankSlip) {
    var wg sync.WaitGroup

    for _, slip := range slips {
        wg.Add(1) // "Another guest is coming in."
        go func(s BankSlip) {
            defer wg.Done() // "Hey bouncer, I'm heading out."
            validateSlip(s)
            saveToDatabase(s)
        }(slip)
    }

    wg.Wait() // "Bouncer, wait for everyone to leave before closing up."
    fmt.Println("All bank slips processed!")
}

Nice! Now we wait. But what about controlling how many goroutines run at the same time? And how do we handle errors? If one slip fails to process, how do we find out?

4. The Complete Solution: Worker Pools with Channels

This is the icing on the cake and the pattern that solves the problem elegantly and robustly. The idea is to create a fixed number of "workers" that wait for tasks.

• jobs channel: A channel to send the bank slips that need processing.
• results channel: A channel to receive the outcomes (success or error).
• Worker Functions: The goroutines that read from the jobs channel, process the slip, and send the result to the results channel.

func ProcessWithWorkerPool(slips []BankSlip) {
    numWorkers := 10 // We control the number of concurrent routines
    jobs := make(chan BankSlip, len(slips))
    results := make(chan error, len(slips))
    var wg sync.WaitGroup

    // 1. Start the workers
    for w := 1; w <= numWorkers; w++ {
        wg.Add(1)
        go worker(w, jobs, results, &wg)
    }

    // 2. Send the jobs to the channel
    for _, slip := range slips {
        jobs <- slip
    }
    close(jobs) // Close the channel to signal there are no more jobs

    // 3. Wait for all workers to finish
    wg.Wait()
    close(results)

    // 4. Collect and process the results/errors
    for err := range results {
        if err != nil {
            log.Printf("Error during processing: %v", err)
        }
    }
}

func worker(id int, jobs <-chan BankSlip, results chan<- error, wg *sync.WaitGroup) {
    defer wg.Done()
    for j := range jobs {
        fmt.Printf("Worker %d processing slip %s\n", id, j.ID)
        // Simulate the work
        err := validateAndProcess(j)
        results <- err
    }
}

Solution Analysis: With this pattern, we have:

• Controlled Concurrency: We avoid overwhelming our resources (database, external APIs).
• Error Handling: We capture all errors in a centralized way.
• Decoupling: The logic for distributing work is separate from the execution logic.
• Performance: We efficiently use the power of multi-core processors.

Conclusion: Concurrency Isn't Rocket Science

As we've seen, moving from sequential to parallel code in Go is a journey of evolution. We started with a for loop, moved through the "excitement" of unleashing loose goroutines, and arrived at a robust and scalable pattern with worker pools.

Understanding these patterns (WaitGroup, channels, worker pools) is what allows you to build high-performance systems in Go that don't just work—they work fast and safely.

The next time you hit a bottleneck, remember: Go gives you the tools, and now you know how to use them to turn a crawl into a sprint!

---

📣 Let's Keep the Conversation Going!

Have you used a similar pattern in a project? What has been your biggest challenge with concurrency in Go? Drop a comment below and let's chat!

👉 Follow me on Hashnode (or wherever you're reading this) and let's continue this chat on LinkedIn!