Go for Backend and CLI DevelopmentLesson 21 / 26

Concurrency and Shipping

Goroutines, Races, and Mutexes

ReadingPreview

আপনি একটি free preview lesson দেখছেন।

Goal

By the end of this lesson, you can run work at the same time with goroutines, wait for it with sync.WaitGroup, and find bugs with Go's race detector. You use these skills to make Tasker's Store safe for many goroutines at once, which the API server in Part 2 needs.

What you will learn

  • What a goroutine is and how to start one with go
  • How to wait for goroutines with sync.WaitGroup
  • What a data race is, and why it causes random, hard-to-find bugs
  • How to find races with go test -race
  • How to protect shared data with sync.Mutex
  • What a deadlock is, and a simple rule that prevents the most common one

Why this matters

Until now, Tasker did one thing at a time. That is fine for a command-line tool. But in Part 2, Tasker becomes a web server, and a web server handles many requests at the same time. Go runs each request in its own goroutine. If two requests add a task at the same moment, they both use the same store. Without protection, the program can crash or lose data.

Concurrency is one of Go's greatest strengths, and one of the main reasons companies choose Go for backends. This module teaches it step by step.

Step 1: Your first goroutines

Create cmd/play/main.go:

package main

import (
	"fmt"
	"sync"
	"time"
)

func work(name string, d time.Duration) {
	time.Sleep(d)
	fmt.Println(name, "finished after", d)
}

func main() {
	start := time.Now()
	work("A", 300*time.Millisecond)
	work("B", 200*time.Millisecond)
	work("C", 100*time.Millisecond)
	fmt.Println("one after another:", time.Since(start).Round(10*time.Millisecond))
}

work simulates a slow job, such as a network call, by sleeping. Run it with go run ./cmd/play:

A finished after 300ms
B finished after 200ms
C finished after 100ms
one after another: 600ms

The jobs run one after another, so the total time is 300 + 200 + 100 = 600 milliseconds.

Now add this to the end of main:

	start = time.Now()
	var wg sync.WaitGroup
	jobs := []struct {
		name string
		d    time.Duration
	}{
		{"A", 300 * time.Millisecond},
		{"B", 200 * time.Millisecond},
		{"C", 100 * time.Millisecond},
	}
	for _, job := range jobs {
		wg.Add(1)
		go func() {
			defer wg.Done()
			work(job.name, job.d)
		}()
	}
	wg.Wait()
	fmt.Println("at the same time:", time.Since(start).Round(10*time.Millisecond))

Output:

C finished after 100ms
B finished after 200ms
A finished after 300ms
at the same time: 300ms

All three jobs ran at the same time, so the total is the time of the slowest job, 300 milliseconds. The jobs also finished in a different order.

Step 2: How it works

go f() starts the function f in a new goroutine and continues immediately, without waiting. A goroutine is a function that runs at the same time as other code. Here, go starts a function literal, func() { ... }(). The () at the end calls it.

Goroutines are very cheap. A new goroutine starts with only a few kilobytes of memory, and the Go runtime spreads them across all CPU cores for you. A program can run hundreds of thousands of goroutines. Compare that with operating system threads, which are much heavier.

sync.WaitGroup waits for a group of goroutines to finish:

  • wg.Add(1) says "one more goroutine to wait for". Call it before go, not inside the goroutine. Otherwise Wait might run before the goroutine has a chance to call Add.
  • defer wg.Done() says "this goroutine is finished". With defer, it runs even if the function returns early.
  • wg.Wait() blocks until every Add has a matching Done.

Without Wait, main would end at once, and the program would stop before the goroutines finish. When main returns, all goroutines are stopped.

Loop variables. Since Go 1.22, each loop round has its own copy of job, so each goroutine sees the right job. In older Go code, you will often see job := job inside the loop. That was the fix for a famous bug, and you no longer need it.

Go 1.25 and newer have a shorter form: wg.Go(func() { work(job.name, job.d) }) does the Add, the go, and the Done in one call.

Step 3: A data race

Goroutines run at the same time, so they can use the same variable at the same time. Add this to main:

	counter := 0
	var wg2 sync.WaitGroup
	for range 1000 {
		wg2.Add(1)
		go func() {
			defer wg2.Done()
			counter++
		}()
	}
	wg2.Wait()
	fmt.Println("counter without a lock:", counter)

for range 1000 repeats the loop 1000 times (Go 1.22 and newer). You expect 1000. Run the program a few times:

counter without a lock: 971
counter without a lock: 967
counter without a lock: 916

The result is wrong, and it changes every time.

counter++ looks like one step, but the computer does three: read the value, add one, write it back. When two goroutines read the same value, say 500, at the same moment, both write 501. One increment is lost.

This is a data race: two goroutines use the same memory at the same time, and at least one of them writes. Data races are some of the worst bugs, because they depend on timing. A program can pass every test on your computer and fail on a busy server.

Step 4: The race detector

Go has a tool that finds data races for you. Add -race to go run or go test:

go run -race ./cmd/play

==================
WARNING: DATA RACE
Read at 0x00c0001820f8 by goroutine 15:
  main.main.func2()
      .../cmd/play/main.go:42 +0x84

Previous write at 0x00c0001820f8 by goroutine 16:
  main.main.func2()
      .../cmd/play/main.go:42 +0x96
...

The race detector watches every memory access while the program runs. It reports the exact line where two goroutines collide: line 42, counter++.

The detector only finds races in code that actually runs, so use it together with tests that run code concurrently. Make this a habit: always run go test -race ./... before you commit. In Lesson 6.4, your release workflow runs it automatically.

If -race does not work: the race detector needs cgo, Go's bridge to C code, on most systems. If you see an error about cgo or a missing C compiler, install gcc (on Windows, for example with MSYS2), or run the race tests in a Linux container or in CI.

Step 5: Protect shared data with a mutex

A mutex ("mutual exclusion") is a lock. Only one goroutine can hold it at a time. Add this to main:

	safe := 0
	var mu sync.Mutex
	var wg3 sync.WaitGroup
	for range 1000 {
		wg3.Add(1)
		go func() {
			defer wg3.Done()
			mu.Lock()
			safe++
			mu.Unlock()
		}()
	}
	wg3.Wait()
	fmt.Println("counter with a lock:", safe)

counter with a lock: 1000
  • mu.Lock() waits until the lock is free, then takes it.
  • mu.Unlock() releases it, so the next goroutine can take it.

The code between Lock and Unlock is a critical section. Only one goroutine runs it at a time, so safe++ is never interrupted. Run with -race again: the warnings for this part are gone.

The zero value of sync.Mutex is an unlocked mutex, ready to use. Never copy a mutex after you use it. go vet warns you if you do.

Step 6: Find the race in Tasker

The Store uses a map and a counter, nextID. In Part 2, many goroutines will call Add at once. Write a test that does exactly that. Create internal/task/concurrency_test.go:

package task_test

import (
	"fmt"
	"sync"
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
)

func TestStoreConcurrentAdd(t *testing.T) {
	store := newTestStore(t)
	const n = 50

	var wg sync.WaitGroup
	for i := range n {
		wg.Add(1)
		go func() {
			defer wg.Done()
			_, err := store.Add(fmt.Sprintf("Task %d", i), nil)
			assert.NoError(t, err)
		}()
	}
	wg.Wait()

	require.Equal(t, n, store.Len())
	ids := make(map[int]bool)
	for _, tk := range store.List() {
		ids[tk.ID] = true
	}
	assert.Len(t, ids, n, "every task must have a unique ID")
}

The test starts 50 goroutines that add tasks at the same time. Then it checks that there are 50 tasks with 50 different IDs.

Run it without the race detector a few times:

go test -count=1 -run Concurrent ./internal/task

Most runs crash the whole test program:

fatal error: concurrent map writes

Go maps are not safe for concurrent writes. The Go runtime notices and stops the program, because continuing could corrupt memory. Sometimes the test even passes by luck. Now run it with the race detector:

go test -race -run Concurrent ./internal/task

WARNING: DATA RACE
Read at 0x00c0000425d8 by goroutine 18:
  github.com/yourname/tasker/internal/task.(*Store).Add()
      .../internal/task/store.go:44 +0x104
...
Previous write at 0x00c0000425d8 by goroutine 10:
  github.com/yourname/tasker/internal/task.(*Store).Add()
      .../internal/task/store.go:51 +0x338

The race detector points to the reason: two Add calls read and write the store's fields at the same time.

Step 7: Make the store safe

Open internal/task/store.go. Add "sync" to the imports and a mutex to the struct:

// Store keeps tasks in memory and saves every change to its storage.
// It is safe to use from several goroutines at the same time.
type Store struct {
	mu      sync.Mutex // protects all fields below
	storage Storage
	tasks   map[int]Task
	nextID  int
}

By convention, the mutex is placed above the fields it protects, with a comment that says so. The doc comment tells users that the type is safe for concurrent use, an important promise.

Now lock the mutex at the start of every exported method. For Add:

func (s *Store) Add(title string, tags []string) (Task, error) {
	s.mu.Lock()
	defer s.mu.Unlock()

	if err := ValidateTitle(title); err != nil {
	...

defer s.mu.Unlock() releases the lock when the method returns, through any return statement. Lock and unlock sit next to each other, and you can never forget to unlock on an early return. This is the most common way to use a mutex in Go.

Step 8: Avoid a deadlock

Here is a trap. Complete calls s.Get, and save calls s.List. If those methods also lock the mutex, look at what happens:

  1. Complete locks s.mu.
  2. Complete calls Get.
  3. Get tries to lock s.mu, which is already locked, so it waits.
  4. Complete waits for Get to return. Nobody can ever continue.

This is a deadlock. Go's mutex is not reentrant: a goroutine that holds the lock cannot lock it again. If you try it, the test simply hangs. With a timeout, you see:

go test -timeout 3s -run TestStoreComplete ./internal/task

panic: test timed out after 3s
	running tests:
		TestStoreComplete (3s)

The solution is a simple rule: exported methods lock; unexported helpers assume the lock is already held. Split Get and List:

// Get returns the task with the given ID.
func (s *Store) Get(id int) (Task, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	return s.get(id)
}

// get returns the task with the given ID. The caller must hold s.mu.
func (s *Store) get(id int) (Task, error) {
	t, ok := s.tasks[id]
	if !ok {
		return Task{}, ErrNotFound
	}
	return t, nil
}

// List returns all tasks, sorted by ID.
func (s *Store) List() []Task {
	s.mu.Lock()
	defer s.mu.Unlock()
	return s.list()
}

// list returns all tasks, sorted by ID. The caller must hold s.mu.
func (s *Store) list() []Task {
	var list []Task
	for _, id := range slices.Sorted(maps.Keys(s.tasks)) {
		list = append(list, s.tasks[id])
	}
	return list
}

Then update the methods that already hold the lock, so they call the helpers:

func (s *Store) Complete(id int) (Task, error) {
	s.mu.Lock()
	defer s.mu.Unlock()

	t, err := s.get(id)
	...
}

func (s *Store) Len() int {
	s.mu.Lock()
	defer s.mu.Unlock()
	return len(s.tasks)
}

// save writes all tasks to the storage. The caller must hold s.mu.
func (s *Store) save() error {
	if err := s.storage.Save(s.list()); err != nil {
		return fmt.Errorf("save tasks: %w", err)
	}
	return nil
}

The comment "The caller must hold s.mu" is important. It tells every future reader the rule for that function.

Open does not need the lock. It builds a new store that no other goroutine can see yet.

Step 9: Run all tests with the race detector

go test -race -count=1 ./...

?   	github.com/yourname/tasker/cmd/tasker	[no test files]
ok  	github.com/yourname/tasker/internal/cli	1.048s
ok  	github.com/yourname/tasker/internal/storage	1.033s
ok  	github.com/yourname/tasker/internal/task	1.022s

No races. Tests run more slowly with -race, because the detector checks every memory access. That is why you use it in tests and CI, not in the program you ship.

Delete the cmd/play folder when you are done.

Why it works: share memory by locking, and keep the lock simple

Every piece of shared data needs a clear owner and a clear rule. In the store, the rule is: every access to tasks and nextID happens while holding s.mu. The exported methods enforce it, the helpers document it, and the race detector checks it.

The store holds the lock while it saves, so saves never overlap. This is simple and correct, but it means only one change happens at a time. For Tasker, that is fine. In Part 2, the database takes over this job, and it handles many writers efficiently.

In Lesson 6.2, you learn the other half of Go's concurrency model: channels, which let goroutines communicate instead of sharing memory.

Exercises

  1. Concurrent complete. Add a test that adds 20 tasks, then completes all of them from 20 goroutines at the same time. Run it with -race. Check that all tasks are done.
  2. Read and write together. Write a test where 10 goroutines call Add and 10 other goroutines call List at the same time. Remove the lock from List and run with -race. What does the detector report? Put the lock back.
  3. sync.RWMutex. Read the documentation of sync.RWMutex. It allows many readers at the same time but only one writer. Change the store to use it, with RLock in Get, List, and Len. When is this faster than a normal mutex?
  4. Measure it. In cmd/play, start 10 goroutines that each call work with 100 milliseconds. How long does the whole program take? Then try 10,000 goroutines. What happens to the time?
  5. Find the bug. What is wrong with this code, and how do you fix it? 
    for _, job := range jobs {
	go func() {
		wg.Add(1)
		defer wg.Done()
		work(job.name, job.d)
	}()
}
wg.Wait()

Checkpoint

You are ready for the next lesson if you can:

  • Start a goroutine and wait for a group of goroutines with sync.WaitGroup
  • Explain what a data race is and why counter++ is not safe
  • Find races with go test -race
  • Protect shared data with sync.Mutex and defer Unlock()
  • Explain why calling Get inside Complete caused a deadlock, and how the "unexported helper" rule prevents it