Proposal: Less Error-Prone Loop Variable Scoping

David Chase
Russ Cox
May 2023

Discussion at https://go.dev/issue/60078.
Pre-proposal discussion at https://go.dev/issue/56010.

§ Abstract

Last fall, we had a GitHub discussion at #56010 about changing for loop variables declared with := from one-instance-per-loop to one-instance-per-iteration. Based on that discussion and further work on understanding the implications of the change, we propose that we make the change in an appropriate future Go version, perhaps Go 1.22 if the stars align, and otherwise a later version.

§ Background

This proposal is about changing for loop variable scoping semantics, so that loop variables are per-iteration instead of per-loop. This would eliminate accidental sharing of variables between different iterations, which happens far more than intentional sharing does. The proposal would fix #20733.

Briefly, the problem is that loops like this one don’t do what they look like they do:

var ids []*int
for i := 0; i < 10; i++ {
	ids = append(ids, &i)
}

That is, this code has a bug. After this loop executes, ids contains 10 identical pointers, each pointing at the value 10, instead of 10 distinct pointers to 0 through 9. This happens because the item variable is per-loop, not per-iteration: &i is the same on every iteration, and item is overwritten on each iteration. The usual fix is to write this instead:

var ids []*int
for i := 0; i < 10; i++ {
	i := i
	ids = append(ids, &i)
}

This bug also often happens in code with closures that capture the address of item implicitly, like:

var prints []func()
for _, v := range []int{1, 2, 3} {
	prints = append(prints, func() { fmt.Println(v) })
}
for _, print := range prints {
	print()
}

This code prints 3, 3, 3, because all the closures print the same v, and at the end of the loop, v is set to 3. Note that there is no explicit &v to signal a potential problem. Again the fix is the same: add v := v.

The same bug exists in this version of the program, with the same fix:

var prints []func()
for i := 1; i <= 3; i++ {
	prints = append(prints, func() { fmt.Println(i) })
}
for _, print := range prints {
	print()
}

Another common situation where this bug arises is in subtests using t.Parallel:

func TestAllEvenBuggy(t *testing.T) {
	testCases := []int{1, 2, 4, 6}
	for _, v := range testCases {
		t.Run("sub", func(t *testing.T) {
			t.Parallel()
			if v&1 != 0 {
				t.Fatal("odd v", v)
			}
		})
	}
}

This test passes, because each all four subtests check that 6 (the final test case) is even.

Goroutines are also often involved in this kind of bug, although as these examples show, they need not be. See also the Go FAQ entry.

Russ talked at Gophercon once about how we need agreement about the existence of a problem before we move on to solutions. When we examined this issue in the run up to Go 1, it did not seem like enough of a problem. The general consensus was that it was annoying but not worth changing. Since then, we suspect every Go programmer in the world has made this mistake in one program or another.

We have talked for a long time about redefining these semantics, to make loop variables per-iteration instead of per-loop. That is, the change would effectively be to add an implicit “x := x” at the start of every loop body for each iteration variable x, just like people do manually today. Making this change would remove the bugs from the programs above.

This proposal does exactly that. Using the go version lines in go.mod files, it only applies the new semantics to new programs, so that existing programs are guaranteed to continue to execute exactly as before.

Before writing this proposal, we collected feedback in a GitHub Discussion in October 2022, #56010. The vast majority of the feedback was positive, although a couple people did say they see no problem with the current semantics and discourage a change. Here are a few representative quotes from that discussion:

One thing to notice in this discussion is that even after having this problem explained multiple times by different people multiple developers were still having trouble understanding exactly what caused it and how to avoid it, and even the people that understood the problem in one context often failed to notice it would affect other types of for loops.

So we could argue that the current semantics make the learning curve for Go steeper.

PS: I have also had problems with this multiple times, once in production, thus, I am very in favor of this change even considering the breaking aspect of it.

— @VinGarcia

This exactly matches my experience. It's relatively easy to understand the first example (taking the same address each time), but somewhat trickier to understand in the closure/goroutine case. And even when you do understand it, one forgets (apparently even Russ forgets!). In addition, issues with this often don't show up right away, and then when debugging an issue, I find it always takes a while to realize that it's "that old loop variable issue again".

— @benhoyt

Go's unusual loop semantics are a consistent source of problems and bugs in the real world. I've been a professional go developer for roughly six years, and I still get bit by this bug, and it's a consistent stumbling block for newer Go programmers. I would strongly encourage this change.

— @efronlicht

I really do not see this as a useful change. These changes always have the best intentions, but the reality is that the language works just fine now. This well intended change slowly creep in over time, until you wind up with the C++ language yet again. If someone can't understand a relatively simple design decision like this one, they are not going to understand how to properly use channels and other language features of Go.

Burying a change to the semantics of the language in go.mod is absolutely bonkers.

— @hydrogen18

Overall, the discussion included 72 participants and 291 total comments and replies. As a rough measure of user sentiment, the discussion post received 671 thumbs up, 115 party, and 198 heart emoji reactions, and not a single thumbs down reaction.

Russ also presented the idea of making this change at GopherCon 2022, shortly after the discussion, and then again at Google Open Source Live's Go Day 2022. Feedback from both talks was entirely positive: not a single person suggested that we should not make this change.

§ Proposal

We propose to change for loop scoping in a future version of Go to be per-iteration instead of per-loop. For the purposes of this document, we are calling that version Go 1.30, but the change would land in whatever version of Go it is ready for. The earliest version of Go that could include the change would be Go 1.22.

This change includes four major parts: (1) the language specification, (2) module-based and file-based language version selection, (3) tooling to help users in the transition, (4) updates to other parts of the Go ecosystem. The implementation of these parts spans the compiler, the go command, the go vet command, and other tools.

Language Specification

In https://go.dev/ref/spec#For_clause, the text currently reads:

The init statement may be a short variable declaration, but the post statement must not. Variables declared by the init statement are re-used in each iteration.

This would be replaced with:

The init statement may be a short variable declaration (:=), but the post statement must not. Each iteration has its own separate declared variable (or variables). The variable used by the first iteration is declared by the init statement. The variable used by each subsequent iteration is declared implicitly before executing the post statement and initialized to the value of the previous iteration's variable at that moment.

var prints []func()
for i := 0; i < 3; i++ {
    prints = append(prints, func() { println(i) })
}
for _, p := range prints {
    p()
}

// Output:
// 0
// 1
// 2

Prior to Go 1.30, iterations shared one set of variables instead of having their own separate variables.

(Remember that in this document, we are using Go 1.30 as the placeholder for the release that will ship the new semantics.)

For precision in this proposal, the spec example would compile to a form semantically equivalent to this Go program:

{
	i_outer := 0
	first := true
	for {
		i := i_outer
		if first {
			first = false
		} else {
			i++
		}
		if !(i < 3) {
			break
		}
		prints = append(prints, func() { println(i) })
		i_outer = i
	}
}

Of course, a compiler can write the code less awkwardly, since it need not limit the translation output to valid Go source code. In particular a compiler is likely to have the concept of the current memory location associated with i and be able to update it just before the post statement.

In https://go.dev/ref/spec#For_range, the text currently reads:

The iteration variables may be declared by the "range" clause using a form of short variable declaration (:=). In this case their types are set to the types of the respective iteration values and their scope is the block of the "for" statement; they are re-used in each iteration. If the iteration variables are declared outside the "for" statement, after execution their values will be those of the last iteration.

This would be replaced with:

The iteration variables may be declared by the "range" clause using a form of short variable declaration (:=). In this case their types are set to the types of the respective iteration values and their scope is the block of the "for" statement; each iteration has its own separate variables. If the iteration variables are declared outside the "for" statement, after execution their values will be those of the last iteration.

var prints []func()
for _, s := range []string{"a", "b", "c"} {
    prints = append(prints, func() { println(s) })
}
for _, p := range prints {
    p()
}

// Output:
// a
// b
// c

Prior to Go 1.30, iterations shared one set of variables instead of having their own separate variables.

For precision in this proposal, the spec example would compile to a form semantically equivalent to this Go program:

{
	var s_outer string
	for _, s_outer = range []string{"a", "b", "c"} {
		s := s_outer
		prints = append(prints, func() { println(s) })
	}
}

Note that in both 3-clause and range forms, this proposal is a complete no-op for loops with no := in the loop header and loops with no variable capture in the loop body. In particular, a loop like the following example, modifying the loop variable during the loop body, continues to execute as it always has:

for i := 0;; i++ {
	if i >= len(s) || s[i] == '"' {
		return s[:i]
	}
	if s[i] == '\\' { // skip escaped char, potentially a quote
		i++
	}
}

Language Version Selection

The change in language specification will fix far more programs than it breaks, but it may break a very small number of programs. To make the potential breakage completely user controlled, the rollout would decide whether to use the new semantics based on the go line in each package’s go.mod file. This is the same line already used for enabling language features; for example, to use generics in a package, the go.mod must say go 1.18 or later. As a special case, for this proposal, we would use the go line for changing semantics instead of for adding or removing a feature.

Modules that say go 1.30 or later would have for loops using per-iteration variables, while modules declaring earlier versions have for loops with per-loop variables:

<img width="734" alt="Code in modules that say go 1.30 gets per-iteration variable semantics; code in modules that say earlier Go versions gets per-loop semantics." src="https://user-images.githubusercontent.com/104030/193599987-19d8f564-cb40-488e-beaa-5093a4823ee0.png">

This mechanism would allow the change to be deployed gradually in a given code base. Each module can update to the new semantics independently, avoiding a bifurcation of the ecosystem.

The forward compatibility work in #57001, which will land in Go 1.21, ensures that Go 1.21 and later will not attempt to compile code marked go 1.30. Even if this change lands in Go 1.22, the previous (and only other supported) Go release would be Go 1.21, which would understand not to compile go 1.22 code. So code opting in to the new loop semantics would never miscompile in older Go releases, because it would not compile at all. If the changes were to be slated for Go 1.22, it might make sense to issue a Go 1.20 point release making its go command understand not to compile go 1.22 code. Strictly speaking, that point release is unnecessary, because if Go 1.22 has been released, Go 1.20 is unsupported and we don't need to worry about its behavior. But in practice people do use older Go releases for longer than they are supported, and if they keep up with point releases we can help them avoid this potential problem.

The forward compatibility work also allows a per-file language version selection using //go:build directives. Specifically, if a file in a go 1.29 module says //go:build go1.30, it gets the Go 1.30 language semantics, and similarly if a file in a go 1.30 module says //go:build go1.29, it gets the Go 1.29 language semantics. This general rule would apply to loop semantics as well, so the files in a module could be converted one at a time in a per-file gradual code repair if necessary.

Vendoring of other Go modules already records the Go version listed in each vendored module's go.mod file, to implement the general language version selection rule. That existing support would also ensure that old vendored modules keep their old loop semantics even in a newer overall module.

Transition Support Tooling

We expect that this change will fix far more programs than it breaks, but it will break some programs. The most common programs that break are buggy tests (see the “fixes buggy code” section below for details). Users who observe a difference in their programs need support to pinpoint the change. We plan to provide two kinds of support tooling, one static and one dynamic.

The static support tooling is a compiler flag that reports every loop that is compiling differently due to the new semantics. Our prototype implementation does a very good job of filtering out loops that are provably unaffected by the change in semantics, so in a typical program very few loops are reported. The new compiler flag, -d=loopvar=2, can be invoked by adding an option to the go build or go test command line: either -gcflags=-d=loopvar=2 for reports about the current package only, or -gcflags=all=-d=loopvar=2 for reports about all packages.

The dynamic support tooling is a new program called bisect that, with help from the compiler, runs a test repeatedly with different sets of loops opted in to the new semantics. By using a binary search-like algorithm, bisect can pinpoint the exact loop or loops that, when converted to the new semantics, cause a test failure. Once you have a test that fails with the new semantics but passes with the old semantics, you run:

bisect -compile=loopvar go test

We have used this dynamic tooling in a conversion of Google's internal monorepo to the new loop semantics. The rate of test failure caused by the change was about 1 in 8,000. Many of these tests took a long time to run and contained complex code that we were unfamiliar with. The bisect tool is especially important in this situation: it runs the search while you are at lunch or doing other work, and when you return it has printed the source file and line number of the loop that causes the test failure when compiled with the new semantics. At that point, it is trivial to rewrite the loop to pre-declare a per-loop variable and no longer use :=, preserving the old meaning even in the new semantics. We also found that code owners were far more likely to see the actual problem when we could point to the specific line of code. As noted in the “fixes buggy code” section below, all but one of the test failures turned out to be a buggy test.

Updates to the Go Ecosystem

Other parts of the Go ecosystem will need to be updated to understand the new loop semantics.

Vet and the golang.org/x/tools/go/analysis framework are being updated as part of #57001 to have access to the per-file language version information. Analyses like the vet loopclosure check will need to tailor their diagnostics based on the language version: in files using the new semantics, there won't be := loop variable problems anymore.

Other analyzers, like staticcheck and golangci-lint, may need updates as well. We will notify the authors of those tools and work with them to make sure they have the information they need.

§ Rationale and Compatibility

In most Go design documents, Rationale and Compatibility are two distinct sections. For this proposal, considerations of compatibility are so fundamental that it makes sense to address them as part of the rationale. To be completely clear: this is a breaking change to Go. However, the specifics of how we plan to roll out the change follow the spirit of the compatibility guidelines if not the “letter of the law.”

In the Go 2 transitions document we gave the general rule that language redefinitions like what we just described are not permitted, giving this very proposal as an example of something that violates the general rule. We still believe that that is the right general rule, but we have come to also believe that the for loop variable case is strong enough to motivate a one-time exception to that rule. Loop variables being per-loop instead of per-iteration is the only design decision we know of in Go that makes programs incorrect more often than it makes them correct. Since it is the only such design decision, we do not see any plausible candidates for additional exceptions.

The rest of this section presents the rationale and compatibility considerations.

A decade of experience shows the cost of the current semantics

Russ talked at Gophercon once about how we need agreement about the existence of a problem before we move on to solutions. When we examined this issue in the run up to Go 1, it did not seem like enough of a problem. The general consensus was that it was annoying but not worth changing.

Since then, we suspect every Go programmer in the world has made this mistake in one program or another. Russ certainly has done it repeatedly over the past decade, despite being the one who argued for the current semantics and then implemented them. (Apologies!)

The current cures for this problem are worse than the disease.

We ran a program to process the git logs of the top 14k modules, from about 12k git repos and looked for commits with diff hunks that were entirely “x := x” lines being added. We found about