Proposal: Adding tool dependencies to go.mod

Author(s): Conrad Irwin

Last updated: 2024-07-18

Discussion at https://golang.org/issue/48429.

§ Abstract

Authors of Go modules frequently use tools that are written in Go and distributed as Go modules. Although Go has good support for managing dependencies imported into their programs, the support for tools used during development is comparatively weak.

To make it easier for Go developers to use tools written in Go go.mod should gain a new directive that lets module authors define which tools are needed.

§ Background

Programs written in Go are often developed using tooling written in Go. There are several examples of these, for example: golang.org/x/tools/cmd/stringer or github.com/kyleconroy/sqlc.

It is desirable that all collaborators on a given project use the same version of tools to avoid the output changing slightly on different people’s machines. This comes up particularly with tools like linters (where changes over time may change whether or not the code is considered acceptable) and code generation (where the generated code must be assumed to match the version of the library that is linked).

The currently recommended approach to this is to create a file called tools.go that imports the package containing the tools to make the dependencies visible to the module graph. To hide this file from the compiler, it is necessary to exclude it from builds by adding an unused build tag such as //go:build tools. To hide this file from other packages that depend on your module, it must be put in its own package inside your module.

This approach is quite fiddly to use correctly, and still has a few downsides:

1. It is hard to type go run golang.org/x/tools/cmd/stringer, and so projects often contain wrapper scripts.
2. go run relinks tools every time they are run, which may be noticeably slow.

People work around this by either globally installing tools, which may lead to version skew, or by installing and using third party tooling (like accio) to manage their tools instead.

§ Proposal

New syntax in go.mod

go.mod gains a new directive: tool path/to/package.

This acts exactly as though you had a correctly set up tools.go that contains import "path/to/package".

As with other directives, multiple tool directives can be factored into a block:

go 1.24

tool (
    golang.org/x/tools/cmd/stringer
    ./cmd/migrate
)

Is equivalent to:

go 1.24

tool golang.org/x/tools/cmd/stringer
tool ./cmd/migrate

To allow automated changes go mod edit will gain two new parameters: -tool path/to/package and -droptool path/to/package that add and remove tool directives respectively.

New behavior for go get

To allow users to easily add new tools, go get will gain a new parameter: -tool.

When go get is run with the -tool parameter, then it will download the specified package and add it to the module graph as it does today. Additionally it will add a new tool directive to the current module’s go.mod.

If you combine the -tool flag with the @none version, then it will also remove the tool directive from your go.mod.

New behavior for go tool

When go tool is run in module mode with an argument that does not match a go builtin tool, it will search the current go.mod for a tool directive that matches the last path segment and compile and run that tool similarly to go run.

For example if your go.mod contains:

tool golang.org/x/tools/cmd/stringer
require golang.org/x/tools v0.9.0

Then go tool stringer will act similarly to go run golang.org/x/tools/cmd/[email protected], and go tool with no arguments will also list stringer as a known tool.

In the case that two tool directives end in the same path segment, go tool X will error. In the case that a tool directive ends in a path segment that corresponds to a builtin Go tool, the builtin tool will be run. In both cases you can use go tool path/to/package to specify what you want unconditionally.

The only difference from go run is that go tool will cache the built binary in $GOCACHE/tool/<current-module-path>/<TOOLNAME>. Subsequent runs of go tool X will then check that the built binary is up to date, and only rebuild it if necessary to speed up re-using tools.

When the Go cache is trimmed, any tools that haven't been used in the last five days will be deleted. Five days was chosen arbitrarily as it matches the expiry used for existing artifacts. Running go clean -cache will also remove all of these binaries.

A tools metapackage

We will add a new metapackage tools that contains all of the tools in the current modules go.mod.

This would allow for the following operations:

# Install all tools in GOBIN
go install tools

# Build and cache tools so `go tool X` is fast:
go build tools

# Update all tools to their latest versions.
go get tools

# Install all tools in the bin/ directory
go build -o bin/ tools

§ Rationale

This proposal tries to improve the workflow of Go developers who use tools packaged as Go modules while developing Go modules. It deliberately does not try and solve the problem of versioning arbitrary binaries: anything not distributed as a Go module is out of scope.

There were a few choices that needed to be made, explained below:

1. We need a mechanism to specify an exact version of a tool to use in a given module. Re-using the require directives in go.mod allows us to do this without introducing a separate dependency tree or resolution path. This also means that you can use require and replace directives to control the dependencies used when building your tools.
2. We need a way to easily run a tool at the correct version. Adding go tool X allows Go to handle versioning for you, unlike installing binaries to your path.
3. We need a way to improve the speed of running tools (compared to go run today) as tools are likely to be reused. Reusing the existing Go cache and expiry allows us to do this in a best-effort way without filling up the users’ disk if they develop many modules with a large number of tools.
4. go tool X always defaults to the tool that ships with the Go distribution in case of conflict, so that it always acts as you expect.

§ Compatibility

There’s no language change, however we are changing the syntax of go.mod. This should be ok, as the file-format was designed with forward compatibility in mind.

If Go adds tools to the distribution in the future that conflict with tools added to projects’ go.mod files, this may cause compatibility issues in the future. I think this is likely not a big problem in practice, as I expect new tools to be rare. Experience from using $PATH as a shared namespace for executables suggests that name conflicts in binaries can be easily avoided in practice.

§ Implementation

I plan to work on this for go1.24.

§ Open questions

How should this work with Workspaces?

This should probably not do anything special with workspaces. Because tools must be present in the require directives of a module, there is no easy way to make them work at a workspace level instead of a module level.

It might be possible to try and union all tools in all modules in the workspace, but I suggest we defer this to future work if it’s desired.