Proposal: extend code coverage testing to include applications

Author(s): Than McIntosh

Last updated: 2022-03-02

Discussion at https://golang.org/issue/51430

§ Abstract

This document contains a proposal for improving/revamping the system used in Go for code coverage testing.

§ Background

Current support for coverage testing

The Go toolchain currently includes support for collecting and reporting coverage data for Go unit tests; this facility is made available via the "go test -cover" and "go tool cover" commands.

The current workflow for collecting coverage data is baked into "go test" command; the assumption is that the source code of interest is a Go package or set of packages with associated tests.

To request coverage data for a package test run, a user can invoke the test(s) via:

  go test -coverprofile=<filename> [package target(s)]

This command will build the specified packages with coverage instrumentation, execute the package tests, and write an output file to "filename" with the coverage results of the run.

The resulting output file can be viewed/examined using commands such as

  go tool cover -func=<covdatafile>
  go tool cover -html=<covdatafile>

Under the hood, the implementation works by source rewriting: when "go test" is building the specified set of package tests, it runs each package source file of interest through a source-to-source translation tool that produces an instrumented/augmented equivalent, with instrumentation that records which portions of the code execute as the test runs.

A function such as

  func ABC(x int) {
    if x < 0 {
      bar()
    }
  }

is rewritten to something like

  func ABC(x int) {GoCover_0_343662613637653164643337.Count[9] = 1;
    if x < 0 {GoCover_0_343662613637653164643337.Count[10] = 1;
      bar()
    }
  }

where "GoCover_0_343662613637653164643337" is a tool-generated structure with execution counters and source position information.

The "go test" command also emits boilerplate code into the generated "_testmain.go" to register each instrumented source file and unpack the coverage data structures into something that can be easily accessed at runtime. Finally, the modified "_testmain.go" has code to call runtime routines that emit the coverage output file when the test completes.

Strengths and weaknesses of what we currently provide

The current implementation is simple and easy to use, and provides a good user experience for the use case of collecting coverage data for package unit tests. Since "go test" is performing both the build and the invocation/execution of the test, it can provide a nice seamless "single command" user experience.

A key weakness of the current implementation is that it does not scale well-- it is difficult or impossible to gather coverage data for applications as opposed to collections of packages, and for testing scenarios involving multiple runs/executions.

For example, consider a medium-sized application such as the Go compiler ("gc"). While the various packages in the compiler source tree have unit tests, and one can use "go test" to obtain coverage data for those tests, the unit tests by themselves only exercise a small fraction of the code paths in the compiler that one would get from actually running the compiler binary itself on a large collection of Go source files.

For such applications, one would like to build a coverage-instrumented copy of the entire application ("gc"), then run that instrumented application over many inputs (say, all the Go source files compiled as part of a "make.bash" run for multiple GOARCH values), producing a collection of coverage data output files, and finally merge together the results to produce a report or provide a visualization.

Many folks in the Golang community have run into this problem; there are large numbers of blog posts and other pages describing the issue, and recommending workarounds (or providing add-on tools that help); doing a web search for "golang integration code coverage" will turn up many pages of links.

An additional weakness in the current Go toolchain offering relates to the way in which coverage data is presented to the user from the "go tool cover") commands. The reports produced are "flat" and not hierarchical (e.g. a flat list of functions, or a flat list of source files within the instrumented packages). This way of structuring a report works well when the number of instrumented packages is small, but becomes less attractive if there are hundreds or thousands of source files being instrumented. For larger applications, it would make sense to create reports with a more hierarchical structure: first a summary by module, then package within module, then source file within package, and so on.

Finally, there are a number of long-standing problems that arise due to the use of source-to-source rewriting used by cmd/cover and the go command, including

#23883 "cmd/go: -coverpkg=all gives different coverage value when run on a package list vs ./..."

#23910 "cmd/go: -coverpkg packages imported by all tests, even ones that otherwise do not use it"

#27336 "cmd/go: test coverpkg panics when defining the same flag in multiple packages"

Most of these problems arise because of the introduction of additional imports in the _testmain.go shim created by the Go command when carrying out a coverage test run (in combination with the "-coverpkg" option).

§ Proposed changes

Building for coverage

While the existing "go test" based coverage workflow will continue to be supported, the proposal is to add coverage as a new build mode for "go build". In the same way that users can build a race-detector instrumented executable using "go build -race", it will be possible to build a coverage-instrumented executable using "go build -cover".

To support this goal, the plan will be to migrate a portion of the support for coverage instrumentation into the compiler, while still retaining the existing source-to-source rewriting strategy (a so-called "hybrid" approach).

Running instrumented applications

Applications are deployed and run in many different ways, ranging from very simple (direct invocation of a single executable) to very complex (e.g. gangs of cooperating processes involving multiple distinct executables). To allow for more complex execution/invocation scenarios, it doesn't make sense to try to serialize updates to a single coverage output data file during the run, since this would require introducing synchronization or some other mechanism to ensure mutually exclusive access.

For non-test applications built for coverage, users will instead select an output directory as opposed to a single file; each run of the instrumented executable will emit data files within that directory. Example:

$ go build -o myapp.exe -cover ...
$ mkdir /tmp/mycovdata
$ export GOCOVERDIR=/tmp/mycovdata
$ <run test suite, resulting in multiple invocations of myapp.exe>
$ go tool cover -html=/tmp/mycovdata
$

For coverage runs in the context of "go test", the default will continue to be emitting a single named output file when the test is run.

File names within the output directory will be chosen at runtime so as to minimize the possibility of collisions, e.g. possibly something to the effect of

  covdata.<metafilehash>.<processid>.<nanotimevalue>.out

When invoked for reporting, the coverage tool itself will test its input argument to see whether it is a file or a directory; in the latter case, it will read and process all of the files in the specified directory.

Programs that call os.Exit(), or never terminate

With the current coverage tooling, if a Go unit test invokes os.Exit() passing a non-zero exit status, the instrumented test binary will terminate immediately without writing an output data file. If a test invokes os.Exit() passing a zero exit status, this will result in a panic.

For unit tests, this is perfectly acceptable-- people writing tests generally have no incentive or need to call os.Exit, it simply would not add anything in terms of test functionality. Real applications routinely finish by calling os.Exit, however, including cases where a non-zero exit status is reported. Integration test suites nearly always include tests that ensure an application fails properly (e.g. returns with non-zero exit status) if the application encounters an invalid input. The Go project's all.bash test suite has many of these sorts of tests, including test cases that are expected to cause compiler or linker errors (and to ensure that the proper error paths in the tool are covered).

To support collecting coverage data from such programs, the Go runtime will need to be extended to detect os.Exit calls from instrumented programs and ensure (in some form) that coverage data is written out before the program terminates. This could be accomplished either by introducing new hooks into the os.Exit code, or possibly by opening and mmap'ing the coverage output file earlier in the run, then letting writes to counter variables go directly to an mmap'd region, which would eliminated the need to close the file on exit (credit to Austin for this idea).

To handle server programs (which in many cases run forever and may not call exit), APIs will be provided for writing out a coverage profile under user control. The first API variants will support writing coverage data to a specific directory path:

  import "runtime/coverage"

  var *coverageoutdir flag.String(...)

  func server() {
    ...
    if *coverageoutdir != "" {
      // Meta-data is already available on program startup; write it now.
      // NB: we're assuming here that the specified dir already exists
      if err := coverage.EmitMetaDataToDir(*coverageoutdir); err != nil {
        log.Fatalf("writing coverage meta-data: %v")
      }
    }
    for {
      ...
      if *coverageoutdir != "" && <received signal to emit coverage data> {
        if err := coverage.EmitCounterDataToDir(*coverageoutdir); err != nil {
          log.Fatalf("writing coverage counter-data: %v")
        }
      }
    }
  }

The second API variants will support writing coverage meta-data and counter data to a user-specified io.Writer (where the io.Writer is presumably backed by a pipe or network connection of some sort):

  import "runtime/coverage"

  var *codecoverageflag flag.Bool(...)

  func server() {
    ...
    var w io.Writer
    if *codecoverageflag {
      // Meta-data is already available on program startup; write it now.
      w = <obtain destination io.Writer somehow>
      if err := coverage.EmitMetaDataToWriter(w); err != nil {
        log.Fatalf("writing coverage meta-data: %v")
      }
    }
    for {
      ...
      if *codecoverageflag && <received signal to emit coverage data> {
        if err := coverage.EmitCounterDataToWriter(w); err != nil {
          log.Fatalf("writing coverage counter-data: %v")
        }
      }
    }
  }

These APIs will return an error if invoked from within an application not built with the "-cover" flag.

Coverage and modules

Most modern Go programs make extensive use of dependent third-party packages; with the advent of Go modules, we now have systems in place to explicitly identify and track these dependencies.

When application writers add a third-party dependency, in most cases the authors will not be interested in having that dependency's code count towards the "percent of lines covered" metric for their application (there will definitely be exceptions to this rule, but it should hold in most cases).

It makes sense to leverage information from the Go module system when collecting code coverage data. Within the context of the module system, a given package feeding into the build of an application will have one of the three following dispositions (relative to the main module):

• Contained: package is part of the module itself (not a dependency)
• Dependent: package is a direct or indirect dependency of the module (appearing in go.mod)
• Stdlib: package is part of the Go standard library / runtime

With this in mind, the proposal when building an application for coverage will be to instrument every package that feeds into the build, but record the disposition for each package (as above), then allow the user to select the proper granularity or treatment of dependencies when viewing or reporting.

As an example, consider the Delve debugger (a Go application). One entry in the Delve V1.8 go.mod file is:

    github.com/cosiner/argv v0.1.0

This package ("argv") has about 500 lines of Go code and a couple dozen Go functions; Delve uses only a single exported function. For a developer trying to generate a coverage report for Delve, it seems unlikely that they would want to include "argv" as part of the coverage statistics (percent lines/functions executed), given the secondary and very modest role that the dependency plays.

On the other hand, it's possible to imagine scenarios in which a specific dependency plays an integral or important role for a given application, meaning that a developer might want to include the package in the applications coverage statistics.

Merging coverage data output files

As part of this work, the proposal is to provide "go tool" utilities for merging coverage data files, so that collection of coverage data files (emitted from multiple runs of an instrumented executable) can be merged into a single summary output file.

More details are provided below in the section 'Coverage data file tooling'.

Differential coverage

When fixing a bug in an application, it is common practice to add a new unit test in addition to the code change that comprises the actual fix. When using code coverage, users may want to learn how many of the changed lines in their code are actually covered when the new test runs.

Assuming we have a set of N coverage data output files (corresponding to those generated when running the existing set of tests for a package) and a new coverage data file generated from a new testpoint, it would be useful to provide a tool to "subtract" out the coverage information from the first set from the second file. This would leave just the set of new lines / regions that the new test causes to be covered above and beyond what is already there.

This feature (profile subtraction) would make it much easier to write tooling that would provide feedback to developers on whether newly written unit tests are covering new code in the way that the developer intended.

§ Design details

This section digs into more of the technical details of the changes needed in the compiler, runtime, and other parts of the toolchain.

Package selection when building instrumented applications

In the existing "go test" based coverage design, the default is to instrument only those packages that are specifically selected for testing. Example:

  $ go test -cover p1 p2/...
  ...
  $

In the invocation above, the Go tool reports coverage for package p1 and for all packages under p2, but not for any other packages (for example, any of the various packages imported by p1).

When building applications for coverage, the default will be to instrument all packages in the main module for the application being built. Here is an example using the "delve" debugger (a Go application):

$ git clone -b v1.3.2 https://github.com/go-delve/delve
...
$ cd delve
$ go list ./...
github.com/go-delve/delve/cmd/dlv
github.com/go-delve/delve/cmd/dlv/cmds
...
github.com/go-delve/delve/service/rpccommon
github.com/go-delve/delve/service/test
$ fgrep spf13 go.mod
github.com/spf13/cobra v0.0.0-20170417170307-b6cb39589372
github.com/spf13/pflag v0.0.0-20170417173400-9e4c21054fa1 // indirect
$ go build -cover -o dlv.inst.exe ./cmd/dlv
$

When the resulting program (dlv.inst.exe) is run, it will capture coverage information for just the subset of dependent packages shown in the go list command above. In particular, not coverage will be collected/reported for packages such as github.com/spf13/cobra or for packages in the Go standard library (ex: fmt).

Users can override this default by passing the -coverpkg option to go build. Some additional examples:

  // Collects coverage for _only_ the github.com/spf13/cobra package
  $ go build -cover -coverpkg=github.com/spf13/cobra ./cmd/dlv
  // Collects coverage for all packages in the main module and 
  // all packages listed as dependencies in go.mod
  $ go build -cover -coverpkg=mod.deps ./cmd/dlv
  // Collects coverage for all packages (including the Go std library)
  $ go build -cover -coverpkg=all ./cmd/dlv
  $

Coverage instrumentation: compiler or tool?

Performing code coverage instrumentation in the compiler (as opposed to prior to compilation via tooling) has some distinct advantages.

Compiler-based instrumentation is potentially much faster than the tool-based approach, for a start. In addition, the compiler can apply special treatment to the coverage meta-data variables generated during instrumentation (marking it read-only), and/or provide special treatment for coverage counter variables (ensuring that they are aggregated).

Compiler-based instrumentation also has disadvantages, however. The "front end" (lexical analysis and parsing) portions of most compilers are typically designed to capture a minimum amount of source position information, just enough to support accurate error reporting, but no more. For example, in this code:

L11:  func ABC(x int) {
L12:    if x < 0 {
L13:      bar()
L14:    }
L15:  }

Consider the { and } tokens on lines 12 and 14. While the compiler will accept these tokens, it will not necessarily create explicit representations for them (with detailed line/column source positions) in its IR, because once parsing is complete (and no syntax errors are reported), there isn't any need to keep this information around (it would just be a waste of memory).

This is a problem for code coverage meta-data generation, since we'd like to record these sorts of source positions for reporting purposes later on (for example, during HTML generation).

This poses a problem: if we change the compiler to capture and hold onto more of this source position info, we risk slowing down compilation overall (even if coverage instrumentation is turned off).

Hybrid instrumentation

To ensure that coverage instrumentation has all of the source position information it needs, and that we gain some of the benefits of using the compiler, the proposal is to use a hybrid approach: employ source-to-source rewriting for the actual counter annotation/insertion, but then pass information about the counter data structures to the compiler (via a config file) so that the compiler can also play a part.

The cmd/cover tool will be modified to operate at the package level and not at the level of an individual source file; the output from the instrumentation process will be a series of modified source files, plus summary file containing things like the the names of generated variables create during instrumentation. This generated file will be passed to the compiler when the instrumented package is compiled.

The new style of instrumentation will segregate coverage meta-data and coverage counters, so as to allow the compiler to place meta-data into the read-only data section of the instrumented binary.

This segregation will continue when the instrumented program writes out coverage data files at program termination: meta-data and counter data will be written to distinct output files.

New instrumentation strategy

Consider the following source fragment:

        package p

  L4:   fun