go. About code coverage by integration tests and the -cover flag

Code coverage tools help you understand what part of the codebase is being executed (or, as they say, covered) when executing a given set of tests. For a while, Go supported measuring code coverage at the package level, introduced in Go 1.2, it was enabled by the command flag go test -cover.

This works well in most cases, but when developing large applications, disadvantages are found. For large applications, developers often write integration tests that test the behavior of the entire program (in addition to unit tests at the package level).

Unlike testing packages individually, this type of testing typically involves building a complete application binary, running it on a set of representative inputs (or under a workload if it’s a server) to ensure that all component packages work together correctly.

The integration test binaries are generated by the command go buildbut not go testso the Go toolkit has not yet provided an easy way to collect a coverage profile of these tests.

As of Go 1.20, coverage instrumented programs can be created with the command go build -coverand then pass those instrumented binaries into an integration test to expand coverage.

Below is an example of how these new features work, along with use cases and a workflow for collecting coverage profiles from integration tests.

Example

Let’s take a very small sample program, write a simple integration test for it, and then collect a coverage profile from the integration test.

Let’s use the Markdown processing tool for this. mdtool from here. This is a demo showing how clients can use the markdown to HTML conversion library gitlab.com/golang-commonmark/markdown.

Download a specific version mdtoolso that these steps can be repeated:

$ git clone https://gitlab.com/golang-commonmark/mdtool.git
...
$ cd mdtool
$ git tag example e210a4502a825ef7205691395804eefce536a02f
$ git checkout example
...
$

Simple integration test

Let’s write a simple integration test mdtool; it will create a binary file mdtool and run it on a set of markdown input files. This very simple script runs the binary mdtool for each file in the test data directory to make sure the tool produces some result and doesn’t crash.

$ cat integration_test.sh
#!/bin/sh
BUILDARGS="$*"
#
# Terminate the test if any command below does not complete successfully.
#
set -e
#
# Download some test inputs (the 'website' repo contains various *.md files).
#
if [ ! -d testdata ]; then
 git clone https://go.googlesource.com/website testdata
 git -C testdata tag example 8bb4a56901ae3b427039d490207a99b48245de2c
 git -C testdata checkout example
fi
#
# Build mdtool binary for testing purposes.
#
rm -f mdtool.exe
go build $BUILDARGS -o mdtool.exe .
#
# Run the tool on a set of input files from 'testdata'.
#
FILES=$(find testdata -name "*.md" -print)
N=$(echo $FILES | wc -w)
for F in $FILES
do
 ./mdtool.exe +x +a $F > /dev/null
done
echo "finished processing $N files, no crashes"
$

Here is an example of running our test:

$ /bin/sh integration_test.sh
...
finished processing 380 files, no crashes
$

Binary mdtool successfully processed a set of input files … but what part of the source code of the tool did we use? Next, we will collect a coverage profile to find out.

How to use an integration test to collect coverage data

Let’s write a wrapper script that calls the previous script and creates a coverage tool and then post-processes the resulting profiles:

$ cat wrap_test_for_coverage.sh
#!/bin/sh
set -e
PKGARGS="$*"
#
# Setup
#
rm -rf covdatafiles
mkdir covdatafiles
#
# Pass in "-cover" to the script to build for coverage, then
# run with GOCOVERDIR set.
#
GOCOVERDIR=covdatafiles \
 /bin/sh integration_test.sh -cover $PKGARGS
#
# Post-process the resulting profiles.
#
go tool covdata percent -i=covdatafiles
$

Here are some key points to note in the above shell:

  • it starts with a flag -coverwhen executed integration_test.shwhich gives us the coverage binary mdtool.exe;
  • it sets the GOCOVERDIR environment variable to the path to the write directory for the coverage data files;
  • at the end of the test to generate a report on the percentage of agents reached, runs go tool covdata percent.

Here is the result of running this new shell:

$ /bin/sh wrap_test_for_coverage.sh
...
 gitlab.com/golang-commonmark/mdtool coverage: 48.1% of statements
$
# Note: covdatafiles now contains 381 files.

Now we have some idea of ​​how well the integration tests work with the source code.

If we make changes to improve the test suite and then run a second run of coverage collection, we will see the changes in the coverage report. Suppose, for example, that we improve the test with new lines in the file integration_test.sh:

./mdtool.exe +ty testdata/README.md > /dev/null
./mdtool.exe +ta < testdata/README.md > /dev/null

Shell launch:

$ /bin/sh wrap_test_for_coverage.sh
finished processing 380 files, no crashes
 gitlab.com/golang-commonmark/mdtool coverage: 54.6% of statements
$

Operator coverage increased from 48% to 54%.

Choice of packages to be covered

Default go build -cover only uses packages that are part of the Go module being created, here it is gitlab.com/golang-commonmark/mdtool. However, it is sometimes useful to extend the coverage toolkit to other packages; this can be done, for example, by passing the flag -coverpkg V go build -cover.

To a large extent mdtool – just a wrapper around the package gitlab.com/golang-commonmark/markdownThat’s why markdown interesting to include in a set of instrumented packages.

Here is the file go.mod For mdtool:

$ head go.mod
module gitlab.com/golang-commonmark/mdtool

go 1.17

require (
 github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8
 gitlab.com/golang-commonmark/markdown v0.0.0-20211110145824-bf3e522c626a
)

To control which packets are included in the coverage analysis, you can use the flag -coverpkg:

$ /bin/sh wrap_test_for_coverage.sh -coverpkg=gitlab.com/golang-commonmark/markdown,gitlab.com/golang-commonmark/mdtool
...
 gitlab.com/golang-commonmark/markdown coverage: 70.6% of statements
 gitlab.com/golang-commonmark/mdtool coverage: 54.6% of statements
$

When the coverage integration test is completed and a set of raw data files (contents covdatafiles), these files can be processed in different ways.

Convert profiles to text format ‘-coverprofile’

When working with unit tests, you can run go test -coverprofile=abc.txt — recording a profile for a given test coverage in the form of text.

With the help of binaries compiled go build -coveryou can generate a profile in text format after the fact by running go tool covdata textfmt on files sent to the GOCOVERDIR directory.

After completing this step, you can use go tool cover -func=<file> or go tool cover -html=<file>to interpret/visualize the data in the same way as with go test -coverprofile.

Example:

$ /bin/sh wrap_test_for_coverage.sh
...
$ go tool covdata textfmt -i=covdatafiles -o=cov.txt
$ go tool cover -func=cov.txt
gitlab.com/golang-commonmark/mdtool/main.go:40: readFromStdin 100.0%
gitlab.com/golang-commonmark/mdtool/main.go:44: readFromFile 80.0%
gitlab.com/golang-commonmark/mdtool/main.go:54: readFromWeb 0.0%
gitlab.com/golang-commonmark/mdtool/main.go:64: readInput 80.0%
gitlab.com/golang-commonmark/mdtool/main.go:74: extractText 100.0%
gitlab.com/golang-commonmark/mdtool/main.go:88: writePreamble 100.0%
gitlab.com/golang-commonmark/mdtool/main.go:111: writePostamble 100.0%
gitlab.com/golang-commonmark/mdtool/main.go:118: handler 0.0%
gitlab.com/golang-commonmark/mdtool/main.go:139: main 51.6%
total: (statements) 54.6%
$

Each execution of an embedded application with -cover will write one or more data files to the directory specified by the GOCOVERDIR environment variable. If there are N runs of the program during an integration test, there will eventually be O(N) files in the output directory. There is usually a lot of duplicate content in data files, so to compress data and/or merge datasets from different runs of integration tests, you can use the merge profiles command go tool covdata merge:

$ /bin/sh wrap_test_for_coverage.sh
finished processing 380 files, no crashes
 gitlab.com/golang-commonmark/mdtool coverage: 54.6% of statements
$ ls covdatafiles
covcounters.13326b42c2a107249da22f6e0d35b638.772307.1677775306041466651
covcounters.13326b42c2a107249da22f6e0d35b638.772314.1677775306053066987
...
covcounters.13326b42c2a107249da22f6e0d35b638.774973.1677775310032569308
covmeta.13326b42c2a107249da22f6e0d35b638
$ ls covdatafiles | wc
 381 381 27401
$ rm -rf merged ; mkdir merged ; go tool covdata merge -i=covdatafiles -o=merged
$ ls merged
covcounters.13326b42c2a107249da22f6e0d35b638.0.1677775331350024014
covmeta.13326b42c2a107249da22f6e0d35b638
$

Team go tool covdata merge also accepts -pkgA that can be used to select a specific package or set of packages.

This is useful for merging the results of different types of test runs, including runs created by other test suites.

With the release of version 1.20, the Go code coverage toolkit is no longer limited to package tests, but supports profiling from larger integration tests. We hope you’ll take advantage of the new features to understand how well large and complex tests perform and what parts of your source code they use.

Try these new features and, as always, if you encounter any problems, please report them at GitHub. Thank you.

Brief catalog of courses

Data Science and Machine Learning

Python, web development

Mobile development

Java and C#

From basics to depth

And

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *