go. About code coverage by integration tests and the -cover flag
Code coverage tools help you understand what part of the codebase is being executed (or, as they say, covered) when executing a given set of tests. For a while, Go supported measuring code coverage at the package level, introduced in Go 1.2, it was enabled by the command flag
go test -cover.
This works well in most cases, but when developing large applications, disadvantages are found. For large applications, developers often write integration tests that test the behavior of the entire program (in addition to unit tests at the package level).
Unlike testing packages individually, this type of testing typically involves building a complete application binary, running it on a set of representative inputs (or under a workload if it’s a server) to ensure that all component packages work together correctly.
The integration test binaries are generated by the command
go buildbut not
go testso the Go toolkit has not yet provided an easy way to collect a coverage profile of these tests.
As of Go 1.20, coverage instrumented programs can be created with the command
go build -coverand then pass those instrumented binaries into an integration test to expand coverage.
Below is an example of how these new features work, along with use cases and a workflow for collecting coverage profiles from integration tests.
Let’s take a very small sample program, write a simple integration test for it, and then collect a coverage profile from the integration test.
Let’s use the Markdown processing tool for this.
mdtool from here. This is a demo showing how clients can use the markdown to HTML conversion library
Download a specific version
mdtoolso that these steps can be repeated:
$ git clone https://gitlab.com/golang-commonmark/mdtool.git ... $ cd mdtool $ git tag example e210a4502a825ef7205691395804eefce536a02f $ git checkout example ... $
Simple integration test
Let’s write a simple integration test
mdtool; it will create a binary file
mdtool and run it on a set of markdown input files. This very simple script runs the binary
mdtool for each file in the test data directory to make sure the tool produces some result and doesn’t crash.
$ cat integration_test.sh #!/bin/sh BUILDARGS="$*" # # Terminate the test if any command below does not complete successfully. # set -e # # Download some test inputs (the 'website' repo contains various *.md files). # if [ ! -d testdata ]; then git clone https://go.googlesource.com/website testdata git -C testdata tag example 8bb4a56901ae3b427039d490207a99b48245de2c git -C testdata checkout example fi # # Build mdtool binary for testing purposes. # rm -f mdtool.exe go build $BUILDARGS -o mdtool.exe . # # Run the tool on a set of input files from 'testdata'. # FILES=$(find testdata -name "*.md" -print) N=$(echo $FILES | wc -w) for F in $FILES do ./mdtool.exe +x +a $F > /dev/null done echo "finished processing $N files, no crashes" $
Here is an example of running our test:
$ /bin/sh integration_test.sh ... finished processing 380 files, no crashes $
mdtool successfully processed a set of input files … but what part of the source code of the tool did we use? Next, we will collect a coverage profile to find out.
How to use an integration test to collect coverage data
Let’s write a wrapper script that calls the previous script and creates a coverage tool and then post-processes the resulting profiles:
$ cat wrap_test_for_coverage.sh #!/bin/sh set -e PKGARGS="$*" # # Setup # rm -rf covdatafiles mkdir covdatafiles # # Pass in "-cover" to the script to build for coverage, then # run with GOCOVERDIR set. # GOCOVERDIR=covdatafiles \ /bin/sh integration_test.sh -cover $PKGARGS # # Post-process the resulting profiles. # go tool covdata percent -i=covdatafiles $
Here are some key points to note in the above shell:
- it starts with a flag
integration_test.shwhich gives us the coverage binary
- it sets the GOCOVERDIR environment variable to the path to the write directory for the coverage data files;
- at the end of the test to generate a report on the percentage of agents reached, runs
go tool covdata percent.
Here is the result of running this new shell:
$ /bin/sh wrap_test_for_coverage.sh ... gitlab.com/golang-commonmark/mdtool coverage: 48.1% of statements $ # Note: covdatafiles now contains 381 files.
Now we have some idea of how well the integration tests work with the source code.
If we make changes to improve the test suite and then run a second run of coverage collection, we will see the changes in the coverage report. Suppose, for example, that we improve the test with new lines in the file
./mdtool.exe +ty testdata/README.md > /dev/null ./mdtool.exe +ta < testdata/README.md > /dev/null
$ /bin/sh wrap_test_for_coverage.sh finished processing 380 files, no crashes gitlab.com/golang-commonmark/mdtool coverage: 54.6% of statements $
Operator coverage increased from 48% to 54%.
Choice of packages to be covered
go build -cover only uses packages that are part of the Go module being created, here it is
gitlab.com/golang-commonmark/mdtool. However, it is sometimes useful to extend the coverage toolkit to other packages; this can be done, for example, by passing the flag
go build -cover.
To a large extent
mdtool – just a wrapper around the package
markdown interesting to include in a set of instrumented packages.
Here is the file
$ head go.mod module gitlab.com/golang-commonmark/mdtool go 1.17 require ( github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 gitlab.com/golang-commonmark/markdown v0.0.0-20211110145824-bf3e522c626a )
To control which packets are included in the coverage analysis, you can use the flag
$ /bin/sh wrap_test_for_coverage.sh -coverpkg=gitlab.com/golang-commonmark/markdown,gitlab.com/golang-commonmark/mdtool ... gitlab.com/golang-commonmark/markdown coverage: 70.6% of statements gitlab.com/golang-commonmark/mdtool coverage: 54.6% of statements $
When the coverage integration test is completed and a set of raw data files (contents
covdatafiles), these files can be processed in different ways.
Convert profiles to text format ‘-coverprofile’
When working with unit tests, you can run
go test -coverprofile=abc.txt — recording a profile for a given test coverage in the form of text.
With the help of binaries compiled
go build -coveryou can generate a profile in text format after the fact by running
go tool covdata textfmt on files sent to the GOCOVERDIR directory.
After completing this step, you can use
go tool cover -func=<file> or
go tool cover -html=<file>to interpret/visualize the data in the same way as with
go test -coverprofile.
$ /bin/sh wrap_test_for_coverage.sh ... $ go tool covdata textfmt -i=covdatafiles -o=cov.txt $ go tool cover -func=cov.txt gitlab.com/golang-commonmark/mdtool/main.go:40: readFromStdin 100.0% gitlab.com/golang-commonmark/mdtool/main.go:44: readFromFile 80.0% gitlab.com/golang-commonmark/mdtool/main.go:54: readFromWeb 0.0% gitlab.com/golang-commonmark/mdtool/main.go:64: readInput 80.0% gitlab.com/golang-commonmark/mdtool/main.go:74: extractText 100.0% gitlab.com/golang-commonmark/mdtool/main.go:88: writePreamble 100.0% gitlab.com/golang-commonmark/mdtool/main.go:111: writePostamble 100.0% gitlab.com/golang-commonmark/mdtool/main.go:118: handler 0.0% gitlab.com/golang-commonmark/mdtool/main.go:139: main 51.6% total: (statements) 54.6% $
Each execution of an embedded application with
-cover will write one or more data files to the directory specified by the GOCOVERDIR environment variable. If there are N runs of the program during an integration test, there will eventually be O(N) files in the output directory. There is usually a lot of duplicate content in data files, so to compress data and/or merge datasets from different runs of integration tests, you can use the merge profiles command
go tool covdata merge:
$ /bin/sh wrap_test_for_coverage.sh finished processing 380 files, no crashes gitlab.com/golang-commonmark/mdtool coverage: 54.6% of statements $ ls covdatafiles covcounters.13326b42c2a107249da22f6e0d35b638.772307.1677775306041466651 covcounters.13326b42c2a107249da22f6e0d35b638.772314.1677775306053066987 ... covcounters.13326b42c2a107249da22f6e0d35b638.774973.1677775310032569308 covmeta.13326b42c2a107249da22f6e0d35b638 $ ls covdatafiles | wc 381 381 27401 $ rm -rf merged ; mkdir merged ; go tool covdata merge -i=covdatafiles -o=merged $ ls merged covcounters.13326b42c2a107249da22f6e0d35b638.0.1677775331350024014 covmeta.13326b42c2a107249da22f6e0d35b638 $
go tool covdata merge also accepts
-pkgA that can be used to select a specific package or set of packages.
This is useful for merging the results of different types of test runs, including runs created by other test suites.
With the release of version 1.20, the Go code coverage toolkit is no longer limited to package tests, but supports profiling from larger integration tests. We hope you’ll take advantage of the new features to understand how well large and complex tests perform and what parts of your source code they use.
Try these new features and, as always, if you encounter any problems, please report them at GitHub. Thank you.
Data Science and Machine Learning
Python, web development
Java and C#
From basics to depth