Go: Goroutine Leak Detector

The translation of the article was prepared on the eve of the start of the course “Golang Developer. Professional”

We also invite everyone to a demo lesson “Blockchain on Go”. In this webinar, we will create a blockchain-based lightweight cryptocurrency.

Leaking goroutines can be easily detected using APM, which tracks the number of running goroutines. Here’s an example from NewRelic – a graph that monitors goroutines:

Source: https://docs.newrelic.com/docs/agents/go-agent/features/go-runtime-page-troubleshoot-performance-problems
Source: https://docs.newrelic.com/docs/agents/go-agent/features/go-runtime-page-troubleshoot-performance-problems

The leak will cause this number to increase continuously until the server crashes. However, there are ways to prevent leaks even before the code is deployed.

Leak detection

The Go team at Uber, very active on Github, created a goroutine leak detector (goroutine leak detector) Is a tool that aims to integrate with unit tests. This package monitors for goroutine leaks in the currently tested piece of code. Here’s an example of a function that leaks a goroutine:

And here’s a test for this function:

Running tests detects the leak:

The error message contains two useful facts that can help us:

  • Top of the stack with a leaked goroutine with its state. This information can help you quickly debug and understand which goroutine is leaking.

  • A goroutine identifier, useful when visualizing execution with a tracer. Here is an example of traces created during tests using go test ./... -trace trace.out:

Then, based on these traces, you can trace the execution of this goroutine in detail.

Well, a goroutine leak has been discovered and we also have information about it. Let’s take a look at how it works now to be aware of the limitations of such detection.

Internal organization

The only requirement for the leak detector to work is to call the library at the end of the test to detect the goroutine leak. In fact, it checks all goroutines, not just the one causing the leak.

The detector first lists all existing goroutines. Here’s a list from the previous example:

The goroutine stack is provided by the exported runtime function. Stack from the standard Go library. Hence, it is available to anyone who needs this information. but goroutine ids are not exported

Then, based on this list, the detector can analyze by parsing these lines and removing the goroutines belonging to the standard library, such as:

  • The goroutines created by the test suite to run tests are the second goroutine (# 1) from the previous example.

  • Runtime-generated goroutines, such as the ones that listen to the received signal. For more information, I suggest you read “Go: gsignal, Master of Signals“.

  • The currently running goroutine is the first goroutine (# 18) in the previous example.

Finally, after filtering, if there are no goroutines left, it means that no leak has occurred. However, we can see that it has some limitations:

  • A third party library or internal goroutine that runs a goroutine in the background that is not properly handled will cause a false positive report.

  • A false positive also occurs if the goroutine leaks in another test not covered by the detector. If the goroutine still works the next time the detector is started, it will falsely report a leak.

This tool is not perfect, but knowing its capabilities and limitations can help detect leaks directly from tests, avoiding the need to debug code already running in production.

It’s worth noting that this method is also widely used in the net / http package for detecting goroutine leaks. Here’s an example of some tests:

Here’s the inner function afterTest looks through the stacks of goroutines to find those that could potentially leak.

Learn more about the course “Golang Developer. Professional”

Watch the webinar “Blockchain on Go”.

Similar Posts

Leave a Reply