Tracing services through the message queue. OpenTelemetry, NATS

This is a short guide on how to provide observability in your event driven cloud system.

A bit of theory

cloud application

Cloud computing enables organizations to build and run scalable applications in today’s dynamic environments such as public, private, and hybrid clouds…

They make loosely coupled systems stable, manageable, and observable. Combined with robust automation, they allow engineers to make important changes frequently and predictably with minimal effort.

– Cloud Native Computing Foundation (CNCF)

By this definition, cloud applications are more than just applications that run in the cloud. They also scalable, loosely coupled, resilient, manageable and observable. We can say that the presence of these “cloud attributes” allows you to call the system cloud.

Event-driven architecture (EDA)

Architecture, event driven, promotes weak interdependence between system components, resulting in greater flexibility. Microservices can scale independently, fail without affecting other services, and reduce the complexity of workflows.

observability

observability a system is a measure of the ease of determining its internal state from observable results. The system is considered observableif you can quickly and consistently get answers to all new questions about it with minimal prior knowledge, without having to hack into existing code or write new one.

opentelemetry

OpenTelemetry, also known as OTel, is an open CNCF standard that provides distributed tracing and metrics collection from your applications.

trace context

This is metadata about the spans in the trace. For example, suppose service A calls service B and you want to trace the call. In this case, OpenTelemetry will use the Trace Context to grab the trace ID and current span from service A so spans created in service B can be connected and added to the trace.

This is known as context dissemination.

Context Propagation

Context propagation is the basic concept that enables distributed tracing. When the context is propagated, spans can be related to each other and collected in a trace, regardless of where they are generated. OpenTelemetry defines context propagation with two subconcepts: Context and Propagation.

The Context is an object that contains information for the sending and receiving services to match one span to another and to associate it with the trace as a whole.

Propagation is a mechanism that moves context between services and processes. In doing so, it collects a distributed trace. It serializes or deserializes the Span Context and provides the appropriate trace information for propagation from one service to another.

On practice

ok let’s create an instance propagation and initialize it:

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
)

tc := propagation.TraceContext{}
// Register the TraceContext propagator globally.
otel.SetTextMapPropagator(tc)

In service A, whose context we want to pass:

// GetTextMapPropagator returns the global TextMapPropagator.
prop := otel.GetTextMapPropagator()
// HeaderCarrier adapts http.Header to satisfy the TextMapCarrier interface.
headers := make(propagation.HeaderCarrier)
prop.Inject(ctx, headers)

after that we have to pass those headers somehow, in the request body, in the request headers, it all depends on your implementation.

In service B where we want to get the context:

var headers propagation.HeaderCarrier
// we get the headers and convert them to HeaderCarrier...
prop := otel.GetTextMapPropagator()
// Extract reads cross-cutting concerns from the carrier into a Context.
ctx = prop.Extract(ctx, headers)

NATS

// Simple Async Subscriber
nc.Subscribe("foo", func(m *nats.Msg) {
    fmt.Printf("Received a message: %s\n", string(m.Data))
})

// Header represents the optional Header for a NATS message,
// based on the implementation of http.Header.
type Header map[string][]string

// Msg represents a message delivered by NATS. This structure is used
// by Subscribers and PublishMsg().
type Msg struct {
	Header  Header
}

it is easy to see that propagation.HeaderCarrier And nats.Header based on implementation http.Header. Therefore, in order to copy data from one structure to another, I used the implementation http.Header.Clone()

In custody

Everything is quite simple if you read the documentation a little. It was very difficult to match Russian terms to the established English definitions, so I simply did not translate most of them so as not to mislead you.

The complete project code is available in my repository – nats-tracing.

Also, when writing the guide, I used open sources: