Configuring distributed tracing in Kubernetes with OpenTracing, Jaeger and Ingress-NGINX

In an era when large companies are using more than 500 microservices at the same time, it is important to be able to quickly figure out what caused the failure or decreased performance. Without specialized tools, it can be like looking for a needle in a haystack.

Distributed Tracing is the method used to monitor applications. For microservices, it is simply irreplaceable.

As an example, we will use the application Meow-Microcreated specially for this article. But you can deploy your own applications if you like.

What do we need?

Initial knowledge GoLang
Understanding the principles Kubernetes
Experience with NGINX / Ingress-NGINX
Install Docker-Desktop
Install Helm v3

This tutorial assumes that you understand Go code, know how to use Ingress-nginx, and know how core Kubernetes objects such as Service and Deployment work.

If you want to brush up on your knowledge, use these resources:

Running Kubernetes in Docker Desktop

Let’s start with the installation Docker Desktop, which allows not only to run containers, but also to create a local Kubernetes cluster.

After installing Docker Desktop, follow these steps to create your Kubernetes cluster:

Click on the icon Preferences

2. Select the tab Kubernetes, check the box Enable Kubernetes and press the button Apply & Restart

3. In the window that appears, click the button Install and wait for the installation to complete

4. Select an item Kubernetes in the taskbar

5. In the context menu, select docker-desktop

6. Check that you are connected to the correct cluster

$ kubectl cluster-info

Kubernetes master is running at 
https://kubernetes.docker.internal:6443
KubeDNS is running at https://kubernetes.docker.internal:6443/api/v1/
namespaces/kube-system/services/kube-dns:dns/proxy

Installing the Ingress-NGINX Controller

1. Use the command from official leadership to install Ingress-NGINX Controller.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/cloud/deploy.yaml

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created]
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created

2. Verify that the Ingress Controller installation was successful. The pods must be running and in the Ready state.

$ kubectl get pods -n ingress-nginx

NAME                                     READY   STATUS    RESTARTS 
ingress-nginx-admission-create-52jsl        0/1     Completed   0          
ingress-nginx-admission-patch-78fkc         0/1     Completed   0          
ingress-nginx-controller-6f5454cbfb-qsfnn   1/1     Running     0

Important: If the Pod is in Pending status due to lack of CPU / Memory resources, you can add resources to your cluster in the Docker Desktop settings.

Installing Jaeger and configuring the Ingress Controller

Jaeger is a distributed tracing platform that we will use to monitor our microservices. Install Jaeger and enable tracing at the Ingress Controller level.

1. Clone the repository first meow-micro, we will use this project in all examples.

$ git clone https://github.com/diazjf/meow-micro.git

Cloning into 'meow-micro'...
remote: Enumerating objects: 105, done.
...
$ cd meow-micro

2. In the repository you will find manifests for jaeger-all-in-one… Let’s install Jaeger from these manifests.

$ kubectl apply -f jaeger/jaeger-all-in-one.yaml

deployment.apps/jaeger created
service/jaeger-query created
service/jaeger-collector created
service/jaeger-agent created
service/zipkin created

3. Make sure Jaeger is up and running.

$ kubectl get pods

NAME                      READY   STATUS    RESTARTS   AGE
jaeger-6f6b5d8689-8gccp   1/1     Running   0          17s

4. It’s time to configure the joint work of Ingress-NGINX and Jaeger, for this you need to add parameters enable-opentracing and jaeger-collector-host in ingress-nginx-controller ConfigMap. In the parameter jaeger-collector-host specify the name of the Jaeger service.

$ echo '
  apiVersion: v1
  kind: ConfigMap
  data:
    enable-opentracing: "true"
    jaeger-collector-host: jaeger-agent.default.svc.cluster.local              
  metadata:
    name: ingress-nginx-controller
    namespace: ingress-nginx
  ' | kubectl replace -f -
configmap/ingress-nginx-controller replaced

5. Make sure that Ingress Controller is enabled and configured in the settings. opentracing…

$ kubectl get pods -n ingress-nginx | grep controller

ingress-nginx-controller-6f5454cbfb-qptxt   1/1     Running     0          8m56s

$ kubectl exec -it ingress-nginx-controller-6f5454cbfb-qptxt -n ingress-nginx -- bash -c "cat nginx.conf | grep ngx_http_opentracing_module.so"

load_module /etc/nginx/modules/ngx_http_opentracing_module.so;

$ kubectl exec -it ingress-nginx-controller-6f5454cbfb-qptxt -n ingress-nginx -- bash -c "cat nginx.conf | grep jaeger"

opentracing_load_tracer /usr/local/lib/libjaegertracing_plugin.so /etc/nginx/opentracing.json;

$ kubectl exec -it ingress-nginx-controller-6f5454cbfb-qptxt -n ingress-nginx -- bash -c "cat /etc/nginx/opentracing.json"

{
  "service_name": "nginx",
  "propagation_format": "jaeger",
  "sampler": {
    "type": "const",
    "param": 1,
    "samplingServerURL": "http://127.0.0.1:5778/sampling"
  },
  "reporter": {
    "endpoint": "",
    "localAgentHostPort": "jaeger-agent.default.svc.cluster.local:6831"
  },
  "headers": {
    "TraceContextHeaderName": "",
    "jaegerDebugHeader": "",
    "jaegerBaggageHeader": "",
    "traceBaggageHeaderPrefix": ""
  }
}

Jaeger and Ingress Controller have been successfully installed and configured. It’s time to deploy our microservices!

Deploying a test application

Test application meow-micro consists of two microservices. Client meow-client – accepts a REST request and sends information to the service meow-server via GRPC.

Details on using REST and GRPC with GoLang:

The test application also contains several additional tools:

tracing.go

Gets parameters for configuring Jaeger from the environment where Helm runs for installation meow-client and meow-server.

cfg, err := config.FromEnv()
if err != nil {
panic(fmt.Sprintf("Could not parse Jaeger env vars: %s", err.Error())) 
}  
tracer, closer, err := cfg.NewTracer()
if err != nil {  
 panic(fmt.Sprintf("Could not initialize jaeger tracer: %s", err.Error())) 
}

client.go

Performs client configuration – sets the service name to trace and defines the span.

Span Is the basic element of distributed tracing. It is a description of a certain workflow (for example, a query to a database). Spans usually contain links to other spans, which allows multiple spans to be combined in a Trace.
Trace – visualization of the life of a request in the process of its movement in a distributed system.

Detailed information on concepts trace and span can be found in the official documentation.

// main function span
os.Setenv("JAEGER_SERVICE_NAME", "meow-client")
tracer, closer := tracing.Init()
defer closer.Close()
http.HandleFunc("https://habr.com/", func(w http.ResponseWriter, r *http.Request) {
spanCtx, _ := tracer.Extract(opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(r.Header))
span := tracer.StartSpan("send-meow-communication", ext.RPCServerOption(spanCtx))
defer span.Finish()
...
// sleep function span
os.Setenv("JAEGER_SERVICE_NAME", "meow-client")
tracer, closer := tracing.Init()
defer closer.Close()
span := tracer.StartSpan("sleep")
defer span.Finish()

Installing the test application

Install Helm v3 can be used on any operating system, for example macOS using brew. We are now ready to deploy microservices on our cluster.

$ brew install helm
...
==> Downloading https://ghcr.io/v2/homebrew/core/helm/manifests/3.5.4
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/helm/blobs/sha256:5dac5803c1ad2db3a91b0928fc472aaf80a4
==> Downloading from https://pkg-containers-az.githubusercontent.com/ghcr1/blobs/sha256:5dac5803c1ad2db
######################################################################## 100.0%
==> Pouring helm--3.5.4.big_sur.bottle.tar.gz
...
$ helm version
version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

Let’s use the Makefile from the repository to deploy the test application to the Kubernetes cluster.

# Build client and server from Dockerfile
$ make build
docker build -t meow-client:1.0 -f client/Dockerfile .
[+] Building 17.1s (10/10) FINISHED
...
docker build -t meow-server:1.0 -f server/Dockerfile .
[+] Building 0.5s (10/10) FINISHED
...
# Install Microservices into Kubernetes via Helm

$ make install
helm install -f helm/Values.yaml meow-micro ./helm
NAME: meow-micro
LAST DEPLOYED: Mon Apr 26 13:42:38 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Let’s check that the Pods of both services are up and running.

$ kubectl get pods

NAME                           READY   STATUS    RESTARTS   AGE
jaeger-6f6b5d8689-s7cln        1/1     Running   0          26m
meow-client-8b974778c-85896    1/1     Running   0          15m
meow-server-56f559db44-5mvgp   1/1     Running   0          15m

Viewing trace data

Now comes the fun part! Let’s take a look at the tracing.

Open the Jaeger console by pointing your browser to the address http: // localhost: 8081…

2. Let’s send a request to our application.

$ curl http://localhost/meow -X POST -d '{"name": "Meow-Mixer"}'
200 - Meow sent: Meow-Mixer

3. Refresh the browser page. From the Service menu choose – nginx…

4. Press the button Find traces…

5. All traces for nginx will be displayed. Each of them can be expanded to get more information.

6. Now you can track how long each request took.

And display additional information for each request.

Despite the fact that this small example gives an idea of what distributed tracing is, you can see its full power only in real applications under load.

Other tracing systems

We’ve covered how to use Jaeger to trace requests. Ingress-Nginx… You can also use Zipkin or DataDog for these purposes.

Configuring distributed tracing in Kubernetes with OpenTracing, Jaeger and Ingress-NGINX

What do we need?

Running Kubernetes in Docker Desktop

Installing the Ingress-NGINX Controller

Installing Jaeger and configuring the Ingress Controller

Deploying a test application

tracing.go

client.go

Installing the test application

Viewing trace data

Other tracing systems

Using the NgOptimizedImage directive to implement image loading in Angular

How We Visualized 150+ Pages of DS Solution Documentation

4 rules from art school that I use in my work

Unity 2022.2 continues to integrate async await

Four Software Development Principles I Learned the Hard Way

Risks of Artificial Intelligence in Critical Infrastructure

Leave a Reply Cancel reply

What do we need?

Running Kubernetes in Docker Desktop

Installing the Ingress-NGINX Controller

Installing Jaeger and configuring the Ingress Controller

Deploying a test application

tracing.go

client.go

Installing the test application

Viewing trace data

Other tracing systems

Similar Posts

Leave a Reply Cancel reply