Kafka UI short guide

background

Just recently, I started creating my own analogue of the Discord service, using web sockets and passing messages through a queue. At one point, I needed to check whether the messages in this queue were passing correctly. Then I realized how inconvenient it is to enter the docker container every time and manually enter the command to view all the messages in the topics. After a day of analysis and advice from my colleague, an experienced DevOps engineer Pasha, I came to the conclusion that kafka-ui from the provectus team is the best and most convenient tool for visualizing data in Kafka.

docker-compose

First, let’s set up our docker-compose for modern kafka realities. To do this, we will use 2 main images

zookeeper
kafka

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-server:7.2.1
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9997:9997"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9997
      KAFKA_JMX_HOSTNAME: kafka

In the image parameters for ZooKeeper, the main thing is to specify the working port. I also added the tick_time parameter, which, although not required, I always specify out of habit. In short, tick_time is used to synchronize replicas and set timeouts for various operations.

The situation is similar for the Kafka configuration: the main thing is the working port and the basic configuration parameters associated with ZooKeeper. But the key point here is working with JMX (Java Management Extensions). JMX is a standard tool for monitoring and managing Java applications. As part of Apache Kafka, JMX allows you to monitor various broker performance metrics. For our needs, it is enough to specify the port and hostname for JMX, which, as you might have guessed, will be used later in kafka-ui.

The last step is to configure the image for kafka-ui. Having become acquainted with official documentationThanks to the provectus team, you will find many templates for kafka-ui in combination with other services. However, there is no starter template for Kafka, ZooKeeper and kafka-ui on the official website. So I’ll give you my working compose file.

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-server:7.2.1
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9997:9997"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9997
      KAFKA_JMX_HOSTNAME: kafka

  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - 8082:8080
    environment:
      DYNAMIC_CONFIG_ENABLED: true

Here, as you can see, we specify the working port, the image (image) that Docker should load, and the “DYNAMIC_CONFIG_ENABLED” variable. It is needed, not surprisingly, to be able to change the configuration of your kafka-ui in real time. You can find the full list of environment variables by clicking on the provided link.

Connecting a broker to Kafka-UI

After successfully running your docker-compose file, you need to navigate to the port you specified for kafka-ui and see the following picture:

Now let’s add our kafka to kafka-ui. To do this, let’s click on the button configure new cluster where you need to fill in the following fields:

Cluster name – You can simply specify it as “Kafka Cluster”
Bootstrap Servers – So you need to enter PLAINTEXT://kafka:29092 or something else depending on your configuration of the “KAFKA_ADVERTISED_LISTENERS” parameter in the kafka image. Accordingly, if you have several kafka replicas raised, you need to enter them all. Apache recommends having 3 kafka nodes in your project.
Metrics
1. metrics type -> JMX
2. port -> 9997 or as you specified in your configuration

After that, your Kafka cluster should show up in your GUI. It looks something like this:

Functional

The most important thing in this service, as for me, is the functionality for viewing a topic inside your kafka. To do this, we need to go to the Topics tab and then go to the messages tab where we can see the full information for each message and a filter to search for specific messages.

You can also view your consumers and their groups in the consumers tab.

To view the configuration, you need to go to the Brokers tab and select the broker you need

Next, go to the Metrics tab and view all the information we need.

Outcome

In the process of creating my analogue of Discord services, I was faced with the need to monitor my message queue in Kafka. I was looking for a convenient tool for this, and thanks to the advice of my colleague Pasha, a DevOps engineer, I noticed kafka-ui from the provectus team. This interface allowed me to easily keep track of all the posts in a topic without having to manually log into the docker container. I would especially like to note the convenience and functionality of kafka-ui: viewing messages in topics, monitoring consumers and their groups, as well as the ability to view the configuration and broker metrics. Using this tool, I greatly simplified the process of working and diagnosing my system. If any of you are working with Kafka, I recommend that you pay attention to kafka-ui!