Security in Apache Kafka
In previous articles, we looked at the architecture of the Apache Kafka solution, deployed the product, and dealt with sending and receiving messages. One could, of course, dive further into the topic of using this solution, but there are many different publications on the Internet with examples of using Kafka for various tasks and various development environments. Therefore, this article will be entirely devoted to such an important topic as securing Apache Kafka.
By its purpose, Kafka is an intermediary between different systems in their exchange of messages. For this, as we remember, there are topics and sections. But the problem is that with the default Kafka setup, any user or application can write any message to any topic, as well as read data from any topic. Naturally, for any industrial application of the system, such an approach is unacceptable. For example, in the case when several applications use the same Kafka cluster, or when the cluster is used as a cloud service for third-party applications, and of course, when confidential information begins to be processed in Kafka.
In the article, we will talk about the security mechanisms built into Kafka and will not touch on the use of any imposed means. Starting with version 0.9.0.0, a number of features have been added to the product that allow you to increase the security level of Kafka. Here are the main features:
Authenticate connections to brokers from clients (producers and consumers), other brokers, and applications using SSL or SASL (Kerberos).
Authentication of connections from brokers to ZooKeeper.
Encryption of data transmitted between brokers and clients, between brokers, or between brokers and instruments, using SSL (note that there is a performance penalty when enabling SSL, the amount of which depends on the type of processor and implementation of the JVM).
Authorization of read/write operations performed by clients.
Authorization is pluggable and supports integration with external authorization services.
Further in the article, I will not pay too much attention to the “materiel”, that is, the description of the operation of certain widespread security mechanisms, such as SSL. If necessary, the reader can find all the necessary information on the Internet. Instead, we will focus more on setting up security mechanisms in Kafka directly.
Working with SSL
SSL (secure sockets layer) is a cryptographic protocol for secure communication. In Kafka, this protocol is disabled by default. But we can enable SSL at any time.
Working with SSL, like in most other systems, in Kafka begins with the creation of a certificate. When we installed Zookeeper and Kafka, Java was previously deployed, which includes the keytool utility. Next, we will generate a key in a temporary key store so that we can later export and sign it with a certificate authority.
keytool -keystore server.keystore.jks -alias localhost -validity {validity} -genkey
In my example, such a key was generated. Please note that a password is required.
The keystroke parameter specifies in which file to store the key, and validity is the validity period of the certificate in days. Keep in mind that for certificates to work correctly, DNS must be properly configured to correctly resolve hostnames.
The steps above to generate keys must be done on all nodes of the cluster.
Now we have certificates, but they are self-signed, that is, an attacker can, if desired, also generate his own certificate, and by exchanging public keys with a pair of legal participants, read and modify all their traffic (Man in the Middle attack). Therefore, it is important to prevent certificate forgery by signing certificates for each machine in the cluster. To accomplish this task, you need a certificate authority (CA) that is responsible for signing certificates. The CA signs the certificates, and the cryptography ensures that the signed certificate is computationally difficult to forge. This way, as long as the CA is authentic and trustworthy, clients can be confident that they are connecting to genuine machines.
Next, we will generate a pair of public and private keys and a certificate with which we will actually sign other certificates. We will then add the generated CA certificate to the client’s trust store so that clients can trust it.
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
Our trust store stores all the client certificates it needs to trust. Importing a certificate into your truststore also means you trust all certificates signed by that certificate. This attribute is called the chain of trust and is especially useful when deploying SSL in a large Kafka cluster. You can sign all certificates in a cluster with a single CA and all computers will use the same trust store that the CA trusts.
Now we need to sign our generated certificate with a CA certificate. To do this, we first export the certificate from the store and then sign it with a CA certificate and import both certificates into our store:
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
Setting up brokers
Setting up brokers begins with specifying a list of ports on which we will accept connections. Since we are using an SSL connection, the following settings must be made on the broker side:
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=…
ssl.key.password=…
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=…To check the correctness of the settings made, you can run the following commands:
openssl s_client -debug -connect localhost:9093 -tls1
Setting up SASL
SASL (Simple Authentication and Security Layer) is a platform for authentication and data protection in Internet protocols. It aims to separate Internet protocols from specific authentication mechanisms. Consider how SASL works.
When connected, the server sends a request to the client with a list of possible authentication mechanisms, and the client sends a response based on the information in the request. The request and response are arbitrarily long byte arrays and can therefore contain any data specific to a particular mechanism.
This communication may continue for several iterations and finally ends when the server no longer issues any calls. The main thing to understand about how SASL works is that this authentication mechanism only provides a framework for exchanging challenge and response data. It does not mention anything about the data itself or how it is exchanged. These tasks must be performed by applications using SASL.
Let’s get back to setting up Kafka. To set up authentication, we need the Kerberos protocol, that is, we can use Active Directory for authentication. If your network does not have AD, then you will need to deploy a third-party Kerberos server. One way or another, you need to create entries (principals) in your AD directory or other system for each Kafka broker in your cluster.
In the broker configuration directory, create the kafka_server_jaas.conf file with the following content:
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_server.keytab"
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};
// Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_server.keytab"
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_server.keytab"
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};
// Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_server.keytab"
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};Next, we need to tell Kafka the paths to the configuration files:
-Djava.security.krb5.conf=/etc/kafka/krb5.conf
-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
Let’s add the ports that the system should listen on:
listeners=SASL_PLAINTEXT://host.name:port
If SASL_SSL is used, then SSL must also be configured. If you are configuring only a SASL port (or if you want Kafka brokers to authenticate each other with SASL), then make sure you set the same SASL protocol for communication between brokers:
security.inter.broker.protocol=SASL_PLAINTEXT
We also need to set the service name in server.properties to match the main name of the kafka brokers. In the example above, the principal is “kafka/kafka1.hostname.com@EXAMPLE.com “, so:
sasl.kerberos.service.name=kafka
We connect clients
The clients of our Kafka cluster will authenticate to the cluster with their own account (usually the same username as the user running the client). For each client, we will need to create a JAAS file in the same way as we did above. The following is an example configuration for a client using keytab:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_client.keytab"
principal="kafka-client-1@EXAMPLE.COM";
};The Kafka Client section describes how clients such as producers and consumers can connect to Kafka Broker. Next, we will pass the paths to the configuration files to the system in the same way as we did above.
-Djava.security.krb5.conf=/etc/kafka/krb5.conf
-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
And finally, we need to configure the following properties on our producers and consumers in producer.properties or consumer.properties
security.protocol=SASL_PLAINTEXT (or SASL_SSL)
sasl.kerberos.service.name=kafka
About ACL
To conclude the Kafka security topic, let’s look at the possibilities for working with ACLs. By default, if a resource is not associated with an ACL, then no one but the superuser will have access to it. You can change these settings by editing the broker.properties file.
allow.everyone.if.no.acl.found=true
If we want to add any users to the superuser group, then we need to list them (note that the delimiter is a semicolon, since SSL usernames can contain a comma).
super.users=User:Bob;User:Alice
Conclusion
On this topic, the basic security configuration in Apache Kafka can be considered complete. We looked at working with SSL, SASL authentication and related settings on clients. And right now I want to invite you to free webinar, within which we will consider how you can work with Kafka in Spring Boot applications. Learn what the Spring framework provides for accelerated development of applications that work with Kafka. Let’s see what the settings are, how it’s all configured. Let’s draw a line between the “native functionality” of Kafka api and the “add-ons” from Spring Boot.
