Keycloak. Account mapping via mTLS with double certificate verification in kubernetes

Setting up mTLS on the ingress controller

Mutual TLS (mTLS), also known as mutual TLS, strengthens the security of entry to protected services by checking the client certificate in addition to the server one. This extension of the TLS protocol is applicable in infrastructures with a predetermined number and composition of users, i.e. for the internal segment of the company.

Most services and web servers that have TLS connection termination functionality out of the box can “basically” check client certificates. Example: nginx, traefik, caddy, HAproxy, envoy, etc.

We terminate TLS connections on the ingress controller of the kubernetes cluster. We use classic nginx as an ingress controller.

For further actions we will need a CA certificate, an Intermediate certificate and a client certificate signed by the above Intermediate certificate. For this purpose, we have deployed HashiCorp Vault, and we use its PKI secrets engine functionality. For testing, readers of this article do not need to deploy HC Vault; it is enough to manually create a key pair for CA, Intermediate and a client certificate using openSSL. Let's give an example.

openssl req -x509 -sha256 -days 3650 -newkey rsa:2048 -keyout rootCA.key -out rootCA.crt

openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes

openssl x509 -req -CA rootCA.crt -CAkey rootCA.key -in server.csr -out server.crt -days 365 -CAcreateserial -extfile server.ext

openssl req -new -newkey rsa:2048 -nodes -keyout user1.key -out user1.csr

openssl x509 -req -CA rootCA.crt -CAkey rootCA.key -in user1.csr -out user1.crt -days 365 -CAcreateserial

openssl pkcs12 -export -out user1.p12 -name "user1" -inkey user1.key -in user1.crt

.ext file
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
subjectAltName = @alt_names
[alt_names]
DNS.1 = server

In order to verify the validity of a user-supplied certificate, you need to place the CA that signed the intermediate (client) certificate somewhere. In our case, we will place it in Secret k8s. The name of the certificate file in the secret must be ca.crt:

apiVersion: v1
kind: Secret
metadata:
  name: <ИМЯ>
  namespace: <ИМЯ НЕЙМСПЕЙСА>
type: Opaque
data:
  ca.crt: <PEM СЕРТИФИКАТ В BASE64>

The certificate is verified on each host separately. Therefore, to enable certificate verification on the desired service, you need to add annotations to the ingress manifest:

annotations:
  nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-secret
  nginx.ingress.kubernetes.io/auth-tls-verify-client: 'on'

The complete manifest will look something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-secret
    nginx.ingress.kubernetes.io/auth-tls-verify-client: 'on'
 
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - test-ingress.local
      secretName: web-tls
  rules:
    - host: test-ingress.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: test-ingress
                port:
                  number: 80

After adding these annotations, the ingress controller will not let you through until you present a personal certificate. It will look something like this:

Upon presentation of the required certificate and its successful verification, you will be redirected to the site you need, and if the attempt is unsuccessful, the answer will be as follows:

Setting up mTLS on keycloak (mapping certificate data and Keycloak accounts)

If there is a desire or need to configure authentication in keycloak using client certificates, then the following is required.

1. Go to keycloak in the authentication settings (Authentication menu item), find browser flow and click Duplicate:

2. Select a new name, for example x509-auth, and remove unnecessary authentication steps, leaving Cookies and, for example, User-Pass-OTP as an alternative authentication option:

3. Click on Add Step and select x509/Validate Username Form:

4. Move this step higher, right after Cookies, set it as Alternative and click on the gear to configure. Here, come up with an Alias, specify the User Identity Source (Subject's email) and User mapping method (Username or Email).

!!! In this case, a check is carried out using the user's email address. Accordingly, the user must have an email address. Save.

5. Now this configuration needs to be bound to browser flow. In the authentication steps settings, in the upper right corner, click Action – Bind flow. Let's select Browser flow:

6. In deployment/statefullset keycloak you need to set some variables necessary to read the client certificate through the ingress controller:

env:
  - name: KC_SPI_X509CERT_LOOKUP_PROVIDER # какой reverse-proxy server (ingress-контроллер) используется
    value: nginx
  - name: KC_SPI_X509CERT_LOOKUP_NGINX_SSL_CLIENT_CERT # имя заголовка, содержащего клиентский сертификат
    value: ssl-client-cert
  - name: KEYCLOAK_X509CERT_LOOKUP_NGINX_SSL_CERT_CHAIN_PREFIX # префикс заголовков, содержащий дополнительные сертификаты в цепочке и используемый для извлечения отдельных сертификатов в соответствии с длиной цепочки
    value: USELESS
  - name: KEYCLOAK_X509CERT_LOOKUP_NGINX_SSL_CERTIFICATE_CHAIN_LENGTH # максимальная длина цепочки сертификатов
    value: '2'

7. In the keycloak Ingress manifest, add the following annotations to verify the client certificate and transfer it to the keycloak service:

annotations:
  nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: 'true' # Указывает, следует ли передавать полученные сертификаты вышестоящему серверу в заголовке ssl-client-cert. Возможные значения: «true» или «false»
  nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-secret # Путь к секрету в виде namespace/secret. Сертификат в секрете обязательно должен лежать в файле с именем ca.crt
  nginx.ingress.kubernetes.io/auth-tls-verify-client: on # Включает проверку пользовательского сертификата. Возможные опции: on - всегда включена. off - всегда выключена. optional - проверка происходит, но необязательна. optional_no_ca - проверка происходит, но не выводит ошибку, если сертификат не подписан указанным CA.
  nginx.ingress.kubernetes.io/auth-tls-verify-depth: '2' # Глубина проверки между предоставленным сертификатом клиента и цепочкой центра сертификации.
  nginx.ingress.kubernetes.io/proxy-ssl-secret: default/ca-secret # Используется для проверки сертификата проксируемого HTTPS-сервера.
  nginx.ingress.kubernetes.io/proxy-ssl-verify: 'off' # Включает или отключает проверку сертификата проксируемого HTTPS-сервера
  nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: '2' # Глубина проверки между предоставленным сертификатом проксируемого HTTPS-сервера и цепочкой центра сертификации.

After performing these steps, when switching to a service with configured authentication using keycloak, we will see the following picture when logging in:

Some of the information is hidden, but it can be understood that keycloak saw the client certificate, matched the email and realized that they matched the user sandzhiev, and offered to log in under this account.

Setting up client certificate re-verification in keycloak

If we read the note from the GitHub guys

mTLS: When certificate authentication is done wrong

(I learned about it from the k8security channel) or watch their performance at blackhat USA 2023, then we’ll understand what we need:

• update keycloak to at least 21.1.2 (since only from this version it verifies the entire chain of certificates);
• configure the verification of client certificates not only on reverse-proxy, but also in iam itself (do not trust the verification of client certificates only to the ingress controller);
• additionally double-check the “network” inside the cube so that keycloak cannot direct the header to bypass the reverse-proxy (ingress controller).

Here's how to set it up.

1. It is necessary to put a CA with the entire chain of certificates inside the keycloak (truststore). To do this, we need to convert the certificates we need (CA and intermediate) into .jks (Java KeyStore, relevant up to version keycloak 23.x, in versions 24+ you can even upload pem), put it in secret k8s and mount this secret to deployment/statefullset keycloak – as shown below.

Create jks and put it in the k8s secret:

keytool -import -alias moi-ca -keystore truststore.jks -file ca.pem
keytool -list -v -keystore truststore.jks
kubectl create secret generic dss-truststore --from-file=./truststore.jks -n keycloak

Let's mount secret to deployment/statefullset keycloak, and also set variables so that keycloak picks up .jks:

apiVersion: apps/v1
kind: StatefulSet
spec:
...
  template:
    spec: 
      containers:
        - name: keycloak
...
          env:
...
            - name: KC_SPI_TRUSTSTORE_FILE_HOSTNAME_VERIFICATION_POLICY # depricated c версии 24.x (Политика проверки имени хоста TLS для исходящих запросов HTTPS и SMTP)
              value: "WILDCARD"
            - name: KC_SPI_TRUSTSTORE_FILE_FILE # depricated c версии 24.x (путь до хранилища доверенных сертификатов)
              value: "/opt/keycloak/spi-certs/truststore.jks"
            - name: KC_SPI_TRUSTSTORE_FILE_TYPE # depricated c версии 24.x (тип хранилища доверенных сертификатов, например jks, pkcs12 или bcfks)
              value: "jks"
            - name: KC_SPI_TRUSTSTORE_FILE_PASSWORD # depricated c версии 24.x
              valueFrom:
                secretKeyRef:
                  name: keycloak-spi-passwords
                  key: "spi-truststore-password"
            #- name: KC_TLS_HOSTNAME_VERIFIER # новый формат c версии 24.x (замена KC_SPI_TRUSTSTORE_FILE_HOSTNAME_VERIFICATION_POLICY)
            #  value: "DEFAULT"
            #- name: KC_TRUSTSTORE_PATHS # новый формат c версии 24.x (замена KC_SPI_TRUSTSTORE_FILE_FILE, теперь принимает pkcs12 (расширения файлов p12 или pfx), файлы PEM или каталоги, содержащие эти файлы)
            #  value: "/opt/keycloak/spi-certs/..."
            - name: KC_SPI_X509CERT_LOOKUP_PROVIDER
              value: nginx
            - name: KC_SPI_X509CERT_LOOKUP_NGINX_SSL_CLIENT_CERT
              value: ssl-client-cert
            - name: KEYCLOAK_X509CERT_LOOKUP_NGINX_SSL_CERT_CHAIN_PREFIX
              value: USELESS
            - name: KEYCLOAK_X509CERT_LOOKUP_NGINX_SSL_CERTIFICATE_CHAIN_LENGTH
              value: '2'
...
          volumeMounts:
            - name: spi-certificates
              mountPath: /opt/keycloak/spi-certs
              readOnly: true
      volumes:
        - name: spi-certificates
          secret:
            secretName: dss-truststore
            defaultMode: 420
...

2. Go to the “admin panel” of keycloak, to the settings of x509/Validate Username Form of the Browser flow we created earlier (Configure → Authentication → → x509/Validate Username Form → Setting)

and change the value of the following parameters to True:

• Check certificate validity
• Revalidate Client Certificate (Certificate Policy Validation Mode=All)

After the above steps, the following workflow starts working in keycloak.

• The client sends an authentication request over an SSL/TLS channel.
• During handshake, the SSL/TLS server and client exchange x.509/v3 certificates.
• The container (WildFly) checks the PKIX certificate path and the certificate's expiration date.
• The x.509 Client Certificate Authenticator verifies the client certificate using the following methods:
  • checks whether the key in the certificate matches the expected key;
  • checks whether the extended key in the certificate matches the expected extended key.
• If any of these checks fail, x.509 authentication fails. Otherwise, the authenticator retrieves the certificate ID and matches it to an existing user.

Checking revoked client certificates using keycloak

The workflow we configured lacks checking for revoked client certificates. Keycloak can check for revoked certificates using the online certificate status protocol (OCSP) and certificate revocation lists (CRLs) models. This article will not describe how to populate CRL lists, create CRL Distribution Points, or how to configure an OSCP server; there is plenty of information on these issues on the Internet.

1. Setting up CRLs.

We go to the keycloak admin panel, to the settings of x509/Validate Username Form of the Browser flow we created earlier (Configure → Authentication → → x509/Validate Username Form → Setting) and find the lines associated with CRLs:

• CRL Checking Enabled — enable/disable CRL checking;
• Enable CRL Distribution Point to check certificate revocation status – enable/disable the search for CRL lists from distribution points. Can only be used if you have a cRLDistributionPoints field in your certificate with a URL to the lists;
• CRL Path – path to the file with CRL lists (can be local or you can specify a URL).

2. Setting up OCSP.

We go to the keycloak admin panel, to the x509/Validate Username Form settings of the Browser flow we created earlier (Configure → Authentication → → x509/Validate Username Form → Setting) and find the lines related to OCSP:

• OCSP Checking Enabled — enable/disable OCSP checking;
• OCSP Fail-Open Behavior – Allow/deny authentication for client certificates that have missing/invalid/undefined OCSP endpoints. By default, a successful OCSP response is required;
• OCSP Responder Uri – The OCSP responder Uri for checking the certificate revocation status;
• OCSP Responder Certificate – An optional certificate used by the responder to sign responses. The certificate must be in PEM format without BEGIN and END tags. It is only used if the OCSP responder URI is set. By default, the OCSP responder certificate is the issuer certificate of the certificate being verified, or a certificate with the OCSPSigning extension, issued by the same certificate authority. This parameter specifies the OCSP responder certificate if default values ​​are not used.

After these settings, keycloak will add the following items to its workflow.

• Check certificate revocation status using CRLs or CRL distribution points.
• Checking certificate revocation status using OCSP (Online Certificate Status Protocol).

If the CRL and/or OCSP is successfully verified, you will be directed to the required resource. If the configuration is incorrect or you try to log in with a revoked certificate, you will receive a response similar to this:

Past publications

You can read previous posts from our team at the following links:

1. Keycloak. Admin factor and authentication ban
2. Keycloak. Standalone-HA in k8s and closing the admin panel on ingress-e with transfer to localhost
3. What and why to read DevSecOps: personal experience