Access to a site on NestJS and Angular by domain name with SSL certificate in Kubernetes via Ingress

Previous article: Installing Kubernetes via MicroK8s and Configuring Deployment of NestJS and Angular Applications

In Kubernetes, it is very easy to configure work with SSL, this is probably one of the main reasons why I started using it, in this article I will describe a simple scenario for connecting it.

1. Installing a certificate manager on a dedicated server

We connect to our server via SSH and enable the cert-manager plugin in MicroK8s.

The cert-manager plugin manages the process of issuing and renewing SSL certificates.

Teams

ssh root@194.226.49.162
microk8s enable cert-manager

Console output

root@vps1724252356:~# microk8s enable cert-manager
Infer repository core for addon cert-manager
Enable DNS addon
Infer repository core for addon dns
Addon core/dns is already enabled
Enabling cert-manager
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
Waiting for cert-manager to be ready.
..ready
Enabled cert-manager

===========================

Cert-manager is installed. As a next step, try creating a ClusterIssuer
for Let's Encrypt by creating the following resource:

$ microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: public
EOF

Then, you can create an ingress to expose 'my-service:80' on 'https://my-service.example.com' with:

$ microk8s enable ingress
$ microk8s kubectl create ingress my-ingress \
    --annotation cert-manager.io/cluster-issuer=letsencrypt \
    --rule 'my-service.example.com/*=my-service:80,tls=my-service-tls'

2. Create a file in the repository with resource parameters for creating certificates

There are various free and paid sites that issue SSL certificates, and we can specify in cert-manager how and from whom to get these certificates.

Personally, I only set up receiving certificates from https://letsencrypt.org And https://www.cloudflare.comthis project will use certificates from Let's Encrypt.

In order to specify a source for obtaining certificates, we need to create a resource for creating certificates.

Create a file .kubernetes/templates/node/8.issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: nestjs-mod@site15.ru
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: public

3. Create a file in the repository with the parameters of the Kubernetes entity that is responsible for proxying external traffic to our services

By default, this entity is created based on Nginx, but you can also configure Traefik, in this project Nginx will be used, since it is substituted by default.

Create a file .kubernetes/templates/node/8.issuer.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: '%NAMESPACE%'
  name: %NAMESPACE%-client-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
spec:
  rules:
    - host: %SERVER_DOMAIN%
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: %NAMESPACE%-client
                port:
                  number: %NGINX_PORT%
  tls:
    - hosts:
        - %SERVER_DOMAIN%
      secretName: %NAMESPACE%-client-tls

4. We slightly change the CI/CD configuration for deployment in Kubernetes to avoid unnecessary redeployment of the infrastructure

The deployment pipeline can be broken down in different ways, as is convenient for everyone and as needed, but in this post I will simply prohibit restarting the infrastructure if the application version has not been changed.

Updated version of the task for deploying an application .github/workflows/kubernetes.yml

# ...
jobs:
  # ...
  deploy:
    environment: kubernetes
    needs: [check-server-image, build-and-push-migrations-image, build-and-push-server-image, build-and-push-nginx-image, build-and-push-e2e-tests-image]
    runs-on: [self-hosted]

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - name: Deploy infrastructure
        if: ${{ needs.check-server-image.outputs.result != 'success' || contains(github.event.head_commit.message, '[skip cache]') || contains(github.event.head_commit.message, '[skip infrastructure]') }}
        env:
          SERVER_APP_DATABASE_NAME: ${{ secrets.SERVER_APP_DATABASE_NAME }}
          SERVER_APP_DATABASE_PASSWORD: ${{ secrets.SERVER_APP_DATABASE_PASSWORD }}
          SERVER_APP_DATABASE_USERNAME: ${{ secrets.SERVER_APP_DATABASE_USERNAME }}
          SERVER_POSTGRE_SQL_POSTGRESQL_DATABASE: ${{ secrets.SERVER_POSTGRE_SQL_POSTGRESQL_DATABASE }}
          SERVER_POSTGRE_SQL_POSTGRESQL_PASSWORD: ${{ secrets.SERVER_POSTGRE_SQL_POSTGRESQL_PASSWORD }}
          SERVER_POSTGRE_SQL_POSTGRESQL_USERNAME: ${{ secrets.SERVER_POSTGRE_SQL_POSTGRESQL_USERNAME }}
        run: |
          rm -rf ./.kubernetes/generated
          . .kubernetes/set-env.sh && npx -y rucken copy-paste --find=templates --replace=generated --replace-plural=generated --path=./.kubernetes/templates --replace-envs=true
          docker compose -f ./.kubernetes/generated/docker-compose-infra.yml --compatibility down || echo 'docker-compose-infra not started'
          docker compose -f ./.kubernetes/generated/docker-compose-infra.yml --compatibility up -d

      - name: Deploy applications
        env:
          DOCKER_SERVER: ${{ env.REGISTRY }}
          DOCKER_USERNAME: ${{ github.actor }}
          DOCKER_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
          SERVER_APP_DATABASE_NAME: ${{ secrets.SERVER_APP_DATABASE_NAME }}
          SERVER_APP_DATABASE_PASSWORD: ${{ secrets.SERVER_APP_DATABASE_PASSWORD }}
          SERVER_APP_DATABASE_USERNAME: ${{ secrets.SERVER_APP_DATABASE_USERNAME }}
          SERVER_DOMAIN: ${{ secrets.SERVER_DOMAIN }}
        run: |
          rm -rf ./.kubernetes/generated
          . .kubernetes/set-env.sh && npx -y rucken copy-paste --find=templates --replace=generated --replace-plural=generated --path=./.kubernetes/templates --replace-envs=true
          chmod +x .kubernetes/generated/install.sh
          .kubernetes/generated/install.sh > /dev/null 2>&1 &

      - name: Run E2E-tests
        env:
          SERVER_DOMAIN: ${{ secrets.SERVER_DOMAIN }}
        run: |
          rm -rf ./.kubernetes/generated
          . .kubernetes/set-env.sh && npx -y rucken copy-paste --find=templates --replace=generated --replace-plural=generated --path=./.kubernetes/templates --replace-envs=true
          docker compose -f ./.kubernetes/generated/docker-compose-e2e-tests.yml --compatibility up

5. Change the address of the tested site in “Docker Compose” – a file for E2E tests

Since we now have a domain with an SSL certificate, we no longer need to specify the port with the global Nginx frontend service when running E2E tests.

Updated file .kubernetes/templates/docker-compose-e2e-tests.yml

version: '3'
networks:
  nestjs-mod-fullstack-network:
    driver: 'bridge'
services:
  nestjs-mod-fullstack-e2e-tests:
    image: 'ghcr.io/nestjs-mod/nestjs-mod-fullstack-e2e-tests:%ROOT_VERSION%'
    container_name: 'nestjs-mod-fullstack-e2e-tests'
    networks:
      - 'nestjs-mod-fullstack-network'
    environment:
      BASE_URL: 'https://%SERVER_DOMAIN%'
    working_dir: '/usr/src/app'
    volumes:
      - './../../apps:/usr/src/app/apps'
      - './../../libs:/usr/src/app/libs'

6. Commit the changes and wait until CI/CD works successfully and manually check the site's operation

Current CI/CD work result: https://github.com/nestjs-mod/nestjs-mod-fullstack/actions/runs/10877250887
Current site: https://fullstack.nestjs-mod.com

7. Delete the global service that was created in the previous post

Now the deployment and testing of the site is carried out, which works through Ingress, and this means that the site launched as a global service through NodePort is no longer needed and can be deleted.

You can delete it by connecting to a dedicated server via ssh using the command sudo microk8s kubectl delete service master-client-global --namespace masterbut it is always advisable to deliver changes via a git repository, since we may have several stands on which we will also need to run this command again.

In this project, we will deliver changes via git, for this we will delete the file .kubernetes/templates/client/4.global-service.yaml and add the removal command to the application installation script.

Updating the file .kubernetes/templates/install.sh

#!/bin/bash
set -e

# docker regcred for pull docker images
sudo microk8s kubectl delete secret docker-regcred || echo 'not need delete secret docker-regcred'
sudo microk8s kubectl create secret docker-registry docker-regcred --docker-server=%DOCKER_SERVER% --docker-username=%DOCKER_USERNAME% --docker-password=%DOCKER_PASSWORD% --docker-email=docker-regcred

# namespace and common config
sudo microk8s kubectl apply -f .kubernetes/generated/node
sudo microk8s kubectl get secret docker-regcred -n default -o yaml || sed s/"namespace: default"/"namespace: %NAMESPACE%"/ || microk8s kubectl apply -n %NAMESPACE% -f - || echo 'not need update docker-regcred'

# server
sudo microk8s kubectl apply -f .kubernetes/generated/server

# client
sudo microk8s kubectl apply -f .kubernetes/generated/client

# depricated
sudo microk8s kubectl delete service master-client-global --namespace master || echo 'not need delete master-client-global'

8. Commit the changes and wait for CI/CD to work successfully and manually check that the site http://fullstack.nestjs-mod.com:30222 no longer works

Current site: https://fullstack.nestjs-mod.com

Conclusion

In this project, Ingress will simply act as a proxy to our own Nginx with a built-in frontend application, and all further new microservices and applications that will be developed and that will need access from the outside will also be described in our own Nginx.

We need our own Nginx with all our routing rules so that we can deploy all our applications without Kubernetes and not have to have several different Nginx configuration files.

Plans

In the next post I will add semantic versioning of applications that will be launched depending on changes in dependent files…

Links

https://nestjs.com – official website of the framework
https://nestjs-mod.com – official site of additional utilities
http://fullstack.nestjs-mod.com:30222 – site from the post
https://github.com/nestjs-mod/nestjs-mod-fullstack – project from the post
https://github.com/nestjs-mod/nestjs-mod-fullstack/compare/28ebc77b38b2b1c9945e87806e5726451b8d22a2..33b51edf67471600e583f89f10b2d99a1b9b79da – changes

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *