Red Hat Advanced Cluster Management and Application Management, Part 1: Deployment in Multiple Environments

We kick off a series of posts in which we’ll show you how Advanced Cluster Management (ACM) delivers rich lifecycle management capabilities for applications that need to exist in multiple environments at once, whether in the cloud or in a corporate data center.

Today we will focus on the GitOps aspects of ACM and break them down using the following model configuration:

So, we have three OpenShift clusters here. ACM uses a “hub-managed” model to manage clusters, where hub is the cluster running ACM and managed are the clusters that ACM is managing. The hub uses the following Red Hat software:

BY	Version
Red Hat OpenShift Container Platform	4.5.7
Red Hat Advanced Cluster Management	2.0 Fix Pack 2.0.2

Please note that managed clusters have different labels, we will actively use them when placing applications in different environments.

As shown in the figure, we have a managed development cluster called managed-cluster1-dev deployed in the AWS cloud in the EU region. And there is also a managed production cluster called managed-cluster2-prod, which is also deployed to AWS, in the US region.

Application life cycle

ACM offers extensive application lifecycle management capabilities. Here we will look at those that belong to the GitOps category and will come in handy in the following scenarios:

Deploy your application to multiple environments.
Deploy Blue / Green.
Application migration.
Disaster recovery

First, let’s define the terms and concepts that we will operate in this article.

Channels

Channels point to some physical location where resources to be deployed are stored. Here we will use pipes of the Git type, although there are other types of pipes (Helm, Namespaces, etc.).

More details

PlacementRules

By creating and managing placement rules, you define where to deploy your Kubernetes resource subscriptions and Helm releases. By applying these rules, you can greatly simplify the deployment of Kubernetes resources across multiple environments.

More details

Subscriptions

Subscriptions are a set of definitions for selecting Kubernetes resources in channels using annotations, labels, and versions. Subscription resources are set up at the hub and pushed to managed clusters. The subscription controller monitors the source location (channel) for new or updated resources. When this happens, he can download the corresponding resource directly from the source location (channel) to the managed cluster without first checking the Hub cluster (since the subscription has already been sent).

A subscription can filter Helm releases to select the correct version of the chart. In this case, the subscription controller looks at the version parameter to understand which version of the Helm release (chart) should be taken for deployment.

More details

Applications

The Application object can be thought of as a way to aggregate subscriptions into a group. For this, there are corresponding tools and a console to perform aggregation and view all the components of the Application.

More details

Git repository

Our applications will be deployed according to GitOps templates, and for different environments you will need different manifests that will be stored in a Git repository, the structure of which is shown in the table below:

Branch	Description
Config	Stores base files for applications that are used in all environments
Prod	Stores overlay files for applications used in production environments
Stage	Stores overlay files for applications that are used in test environments

Note. Red Hat ACM restricts how you structure your Git repository. It can be organized not only as shown in this table, but also in any other way convenient for you.

Deploying the application to multiple environments

Now let’s look at how ACM can help you deploy your application across multiple environments using a simple web service that inverts words. This service has two releases: stage (this is the version that the developers are currently testing) and production (this is the version that customers are using).

ACM has support Kustomize, which greatly facilitates customizing applications for the target deployment environment.

As already mentioned, in both environments we use the same application, but only in different releases.

To deploy the application, we will use the oc tool and a set of yaml-manifests with the necessary configurations for ACM, which set Channel, Subscription, and PlacementRule. Everything we do from the command line can be done from the web console.

In the oc tool, we will have three configured contexts, one for each environment:

Context	Description
Hub	CLI profile for the HUB cluster (where ACM is deployed)
Dev	CLI profile for managed development-cluster (managed-cluster1-dev)
Pro	CLI profile for managed production cluster (managed-cluster2-prod)

You can read more about CLI profiles here…

Now let’s take a look at the resources that will be used in our example:

Channel

apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
  name: acm-app-lifecycle-blog
  namespace: open-cluster-management
spec:
  type: Git
  pathname: https://github.com/RHsyseng/acm-app-lifecycle-blog.git

We set Channel to be a Git of type Channel, which our subscriptions will use to get Kubernetes resources deploying our application.

In our case, the channel is configured to receive Kubernetes resources from the github.com/RHsyseng/acm-app-lifecycle-blog.git Git repository.

Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: reverse-words-stage

When we use Subscription, the namespace containing this subscription is passed to the target deployment cluster. Therefore, here we create a namespace called reverse-words-stage that will be passed to our dev clusters through this Subscription.

PlacementRule

apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: development-clusters
  namespace: reverse-words-stage
spec:
  clusterConditions:
    - type: "ManagedClusterConditionAvailable"
      status: "True"
  clusterSelector:
    matchExpressions: []
    matchLabels:
      environment: "dev"

The list of clusters where subscriptions are passed is taken from what the PlacementRule returns. In other words, we need to somehow select certain clusters from our environments and transfer them to various subscriptions, this is exactly the task that PlacementRules solve.

In our example, a PlacementRule named development-clusters returns all clusters that are labeled Available and labeled with environment: dev. That is, in our case, the output will be a managed dev-cluster named managed-cluster1-dev.

Subscription

apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  name: reversewords-dev-app-subscription
  namespace: reverse-words-stage
  labels:
    app: reversewords-dev-app
  annotations:
    apps.open-cluster-management.io/git-path: apps/reversewords/
    apps.open-cluster-management.io/git-branch: stage
spec:
  channel: open-cluster-management/acm-app-lifecycle-blog
  placement:
    placementRef:
      kind: PlacementRule
      name: development-clusters

In the example above, Subscription is responsible for deploying a list of Kubernetes resources (taken from the Channel) to clusters from the list (taken from the PlacementRule). But besides that, you can also specify where exactly these Kubernetes resources are located in the Git repository (channel).

Our subscription uses the Channel we defined above and takes Kubernetes resources from the stage branch, where it looks for them in the apps / reversewords / folder.

Application

apiVersion: app.k8s.io/v1beta1
kind: Application
metadata:
  name: reversewords-dev-app
  namespace: reverse-words-stage
spec:
  componentKinds:
  - group: apps.open-cluster-management.io
    kind: Subscription
  descriptor: {}
  selector:
    matchExpressions:
    - key: app
      operator: In
      values:
      - reversewords-dev-app

Application helps to create the topology of our application on clusters managed by ACM. To do this, the Application selects one or more subscriptions, which ultimately create resources on various clusters, so we can track who created what and where it was created.

Note. Here, we have covered only those resources that will be used when deploying an application in a dev environment. Resources for other environments can be found in the GitHub repository and are pretty self-explanatory and similar to the ones we’ve covered.

Deploy the application in a development environment

1. The first step is to create a Channel definition.

oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/base/00_channel.yaml

2. Next, we create a Namespace to store our application’s manifests.

oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-stage/00_namespace.yaml

3. Now we create a PlacementRule that will select our managed dev-clusters.

oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-stage/01_placement_rule.yaml

See the status of PlacementRule. Note that this rule selected the managed-cluster1-dev dev-cluster:

oc --context hub -n reverse-words-stage get placementrule development-clusters -o yaml

<OMITTED_OUTPUT>
status:
  decisions:
  - clusterName: managed-cluster1-dev
    clusterNamespace: managed-cluster1-dev

4. You can now create a Subscription and Application to target the dev cluster using PlacementRule.

oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-stage/02_subscription-dev.yaml
oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-stage/03_application-dev.yaml

See the Subscription status. Pay attention to the word propagated, it means that the subscription was sent to the target cluster:

oc --context hub -n reverse-words-stage get subscription reversewords-dev-app-subscription -o yaml
<OMITTED_OUTPUT>
status:
  message: Active
  phase: Propagated

5. Finally, we look at the dev-cluster and see that the application has been deployed and is working.

oc --context dev -n reverse-words-stage get deployments,services,pods
NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/reverse-words   1/1     1            1           73s

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)          AGE
service/reverse-words   LoadBalancer   172.30.217.208   a84668cb23acf4d109a78b119dfddbef-750551.eu-central-1.elb.amazonaws.com   8080:30053/TCP   73s

NAME                                 READY   STATUS    RESTARTS   AGE
pod/reverse-words-68b9b894dd-jfgpf   1/1     Running   0          73s

If we try to execute requests to the production cluster, we will see that the application is not running there.

oc --context pro -n reverse-words-stage get deployments,services,pods
No resources found in reverse-words-stage namespace.

6. Now let’s execute a request to our application and make sure that we have deployed the required release, namely staging:

curl http://$(oc --context dev -n reverse-words-stage get service reverse-words -o jsonpath="{.status.loadBalancer.ingress[0].hostname}"):8080
Reverse Words Release: Stage Release v0.0.3. App version: v0.0.3

Deploying the application in a production environment

1. There is no need to create a new Channel, since we will use the same Git repo as the source, but only a different branch.

2. Create a Namespace to store our application’s manifests.

oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-prod/00_namespace.yaml

3. Now we create a PlacementRule that selects production clusters:

oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-prod/01_placement_rule.yaml

See the status of PlacementRule. Note that this rule selected the managed-cluster2-prod production cluster.

oc --context hub -n reverse-words-prod get placementrule production-clusters -o yaml

<OMITTED_OUTPUT>
status:
  decisions:
  - clusterName: managed-cluster2-prod
    clusterNamespace: managed-cluster2-prod

4. You can now create a Subscription and Application to set the production cluster as the target using PlacementRule.

oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-prod/02_subscription-pro.yaml
oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-prod/03_application-pro.yaml

See the Subscription status. Pay attention to the word propagated, it means that the subscription was sent to the target cluster:

oc --context hub -n reverse-words-prod get subscription reversewords-pro-app-subscription -o yaml

<OMITTED_OUTPUT>
status:
  message: Active
  phase: Propagated

5. And finally, we look at the production cluster and see that the application has been deployed and is working.

oc --context pro -n reverse-words-prod get deployments,services,pods
NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/reverse-words   1/1     1            1           93s

NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/reverse-words   LoadBalancer   172.30.100.0   a6067d9a2cd904003a1b53b65f9e1cb3-450574743.us-west-2.elb.amazonaws.com   8080:30293/TCP   96s

NAME                                READY   STATUS    RESTARTS   AGE
pod/reverse-words-7dd94446c-vkzr8   1/1     Running   0          94s

6. Now let’s run a request to our application and make sure that we have deployed the required release, namely production:

curl http://$(oc --context pro -n reverse-words-prod get service reverse-words -o jsonpath="{.status.loadBalancer.ingress[0].hostname}"):8080
Reverse Words Release: Production release v0.0.2. App version: v0.0.2

7. We now have different versions of our application for different deployment environments:

# Query development environment
curl http://$(oc --context dev -n reverse-words-stage get service reverse-words -o jsonpath="{.status.loadBalancer.ingress[0].hostname}"):8080
# Query production environment
curl http://$(oc --context pro -n reverse-words-prod get service reverse-words -o jsonpath="{.status.loadBalancer.ingress[0].hostname}"):8080
# Dev Query
Reverse Words Release: Stage Release v0.0.3. App version: v0.0.3
# Pro Query
Reverse Words Release: Production release v0.0.2. App version: v0.0.2

And finally, let’s see how it looks in the web console:

ACM Applications General View