cross plane. Where Kubernetes and clouds meet


The Kubernetes container orchestration system has become the de facto gold standard in DevOps, as a versatile tool for managing configuration and automatically maintaining the expected state of the system. But, even when using Managed Kubernetes solutions (for example, Amazon EX, GKE or decisions from Yandex Clouds or Mail.Ru), some provider services are hosted outside of Kubernetes (for example, S3-compatible storage, queue brokers, or databases) and would like to be able to manage using a common configuration. In the article we will discuss the possibilities of using the Crossplane project (included in Incubating CNCF), and also look at the provider code using the example of managing MySQL databases.

cross plane installed as an operator in a Kubernetes cluster and can be managed through the kubectl extension (kubectl crossplane subcommand) or through creating a Custom Resource on the cluster. The provider of additional resources and their preconfigured combinations (which are called CombinedResource or XR) are providers (Provider), which is a controller for registering resource types and implementing the logic for creating, updating, coordinating the state with an external resource and deleting them. The provider can be installed via kubectl crossplane install provider crossplane/provider- (can also install an external package from Docker Hub or any other OCI compliant registry), or by creating a resource with type ClusterPackageInstall. The provider can be made or independently (you can use the template https://github.com/crossplane/provider-template) or generated from the Terraform provider using Terrajet (for example, https://github.com/yandex-cloud/provider-jet-yc)

For example, the following providers are available out of the box:

  • provider-aws – used to manage Amazon cloud resources

  • provider-gcp – for Google Cloud Platform

  • provider-azure – for Microsoft Azure resources

  • provider-digitalocean – for Digital Ocean resources

  • provider-alibaba – for Alibaba Cloud

  • provider-ibm-cloud – for IBM Cloud

  • provider-jet-yc – to manage Yandex Cloud resources

  • provider-cloudflare – cloudflare management

  • provider-linode – Linode provider resource management

  • provider-terraform – resource management using terraform

  • provider-kubernetes – remote Kubernetes cluster management

  • provider-sql – used to manage databases and data schema in relational DBMS

  • provider-kafka – kafka resource management

  • provider-influxdb – influxdb resource management

  • provider-rook – resource management of the rook operator (used to deploy distributed storage, for example based on ceph)

Consider using a provider using the example of Yandex Cloud (now this is the most accessible resource in the face of problems with paying for foreign services). Let’s start by installing the Crossplane operator in the cluster (we will use minikube, helm must also be installed in advance):

minikube start
kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane

Let’s create a key to access Yandex Cloud (after creating a service account):

yc iam service-account create --name crossplane
yc resource-manager folder add-access-binding <folder_id> --service-account-name crossplane --role admin
yc iam service-account get crossplane
yc iam key create --service-account-id service_account_id --output key.json
kubectl create secret generic yc-creds -n "crossplane-system" --from-file=credentials=./key.json

And we will create a resource for connecting the yandex-cloud provider and a configuration that determines the location of the secret for accessing the cloud:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-yandex-cloud
spec:
  package: "cr.yandex/crp0kch415f0lke009ft/crossplane/provider-jet-yc:v0.1.28"
---
apiVersion: yandex-cloud.jet.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      name: yc-creds
      namespace: crossplane-system
      key: credentials

And check the availability of the provider: kubectl get provider

NAME                    INSTALLED   HEALTHY   PACKAGE                                                             AGE
provider-yandex-cloud   True        True      cr.yandex/crp0kch415f0lke009ft/crossplane/provider-jet-yc:v0.1.28   28m

We can also verify that the provider has registered additional resource types (output snippet):

buckets                                        storage.yandex-cloud.jet.crossplane.io/v1alpha1           false        Bucket
objects                                        storage.yandex-cloud.jet.crossplane.io/v1alpha1           false        Object
defaultsecuritygroups                          vpc.yandex-cloud.jet.crossplane.io/v1alpha1               false        DefaultSecurityGroup
networks                                       vpc.yandex-cloud.jet.crossplane.io/v1alpha1               false        Network

When creating a provider, you can use Vault to access secrets. Now we can use the created resources to manage the cloud, for example, we can create a bucket in S3:

apiVersion: iam.yandex-cloud.jet.crossplane.io/v1alpha1
kind: ServiceAccountStaticAccessKey
metadata:
  name: bucket-creds
spec:
  forProvider:
    description: "static access key for object storage"
    serviceAccountIdRef:
      name: crossplane
  providerConfigRef:
    name: provider-yandex-cloud
  writeConnectionSecretToRef:
    name: bucket-creds
    namespace: crossplane-system
---
apiVersion: storage.yandex-cloud.jet.crossplane.io/v1alpha1
kind: Bucket
metadata:
  name: example-bucket
spec:
  forProvider:
    accessKeyRef:
      name: bucket-creds
    secretKeySecretRef:
      name: bucket-creds
      namespace: crossplane-system
      key: attribute.secret_key
    bucket: "example-test-bucket"
    acl: "public-read"
  providerConfigRef:
    name: provider-yandex-cloud

Using the resource example, you can see that the specification is defined by the following keys (inside the spec):

  • forProvider – this object is passed to the provider unchanged and is intended to determine the configuration of a particular object

  • providerConfigRef.name – link to the corresponding provider (by name), if not specified, the default provider will be used

  • publishConnectionDetailsTo – configuration to save connection information

  • writeConnectionSecretToRef – the name of the secret where the configuration can be written after the object is created (for example, generated access keys)

It is important to note that the provider not only creates resources, but also keeps them up to date, so if the bucket is deleted through the cloud web interface, it will be recreated. Also, in some cases, it is possible to apply a configuration update over an existing object. When the Bucket is deleted, the object will also be deleted from the cloud (unless Orphan was specified in spec.deletionPolicy). To get detailed information about the spec structure, you can use the explain mechanism in kubernetes:

kubectl explain Bucket.spec

Similarly, other types of objects are manipulated (including those with other providers), including virtual machines, message queues, and other managed services (databases, network balancers, etc.). You can also define your own compositions of objects (a kind of “macro”) for uniform deployment. To do this, you need to create a CompositeResourceDefinition resource in apiextensions.crossplane.io/v1, specify in the specification group (group for composition, will be used with versions.name as apiVersion), names.kind (resource definition name), names.plural (plural for resource definition name), claimName.kind (name of the resource to be created), claimName.plurlal (plural for the resource to be created). The versions enumerates the list of resource versions to be defined with the structure:

  • name – name (for example, v1alpha1, full apiVersion name will be collected from group/name)

  • served=true – if the resource version is available via the API

  • referenceable=true – if it is possible to instantiate (should be set on only one entry)

  • schema.openAPIV3Schema – schema definition (define spec structures for a resource)

For example, we can define a resource for creating a bucket, for which two parameters will be passed: the name of the secret to store access keys and the name of the bucket:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresqlinstances.database.example.org
spec:
  group: bucket.yandexcloud.ru
  names:
    kind: XBucketInstance
    plural: xbucketinstances
  claimNames:
    kind: BucketInstance
    plural: bucketinstances
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  name:
                    type: string
                  secretName:
                    type: string
                required:
                - name
                - secretName
            required:
            - parameters

This definition creates a new resource type BucketInstance with two parameters that can be used to allocate a resource in a provider:

apiVersion: bucket.yandexcloud.ru/v1alpha1
kind: BucketInstance
metadata:
  name: images
spec:
  parameters:
    name: images
    secretName: imagesSecret
  compositionRef:
    name: production
  writeConnectionSecretToRef:
    name: bucketSecret

But in order for this to work correctly, you need to determine how the resource is created (what resources it will consist of, and what configuration fields the parameters will be applied to):

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: example
  labels:
    crossplane.io/xrd: xbucketinstances.bucket.yandexcloud.ru
    provider: provider-yandex-cloud
spec:
  writeConnectionSecretsToNamespace: crossplane-system
  compositeTypeRef:
    apiVersion: database.example.org/v1alpha1
    kind: XBucketInstance
  resources:
  - name: serviceAccountKey
	  base:
      apiVersion: iam.yandex-cloud.jet.crossplane.io/v1alpha1
      kind: ServiceAccountStaticAccessKey
      metadata:
        name: bucket-creds
      spec:
        forProvider:
          description: "static access key for object storage"
          serviceAccountIdRef:
            name: crossplane
        providerConfigRef:
          name: provider-yandex-cloud
        writeConnectionSecretToRef:
          name: bucket-creds
          namespace: crossplane-system
    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.name
      toFieldPath: metadata.name
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.secretName
      toFieldPath: spec.writeConnectionSecretToRef.name
  - name: bucket
    base:
      apiVersion: storage.yandex-cloud.jet.crossplane.io/v1alpha1
      kind: Bucket
      metadata:
        name: example-bucket
      spec:
        forProvider:
          accessKeyRef:
            name: bucket-creds
          secretKeySecretRef:
            name: bucket-creds
            namespace: crossplane-system
            key: attribute.secret_key
          bucket: "example-test-bucket"
          acl: "public-read"
        providerConfigRef:
    			name: provider-yandex-cloud
    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.name
      toFieldPath: metadata.name
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.name
      toFieldPath: spec.forProvider.bucket
		- type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.secretName
      toFieldPath: spec.forProvider.accessKeyRef.name
		- type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.secretName
      toFieldPath: spec.forProvider.secretKeySecretRef.name

The definition includes an enumeration of objects made up of the original configuration (in base) and replacement of field values ​​in patches (the paths in the configuration of the Claim object are specified in fromFieldPath, and the object being created is specified in toFieldPath).

Now that we have dealt with creating simple and defining combined resources, we can move on to creating our own provider. Let’s take the repository as a basis https://github.com/crossplane/provider-template. The project registers the MyType resource type in apiVersion by default: sample.template.crossplane.io/v1alpha1.

To create a new type, you can use the hack/helpers/addtype.sh script, when you call it, you need to define environment variables:

  • APIVERSION – API version (v1alpha1 by default)

  • GROUP – group for apiVersion

  • KIND – resource type

  • PROVIDER – provider name

Calling the script will create API description structures and an empty controller. Structs are defined in apis/sample (or another group). groupversion_info.go defines the API metadata (group and version) and prepares the SchemeGroupVersion (metadata) and SchemeBuilder objects – which will be used to register the schema. The schema itself is defined using the kubebuilder in _types.go (for example, mytype_types.go). We discussed the use of the Operator SDK in more detail in this article.

The general provider configuration (and resource registration to define the configuration) is defined in apis/v1alpha1 (providerconfig_types.go, providerconfigusage_types.go, storeconfig_types.go). In these files, the provider configuration structure can be changed.

Because the operator must monitor the external resource, status.conditions.type must specify the state of synchronization (Ready can be set when the operator has connected to the server, Synced when the state of the external resource is consistent with the expected configuration).

The controller itself is defined in internal/controller//.go (for example, internal/controller/mytype/mytype.go). The controller defines several methods:

  • Setup – registers a controller for resource management (used by ControllerManager)

  • Connect – creates a managed.ExternalClient from a connection to an external resource (such as a database or cloud management API)

  • Observe – monitoring the state of an external resource, returns a managed.ExternalObservation structure with the following fields:

    • ResourceExists – the resource exists in the external system

    • ResourceUpToDate – external resource up to date

    • ConnectionDetails – structure managed.ConnectionDetails describing connection details to an external resource

  • Create – creation of an external resource (returns managed.ExternalCreation with ConnectionDetails – connection details to the created resource)

  • Update – updating the configuration of an external resource (returns managed.ExternalUpdate with ConnectionDetails – connection details to the created resource)

  • Delete – deletion of an external resource (described by the structure of the type we define, for example v1alpha1.MyType)

At a minimum, the Setup method must be defined and is used to register the controller via the ControllerManager (the code can be borrowed from the controller example). A controller is created for each resource type that is defined (for configuration, the resource is defined in internal/controller/config/config.go).

In addition to the API and the controller, for the correct assembly of the provider, you need to define metadata in the package:

  • crossplane.yaml contains metadata about the author, source code, license and provider description, as well as the location of the Docker container image in spec.controller.image

  • crds/*.yaml – definitions of created resources (CustomResourceDefinition) – configuration of the provider and additional types of resources

An example of a CRD yaml file to describe the database definition can be found in this file. The definition of the database specification is done in file. The controller implementation for managing databases is available at reconciler.go.

To create a package (provider image), you can use the kubectl extension:

kubectl crossplane build provider

As a result, the container image will be assembled and loaded into an OCI-compatible registry (for example, Docker Hub) and later it can be installed in a Kubernetes cluster with the creation of resource types defined in the provider. More details can be found in documentation creating providers.

Thus, using Crossplane, specialized operators (providers) can be created that are focused on managing external resources (both cloud services and any other systems for which it is possible to obtain the current state).

The number of Ansible modules is large and their capabilities are diverse. But sometimes even that is not enough. And in this case, we can develop our own module, use it in our work and maybe even share it with the community. You can learn how to do this in the free Ansible Module Development tutorial from my colleagues at OTUS. Learn more about the course “Infrastructure as a code” and register for a free lesson at the link below.

Learn more about the course.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *