cross plane. Where Kubernetes and clouds meet
The Kubernetes container orchestration system has become the de facto gold standard in DevOps, as a versatile tool for managing configuration and automatically maintaining the expected state of the system. But, even when using Managed Kubernetes solutions (for example, Amazon EX, GKE or decisions from Yandex Clouds or Mail.Ru), some provider services are hosted outside of Kubernetes (for example, S3-compatible storage, queue brokers, or databases) and would like to be able to manage using a common configuration. In the article we will discuss the possibilities of using the Crossplane project (included in Incubating CNCF), and also look at the provider code using the example of managing MySQL databases.
cross plane installed as an operator in a Kubernetes cluster and can be managed through the kubectl extension (kubectl crossplane subcommand) or through creating a Custom Resource on the cluster. The provider of additional resources and their preconfigured combinations (which are called CombinedResource or XR) are providers (Provider), which is a controller for registering resource types and implementing the logic for creating, updating, coordinating the state with an external resource and deleting them. The provider can be installed via kubectl crossplane install provider crossplane/provider-
ClusterPackageInstall. The provider can be made or independently (you can use the template https://github.com/crossplane/provider-template) or generated from the Terraform provider using Terrajet (for example, https://github.com/yandex-cloud/provider-jet-yc)
For example, the following providers are available out of the box:
provider-aws– used to manage Amazon cloud resources
provider-gcp– for Google Cloud Platform
provider-azure– for Microsoft Azure resources
provider-digitalocean– for Digital Ocean resources
provider-alibaba– for Alibaba Cloud
provider-ibm-cloud– for IBM Cloud
provider-jet-yc– to manage Yandex Cloud resources
provider-cloudflare– cloudflare management
provider-linode– Linode provider resource management
provider-terraform– resource management using terraform
provider-kubernetes– remote Kubernetes cluster management
provider-sql– used to manage databases and data schema in relational DBMS
provider-kafka– kafka resource management
provider-influxdb– influxdb resource management
provider-rook– resource management of the rook operator (used to deploy distributed storage, for example based on ceph)
Consider using a provider using the example of Yandex Cloud (now this is the most accessible resource in the face of problems with paying for foreign services). Let’s start by installing the Crossplane operator in the cluster (we will use minikube, helm must also be installed in advance):
minikube start kubectl create namespace crossplane-system helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update helm install crossplane --namespace crossplane-system crossplane-stable/crossplane
Let’s create a key to access Yandex Cloud (after creating a service account):
yc iam service-account create --name crossplane yc resource-manager folder add-access-binding <folder_id> --service-account-name crossplane --role admin yc iam service-account get crossplane yc iam key create --service-account-id service_account_id --output key.json kubectl create secret generic yc-creds -n "crossplane-system" --from-file=credentials=./key.json
And we will create a resource for connecting the yandex-cloud provider and a configuration that determines the location of the secret for accessing the cloud:
apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-yandex-cloud spec: package: "cr.yandex/crp0kch415f0lke009ft/crossplane/provider-jet-yc:v0.1.28" --- apiVersion: yandex-cloud.jet.crossplane.io/v1alpha1 kind: ProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: name: yc-creds namespace: crossplane-system key: credentials
And check the availability of the provider:
kubectl get provider
NAME INSTALLED HEALTHY PACKAGE AGE provider-yandex-cloud True True cr.yandex/crp0kch415f0lke009ft/crossplane/provider-jet-yc:v0.1.28 28m
We can also verify that the provider has registered additional resource types (output snippet):
buckets storage.yandex-cloud.jet.crossplane.io/v1alpha1 false Bucket objects storage.yandex-cloud.jet.crossplane.io/v1alpha1 false Object defaultsecuritygroups vpc.yandex-cloud.jet.crossplane.io/v1alpha1 false DefaultSecurityGroup networks vpc.yandex-cloud.jet.crossplane.io/v1alpha1 false Network
When creating a provider, you can use Vault to access secrets. Now we can use the created resources to manage the cloud, for example, we can create a bucket in S3:
apiVersion: iam.yandex-cloud.jet.crossplane.io/v1alpha1 kind: ServiceAccountStaticAccessKey metadata: name: bucket-creds spec: forProvider: description: "static access key for object storage" serviceAccountIdRef: name: crossplane providerConfigRef: name: provider-yandex-cloud writeConnectionSecretToRef: name: bucket-creds namespace: crossplane-system --- apiVersion: storage.yandex-cloud.jet.crossplane.io/v1alpha1 kind: Bucket metadata: name: example-bucket spec: forProvider: accessKeyRef: name: bucket-creds secretKeySecretRef: name: bucket-creds namespace: crossplane-system key: attribute.secret_key bucket: "example-test-bucket" acl: "public-read" providerConfigRef: name: provider-yandex-cloud
Using the resource example, you can see that the specification is defined by the following keys (inside the spec):
forProvider– this object is passed to the provider unchanged and is intended to determine the configuration of a particular object
providerConfigRef.name– link to the corresponding provider (by name), if not specified, the default provider will be used
publishConnectionDetailsTo– configuration to save connection information
writeConnectionSecretToRef– the name of the secret where the configuration can be written after the object is created (for example, generated access keys)
It is important to note that the provider not only creates resources, but also keeps them up to date, so if the bucket is deleted through the cloud web interface, it will be recreated. Also, in some cases, it is possible to apply a configuration update over an existing object. When the Bucket is deleted, the object will also be deleted from the cloud (unless Orphan was specified in spec.deletionPolicy). To get detailed information about the spec structure, you can use the explain mechanism in kubernetes:
kubectl explain Bucket.spec
Similarly, other types of objects are manipulated (including those with other providers), including virtual machines, message queues, and other managed services (databases, network balancers, etc.). You can also define your own compositions of objects (a kind of “macro”) for uniform deployment. To do this, you need to create a CompositeResourceDefinition resource in apiextensions.crossplane.io/v1, specify in the specification group (group for composition, will be used with versions.name as apiVersion), names.kind (resource definition name), names.plural (plural for resource definition name), claimName.kind (name of the resource to be created), claimName.plurlal (plural for the resource to be created). The versions enumerates the list of resource versions to be defined with the structure:
name – name (for example, v1alpha1, full apiVersion name will be collected from group/name)
served=true – if the resource version is available via the API
referenceable=true – if it is possible to instantiate (should be set on only one entry)
schema.openAPIV3Schema – schema definition (define spec structures for a resource)
For example, we can define a resource for creating a bucket, for which two parameters will be passed: the name of the secret to store access keys and the name of the bucket:
apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: name: xpostgresqlinstances.database.example.org spec: group: bucket.yandexcloud.ru names: kind: XBucketInstance plural: xbucketinstances claimNames: kind: BucketInstance plural: bucketinstances versions: - name: v1alpha1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: parameters: type: object properties: name: type: string secretName: type: string required: - name - secretName required: - parameters
This definition creates a new resource type BucketInstance with two parameters that can be used to allocate a resource in a provider:
apiVersion: bucket.yandexcloud.ru/v1alpha1 kind: BucketInstance metadata: name: images spec: parameters: name: images secretName: imagesSecret compositionRef: name: production writeConnectionSecretToRef: name: bucketSecret
But in order for this to work correctly, you need to determine how the resource is created (what resources it will consist of, and what configuration fields the parameters will be applied to):
apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: example labels: crossplane.io/xrd: xbucketinstances.bucket.yandexcloud.ru provider: provider-yandex-cloud spec: writeConnectionSecretsToNamespace: crossplane-system compositeTypeRef: apiVersion: database.example.org/v1alpha1 kind: XBucketInstance resources: - name: serviceAccountKey base: apiVersion: iam.yandex-cloud.jet.crossplane.io/v1alpha1 kind: ServiceAccountStaticAccessKey metadata: name: bucket-creds spec: forProvider: description: "static access key for object storage" serviceAccountIdRef: name: crossplane providerConfigRef: name: provider-yandex-cloud writeConnectionSecretToRef: name: bucket-creds namespace: crossplane-system patches: - type: FromCompositeFieldPath fromFieldPath: spec.parameters.name toFieldPath: metadata.name - type: FromCompositeFieldPath fromFieldPath: spec.parameters.secretName toFieldPath: spec.writeConnectionSecretToRef.name - name: bucket base: apiVersion: storage.yandex-cloud.jet.crossplane.io/v1alpha1 kind: Bucket metadata: name: example-bucket spec: forProvider: accessKeyRef: name: bucket-creds secretKeySecretRef: name: bucket-creds namespace: crossplane-system key: attribute.secret_key bucket: "example-test-bucket" acl: "public-read" providerConfigRef: name: provider-yandex-cloud patches: - type: FromCompositeFieldPath fromFieldPath: spec.parameters.name toFieldPath: metadata.name - type: FromCompositeFieldPath fromFieldPath: spec.parameters.name toFieldPath: spec.forProvider.bucket - type: FromCompositeFieldPath fromFieldPath: spec.parameters.secretName toFieldPath: spec.forProvider.accessKeyRef.name - type: FromCompositeFieldPath fromFieldPath: spec.parameters.secretName toFieldPath: spec.forProvider.secretKeySecretRef.name
The definition includes an enumeration of objects made up of the original configuration (in base) and replacement of field values in patches (the paths in the configuration of the Claim object are specified in fromFieldPath, and the object being created is specified in toFieldPath).
Now that we have dealt with creating simple and defining combined resources, we can move on to creating our own provider. Let’s take the repository as a basis https://github.com/crossplane/provider-template. The project registers the MyType resource type in apiVersion by default: sample.template.crossplane.io/v1alpha1.
To create a new type, you can use the hack/helpers/addtype.sh script, when you call it, you need to define environment variables:
APIVERSION – API version (v1alpha1 by default)
GROUP – group for apiVersion
KIND – resource type
PROVIDER – provider name
Calling the script will create API description structures and an empty controller. Structs are defined in apis/sample (or another group). groupversion_info.go defines the API metadata (group and version) and prepares the SchemeGroupVersion (metadata) and SchemeBuilder objects – which will be used to register the schema. The schema itself is defined using the kubebuilder in
The general provider configuration (and resource registration to define the configuration) is defined in apis/v1alpha1 (providerconfig_types.go, providerconfigusage_types.go, storeconfig_types.go). In these files, the provider configuration structure can be changed.
Because the operator must monitor the external resource, status.conditions.type must specify the state of synchronization (Ready can be set when the operator has connected to the server, Synced when the state of the external resource is consistent with the expected configuration).
The controller itself is defined in internal/controller/
Setup– registers a controller for resource management (used by ControllerManager)
Connect– creates a managed.ExternalClient from a connection to an external resource (such as a database or cloud management API)
Observe– monitoring the state of an external resource, returns a managed.ExternalObservation structure with the following fields:
ResourceExists– the resource exists in the external system
ResourceUpToDate– external resource up to date
ConnectionDetails– structure managed.ConnectionDetails describing connection details to an external resource
Create– creation of an external resource (returns
managed.ExternalCreationwith ConnectionDetails – connection details to the created resource)
Update– updating the configuration of an external resource (returns managed.ExternalUpdate with ConnectionDetails – connection details to the created resource)
Delete– deletion of an external resource (described by the structure of the type we define, for example v1alpha1.MyType)
At a minimum, the Setup method must be defined and is used to register the controller via the ControllerManager (the code can be borrowed from the controller example). A controller is created for each resource type that is defined (for configuration, the resource is defined in internal/controller/config/config.go).
In addition to the API and the controller, for the correct assembly of the provider, you need to define metadata in the package:
crossplane.yamlcontains metadata about the author, source code, license and provider description, as well as the location of the Docker container image in spec.controller.image
crds/*.yaml– definitions of created resources (CustomResourceDefinition) – configuration of the provider and additional types of resources
An example of a CRD yaml file to describe the database definition can be found in this file. The definition of the database specification is done in file. The controller implementation for managing databases is available at reconciler.go.
To create a package (provider image), you can use the kubectl extension:
kubectl crossplane build provider
As a result, the container image will be assembled and loaded into an OCI-compatible registry (for example, Docker Hub) and later it can be installed in a Kubernetes cluster with the creation of resource types defined in the provider. More details can be found in documentation creating providers.
Thus, using Crossplane, specialized operators (providers) can be created that are focused on managing external resources (both cloud services and any other systems for which it is possible to obtain the current state).
The number of Ansible modules is large and their capabilities are diverse. But sometimes even that is not enough. And in this case, we can develop our own module, use it in our work and maybe even share it with the community. You can learn how to do this in the free Ansible Module Development tutorial from my colleagues at OTUS. Learn more about the course “Infrastructure as a code” and register for a free lesson at the link below.