cross plane. Where Kubernetes and clouds meet
The Kubernetes container orchestration system has become the de facto gold standard in DevOps, as a versatile tool for managing configuration and automatically maintaining the expected state of the system. But, even when using Managed Kubernetes solutions (for example, Amazon EX, GKE or decisions from Yandex Clouds or Mail.Ru), some provider services are hosted outside of Kubernetes (for example, S3-compatible storage, queue brokers, or databases) and would like to be able to manage using a common configuration. In the article we will discuss the possibilities of using the Crossplane project (included in Incubating CNCF), and also look at the provider code using the example of managing MySQL databases.
cross plane installed as an operator in a Kubernetes cluster and can be managed through the kubectl extension (kubectl crossplane subcommand) or through creating a Custom Resource on the cluster. The provider of additional resources and their preconfigured combinations (which are called CombinedResource or XR) are providers (Provider), which is a controller for registering resource types and implementing the logic for creating, updating, coordinating the state with an external resource and deleting them. The provider can be installed via kubectl crossplane install provider crossplane/provider-ClusterPackageInstall
. The provider can be made or independently (you can use the template https://github.com/crossplane/provider-template) or generated from the Terraform provider using Terrajet (for example, https://github.com/yandex-cloud/provider-jet-yc)
For example, the following providers are available out of the box:
provider-aws
– used to manage Amazon cloud resourcesprovider-gcp
– for Google Cloud Platformprovider-azure
– for Microsoft Azure resourcesprovider-digitalocean
– for Digital Ocean resourcesprovider-alibaba
– for Alibaba Cloudprovider-ibm-cloud
– for IBM Cloudprovider-jet-yc
– to manage Yandex Cloud resourcesprovider-cloudflare
– cloudflare managementprovider-linode
– Linode provider resource managementprovider-terraform
– resource management using terraformprovider-kubernetes
– remote Kubernetes cluster managementprovider-sql
– used to manage databases and data schema in relational DBMSprovider-kafka
– kafka resource managementprovider-influxdb
– influxdb resource managementprovider-rook
– resource management of the rook operator (used to deploy distributed storage, for example based on ceph)
Consider using a provider using the example of Yandex Cloud (now this is the most accessible resource in the face of problems with paying for foreign services). Let’s start by installing the Crossplane operator in the cluster (we will use minikube, helm must also be installed in advance):
minikube start
kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane
Let’s create a key to access Yandex Cloud (after creating a service account):
yc iam service-account create --name crossplane
yc resource-manager folder add-access-binding <folder_id> --service-account-name crossplane --role admin
yc iam service-account get crossplane
yc iam key create --service-account-id service_account_id --output key.json
kubectl create secret generic yc-creds -n "crossplane-system" --from-file=credentials=./key.json
And we will create a resource for connecting the yandex-cloud provider and a configuration that determines the location of the secret for accessing the cloud:
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-yandex-cloud
spec:
package: "cr.yandex/crp0kch415f0lke009ft/crossplane/provider-jet-yc:v0.1.28"
---
apiVersion: yandex-cloud.jet.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
name: yc-creds
namespace: crossplane-system
key: credentials
And check the availability of the provider: kubectl get provider
NAME INSTALLED HEALTHY PACKAGE AGE
provider-yandex-cloud True True cr.yandex/crp0kch415f0lke009ft/crossplane/provider-jet-yc:v0.1.28 28m
We can also verify that the provider has registered additional resource types (output snippet):
buckets storage.yandex-cloud.jet.crossplane.io/v1alpha1 false Bucket
objects storage.yandex-cloud.jet.crossplane.io/v1alpha1 false Object
defaultsecuritygroups vpc.yandex-cloud.jet.crossplane.io/v1alpha1 false DefaultSecurityGroup
networks vpc.yandex-cloud.jet.crossplane.io/v1alpha1 false Network
When creating a provider, you can use Vault to access secrets. Now we can use the created resources to manage the cloud, for example, we can create a bucket in S3:
apiVersion: iam.yandex-cloud.jet.crossplane.io/v1alpha1
kind: ServiceAccountStaticAccessKey
metadata:
name: bucket-creds
spec:
forProvider:
description: "static access key for object storage"
serviceAccountIdRef:
name: crossplane
providerConfigRef:
name: provider-yandex-cloud
writeConnectionSecretToRef:
name: bucket-creds
namespace: crossplane-system
---
apiVersion: storage.yandex-cloud.jet.crossplane.io/v1alpha1
kind: Bucket
metadata:
name: example-bucket
spec:
forProvider:
accessKeyRef:
name: bucket-creds
secretKeySecretRef:
name: bucket-creds
namespace: crossplane-system
key: attribute.secret_key
bucket: "example-test-bucket"
acl: "public-read"
providerConfigRef:
name: provider-yandex-cloud
Using the resource example, you can see that the specification is defined by the following keys (inside the spec):
forProvider
– this object is passed to the provider unchanged and is intended to determine the configuration of a particular objectproviderConfigRef.name
– link to the corresponding provider (by name), if not specified, the default provider will be usedpublishConnectionDetailsTo
– configuration to save connection informationwriteConnectionSecretToRef
– the name of the secret where the configuration can be written after the object is created (for example, generated access keys)
It is important to note that the provider not only creates resources, but also keeps them up to date, so if the bucket is deleted through the cloud web interface, it will be recreated. Also, in some cases, it is possible to apply a configuration update over an existing object. When the Bucket is deleted, the object will also be deleted from the cloud (unless Orphan was specified in spec.deletionPolicy). To get detailed information about the spec structure, you can use the explain mechanism in kubernetes:
kubectl explain Bucket.spec
Similarly, other types of objects are manipulated (including those with other providers), including virtual machines, message queues, and other managed services (databases, network balancers, etc.). You can also define your own compositions of objects (a kind of “macro”) for uniform deployment. To do this, you need to create a CompositeResourceDefinition resource in apiextensions.crossplane.io/v1, specify in the specification group (group for composition, will be used with versions.name as apiVersion), names.kind (resource definition name), names.plural (plural for resource definition name), claimName.kind (name of the resource to be created), claimName.plurlal (plural for the resource to be created). The versions enumerates the list of resource versions to be defined with the structure:
name – name (for example, v1alpha1, full apiVersion name will be collected from group/name)
served=true – if the resource version is available via the API
referenceable=true – if it is possible to instantiate (should be set on only one entry)
schema.openAPIV3Schema – schema definition (define spec structures for a resource)
For example, we can define a resource for creating a bucket, for which two parameters will be passed: the name of the secret to store access keys and the name of the bucket:
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xpostgresqlinstances.database.example.org
spec:
group: bucket.yandexcloud.ru
names:
kind: XBucketInstance
plural: xbucketinstances
claimNames:
kind: BucketInstance
plural: bucketinstances
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
parameters:
type: object
properties:
name:
type: string
secretName:
type: string
required:
- name
- secretName
required:
- parameters
This definition creates a new resource type BucketInstance with two parameters that can be used to allocate a resource in a provider:
apiVersion: bucket.yandexcloud.ru/v1alpha1
kind: BucketInstance
metadata:
name: images
spec:
parameters:
name: images
secretName: imagesSecret
compositionRef:
name: production
writeConnectionSecretToRef:
name: bucketSecret
But in order for this to work correctly, you need to determine how the resource is created (what resources it will consist of, and what configuration fields the parameters will be applied to):
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: example
labels:
crossplane.io/xrd: xbucketinstances.bucket.yandexcloud.ru
provider: provider-yandex-cloud
spec:
writeConnectionSecretsToNamespace: crossplane-system
compositeTypeRef:
apiVersion: database.example.org/v1alpha1
kind: XBucketInstance
resources:
- name: serviceAccountKey
base:
apiVersion: iam.yandex-cloud.jet.crossplane.io/v1alpha1
kind: ServiceAccountStaticAccessKey
metadata:
name: bucket-creds
spec:
forProvider:
description: "static access key for object storage"
serviceAccountIdRef:
name: crossplane
providerConfigRef:
name: provider-yandex-cloud
writeConnectionSecretToRef:
name: bucket-creds
namespace: crossplane-system
patches:
- type: FromCompositeFieldPath
fromFieldPath: spec.parameters.name
toFieldPath: metadata.name
- type: FromCompositeFieldPath
fromFieldPath: spec.parameters.secretName
toFieldPath: spec.writeConnectionSecretToRef.name
- name: bucket
base:
apiVersion: storage.yandex-cloud.jet.crossplane.io/v1alpha1
kind: Bucket
metadata:
name: example-bucket
spec:
forProvider:
accessKeyRef:
name: bucket-creds
secretKeySecretRef:
name: bucket-creds
namespace: crossplane-system
key: attribute.secret_key
bucket: "example-test-bucket"
acl: "public-read"
providerConfigRef:
name: provider-yandex-cloud
patches:
- type: FromCompositeFieldPath
fromFieldPath: spec.parameters.name
toFieldPath: metadata.name
- type: FromCompositeFieldPath
fromFieldPath: spec.parameters.name
toFieldPath: spec.forProvider.bucket
- type: FromCompositeFieldPath
fromFieldPath: spec.parameters.secretName
toFieldPath: spec.forProvider.accessKeyRef.name
- type: FromCompositeFieldPath
fromFieldPath: spec.parameters.secretName
toFieldPath: spec.forProvider.secretKeySecretRef.name
The definition includes an enumeration of objects made up of the original configuration (in base) and replacement of field values in patches (the paths in the configuration of the Claim object are specified in fromFieldPath, and the object being created is specified in toFieldPath).
Now that we have dealt with creating simple and defining combined resources, we can move on to creating our own provider. Let’s take the repository as a basis https://github.com/crossplane/provider-template. The project registers the MyType resource type in apiVersion by default: sample.template.crossplane.io/v1alpha1.
To create a new type, you can use the hack/helpers/addtype.sh script, when you call it, you need to define environment variables:
APIVERSION – API version (v1alpha1 by default)
GROUP – group for apiVersion
KIND – resource type
PROVIDER – provider name
Calling the script will create API description structures and an empty controller. Structs are defined in apis/sample (or another group). groupversion_info.go defines the API metadata (group and version) and prepares the SchemeGroupVersion (metadata) and SchemeBuilder objects – which will be used to register the schema. The schema itself is defined using the kubebuilder in
The general provider configuration (and resource registration to define the configuration) is defined in apis/v1alpha1 (providerconfig_types.go, providerconfigusage_types.go, storeconfig_types.go). In these files, the provider configuration structure can be changed.
Because the operator must monitor the external resource, status.conditions.type must specify the state of synchronization (Ready can be set when the operator has connected to the server, Synced when the state of the external resource is consistent with the expected configuration).
The controller itself is defined in internal/controller/
Setup
– registers a controller for resource management (used by ControllerManager)Connect
– creates a managed.ExternalClient from a connection to an external resource (such as a database or cloud management API)Observe
– monitoring the state of an external resource, returns a managed.ExternalObservation structure with the following fields:ResourceExists
– the resource exists in the external systemResourceUpToDate
– external resource up to dateConnectionDetails
– structure managed.ConnectionDetails describing connection details to an external resource
Create
– creation of an external resource (returnsmanaged.ExternalCreation
with ConnectionDetails – connection details to the created resource)Update
– updating the configuration of an external resource (returns managed.ExternalUpdate with ConnectionDetails – connection details to the created resource)Delete
– deletion of an external resource (described by the structure of the type we define, for example v1alpha1.MyType)
At a minimum, the Setup method must be defined and is used to register the controller via the ControllerManager (the code can be borrowed from the controller example). A controller is created for each resource type that is defined (for configuration, the resource is defined in internal/controller/config/config.go).
In addition to the API and the controller, for the correct assembly of the provider, you need to define metadata in the package:
crossplane.yaml
contains metadata about the author, source code, license and provider description, as well as the location of the Docker container image in spec.controller.imagecrds/*.yaml
– definitions of created resources (CustomResourceDefinition) – configuration of the provider and additional types of resources
An example of a CRD yaml file to describe the database definition can be found in this file. The definition of the database specification is done in file. The controller implementation for managing databases is available at reconciler.go.
To create a package (provider image), you can use the kubectl extension:
kubectl crossplane build provider
As a result, the container image will be assembled and loaded into an OCI-compatible registry (for example, Docker Hub) and later it can be installed in a Kubernetes cluster with the creation of resource types defined in the provider. More details can be found in documentation creating providers.
Thus, using Crossplane, specialized operators (providers) can be created that are focused on managing external resources (both cloud services and any other systems for which it is possible to obtain the current state).
The number of Ansible modules is large and their capabilities are diverse. But sometimes even that is not enough. And in this case, we can develop our own module, use it in our work and maybe even share it with the community. You can learn how to do this in the free Ansible Module Development tutorial from my colleagues at OTUS. Learn more about the course “Infrastructure as a code” and register for a free lesson at the link below.