talking about the KWOK tool

Have you ever wondered how:

• set up a cluster of thousands of nodes in seconds

• simulate real nodes with low resource consumption

• test the K8s controller at scale, but without spending a lot of infrastructure

If you answered “Yes” to most of the questions, then you will probably be interested in learning about KWOK, a tool that allows you to create a cluster of thousands of nodes.

What is it all about?

KWOK stands for Kubernetes WithOut Kubelet. KWOK includes two utilities:

• kwok – responsible for modeling the life cycle of nodes, pods and other resources of the K8s API.

• kwokctl is a command line tool designed to make it easy to create and manage clusters with nodes created by kwok

Why Use KWOK?

• Speed: you can create and remove clusters and nodes almost instantly without waiting for a full load

• Compatibility: KWOK works with any K8s API compatible tools or clients: kubectl, helm, kui, etc.

• Portability: KWOK has no special requirements for hardware or software. You can run it using prebuilt Docker images, Nerdctl, or binaries.

• Flexibility: You can customize node types, labels, clusters, and more, as well as customize pod behaviors to test different scenarios and edge cases.

• Performance: You can create thousands of nodes without significant CPU or RAM consumption.

Why use KWOK

• For learning: You can use KWOK to learn about the features and functions of Kubernetes without worrying about wasting resources or any other issues.

• For Development: KWOK is good for creating new Kubernetes features or tools without needing to use a real cluster.

• For testing. You can:

  • Measure how well your application scales with different number of nodes and pods.

  • Increase the load on the cluster by creating many pods or services with different resource requests or limits.

  • Simulate node failures or network partitions by changing node conditions or randomly deleting nodes.

  • Check how your controller interacts with other K8s components, use different versions of the API.

What are the restrictions?

KWOK is not intended to be a complete replacement for other tools: it has limitations.

• Functionality: KWOK is not a kubelet, behavior may differ when managing pod lifecycles, mounting volumes and device plug-ins. Its main function is to simulate node and pod updates.

• Accuracy: KWOK does not accurately represent the performance or behavior of real nodes under different workloads or environments. Instead, it approximates some behaviors using simple formulas.

• Security: KWOK does not enforce any security policies or mechanisms.

Getting started with KWOK

Starting the cluster

docker run --rm -it -p 8080:8080 registry.k8s.io/kwok/cluster:v1.26.0

Creating cluster                                                                                                   cluster=kwok
Starting cluster                                                                                                   cluster=kwok
Cluster is created                                                                                      cluster=kwok elapsed=1s
You can now use your cluster with:

    kubectl config use-context kwok-kwok

Thanks for using kwok!
###############################################################################
> kubectl -s :8080 version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:51:45Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
###############################################################################
# The following kubeconfig can be used to connect to the Kubernetes API server
apiVersion: v1
clusters:
- cluster:
    server: http://127.0.0.1:8080
  name: kwok
contexts:
- context:
    cluster: kwok
  name: kwok
current-context: kwok
kind: Config
preferences: {}
users: null
###############################################################################
> kubectl -s :8080 get ns
NAME              STATUS   AGE
default           Active   0s
kube-node-lease   Active   1s
kube-public       Active   1s
kube-system       Active   1s
###############################################################################
# The above example works if your host's port is the same as the container's,
# otherwise, change it to your host's port
Starting to serve on [::]:8080

How to install kwokctl and kwok:

1. On Linux/MacOS: using the brew command

brew install kwok

2. For binary versions:

# KWOK repository
KWOK_REPO=kubernetes-sigs/kwok
# Get latest
KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name')

wget -O kwokctl -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwokctl-$(go env GOOS)-$(go env GOARCH)"
chmod +x kwokctl
sudo mv kwokctl /usr/local/bin/kwokctl

wget -O kwok -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwok-$(go env GOOS)-$(go env GOARCH)"
chmod +x kwok
sudo mv kwok /usr/local/bin/kwok

How to deploy kwok in a cluster:

1. Prepare variables:

# Temporary directory
KWOK_WORK_DIR=$(mktemp -d)
# KWOK repository
KWOK_REPO=kubernetes-sigs/kwok
# Get latest
KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name')

2. Create a customization template YAML for the previously defined temporary directory:

cat <<EOF > "${KWOK_WORK_DIR}/kustomization.yaml"
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
  - name: registry.k8s.io/kwok/kwok
    newTag: "${KWOK_LATEST_RELEASE}"
resources:
  - "https://github.com/${KWOK_REPO}/kustomize/kwok?ref=${KWOK_LATEST_RELEASE}"
EOF

Then apply the customization template.

kubectl kustomize "${KWOK_WORK_DIR}" > "${KWOK_WORK_DIR}/kwok.yaml"

3. Now you can deploy kwok

kubectl apply -f "${KWOK_WORK_DIR}/kwok.yaml"

After that, you can start the cluster on the local computer and monitor its status:

kwok \
  --kubeconfig=~/.kube/config \
  --manage-all-nodes=false \
  --manage-nodes-with-annotation-selector=kwok.x-k8s.io/node=fake \
  --manage-nodes-with-label-selector= \
  --disregard-status-with-annotation-selector=kwok.x-k8s.io/status=custom \
  --disregard-status-with-label-selector= \
  --cidr=10.0.0.1/24 \
  --node-ip=10.0.0.1

How to run kwokctl to manage mock clusters:

1. Install kwokctl

2. Create a cluster

$ kwokctl create cluster --name=kwok
kwokctl create cluster
Creating cluster "kwok-kwok"
Starting cluster "kwok-kwok"
Cluster "kwok-kwok" is ready
You can now use your cluster with:

    kubectl config use-context kwok-kwok

Thanks for using kwok!

Switching configurations

kubectl config use-context kwok-kwok

3. We transfer the cluster under the control of kwokctl

$ kwokctl get clusters
kwok

4. Delete the cluster

$ kwokctl delete cluster --name=kwok
Stopping cluster "kwok-kwok"
Deleting cluster "kwok-kwok"
Cluster "kwok-kwok" deleted

How to manage nodes and pods with kwok

1. kwok with arguments –manage-all-nodes=true

   With the –manage-all-nodes=true argument, kwok will be in charge of all nodes in the cluster and keep them running on the API server. All nodes will behave like real nodes and remain in the Ready state.

2. kwok with arguments –manage-nodes-with-annotation-selector=kwok.x-k8s.io/node=fake

   Here kwok will be responsible for all pods with an annotation kwok.x-k8s.io/node=fake. If the pod’s .spec.nodeName field has a value, kwok will leave them in the Running state.

3. With kwok, you can join an arbitrary node or nodes by simply creating a v1.Node object(s):

kubectl apply -f - <<EOF
apiVersion: v1
kind: Node
metadata:
  annotations:
    node.alpha.kubernetes.io/ttl: "0"
    kwok.x-k8s.io/node: fake
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: kwok-node-0
    kubernetes.io/os: linux
    kubernetes.io/role: agent
    node-role.kubernetes.io/agent: ""
    type: kwok
  name: kwok-node-0
spec:
  taints: # Avoid scheduling actual running pods to fake Node
    - effect: NoSchedule
      key: kwok.x-k8s.io/node
      value: fake
status:
  allocatable:
    cpu: 32
    memory: 256Gi
    pods: 110
  capacity:
    cpu: 32
    memory: 256Gi
    pods: 110
  nodeInfo:
    architecture: amd64
    bootID: ""
    containerRuntimeVersion: ""
    kernelVersion: ""
    kubeProxyVersion: fake
    kubeletVersion: fake
    machineID: ""
    operatingSystem: linux
    osImage: ""
    systemUUID: ""
  phase: Running
EOF

After the node is created, kwok will continue to run and maintain the resulting node.

$ kubectl get node -o wide
NAME          STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE    KERNEL-VERSION   CONTAINER-RUNTIME
kwok-node-0   Ready    agent   5s    fake      196.168.0.1   <none>        <unknown>   <unknown>        <unknown>

4. Creating a Pod

   Now we create some pods to see if they can join the previously created nodes

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fake-pod
  namespace: default
spec:
  replicas: 10
  selector:
    matchLabels:
      app: fake-pod
  template:
    metadata:
      labels:
        app: fake-pod
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: type
                    operator: In
                    values:
                      - kwok
      # A taints was added to an automatically created Node.
      # You can remove taints of Node or add this tolerations.
      tolerations:
        - key: "kwok.x-k8s.io/node"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: fake-container
          image: fake-image
EOF

After the pod is created, we see that all pods are hosted on nodes and have a status of Running.

$ kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP          NODE          NOMINATED NODE   READINESS GATES
fake-pod-59bb47845f-4vl9f   1/1     Running   0          5s    10.0.0.5    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-bc49m   1/1     Running   0          5s    10.0.0.4    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-cnjsv   1/1     Running   0          5s    10.0.0.7    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-g29wz   1/1     Running   0          5s    10.0.0.2    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-gxq88   1/1     Running   0          5s    10.0.0.10   kwok-node-0   <none>           <none>
fake-pod-59bb47845f-pnzmn   1/1     Running   0          5s    10.0.0.9    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-sfkk4   1/1     Running   0          5s    10.0.0.3    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-vl2z5   1/1     Running   0          5s    10.0.0.8    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-vpfhv   1/1     Running   0          5s    10.0.0.6    kwok-node-0   <none>           <none>
fake-pod-59bb47845f-wxn4b   1/1     Running   0          5s    10.0.0.1    kwok-node-0   <none>           <none>

5. Update specification

   In kwok, nodes and pods are pure API objects, so you can change their specification to simulate or test.

How to save or restore a cluster using kwokctl

1. Setting up an audit policy

cat <<EOF > audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF

2. Create a cluster

kwokctl create cluster --kube-audit-policy audit-policy.yaml

3. Getting the audit log

kwokctl logs audit

4. Audit Log Example link

How to configure kwok/kwokctl functions

To configure, you will need to create a configuration file YAML . The minimum valid configuration file looks like this:

kind: KwokConfiguration
apiVersion: kwok.x-k8s.io/v1alpha1
options:
---
kind: KwokctlConfiguration
apiVersion: kwok.x-k8s.io/v1alpha1
options:

This configuration shows that we are configuring kwok/kwokctl using version v1alpha1( apiVersion: kwok.x-k8s.io/v1alpha1).

To set the correct parameters for the cluster, you must specify the version of kwok

To use this configuration, place the content in the ~/.kwok/kwok.yaml file, or run the –config=kwok.yaml command from the same directory.

A note about CLI flags, environment variables, and configuration files

Configuration priority order for kwok:

1. flags specified on the command line

2. environment variables (prefixed with KWOK_)

3. values ​​specified in the configuration file ( –config= or ~/.kwok/kwok.yaml)

4. default values

Using kwok

When used, kwok takes its configuration from the appropriate file and ignores all other configurations.

Using kwokctl

kwokctl takes its configuration from the configuration file and passes it to kwok.

Stage Configuration

The Stage API is a kwok configuration that allows users to define and model various stages in the life cycle of Kubernetes resources: nodes and pods. Each Stage resource specifies a resourceRef field that specifies the type of resource to which it applies, and a selector field that specifies when the stage should be executed.

kind: Stage
apiVersion: kwok.x-k8s.io/v1alpha1
metadata:
  name: <string>
spec:
  resourceRef:
    apiGroup: <string>
    kind: <string>
  selector:
    matchLabels:
      <string>: <string>
    matchAnnotations:
      <string>: <string>
    matchExpressions:
      - key: <expressions-string>
        operator: <string>
        values:
          - <string>
  delay:
    durationMilliseconds: <int>
    durationFrom:
      expressionFrom: <expressions-string>
    jitterDurationMilliseconds: <int>
    jitterDurationFrom:
      expressionFrom: <expressions-string>
  next:
    statusTemplate: <string>
    finalizers:
      add:
        - value: <string>
      remove:
        - value: <string>
      empty: <bool>
    delete: <bool>

By setting the selector and next fields in the spec Stage, you can specify the conditions under which the configuration is applied, as well as the changes that will be made to the resource when the stage is applied. The next field allows users to define the new state of the resource using the statusTemplate field, modify the resource’s finalizers, or perform delete on the resource.

In addition, the delay field on the Stage resource allows users to specify a delay before applying the stage and make it float. This will help simulate real scenarios where events occur at different times.

By setting the delay, selector, and next fields, you can control when and how the stage is applied. This allows you to create complex and realistic models for testing, validation and experimentation, and gain insight into the behavior and performance of your applications and infrastructure.

Examples

1. Showing how to set up simple knots for kwok

The node-initialize stage applies to nodes that do not have any status.conditions set in their field. When applied, this step sets the status.conditions field for the host, as well as the status.addresses, status.allocatable, and status.capacity fields.

The node-heartbeat stage applies to nodes that have a Ready condition in their field. When applied, this stage maintains a field for the node.Truestatus.conditionsstatus.conditions

2. How to set up a pod

Stage pod-create-and-ready applies to pods that don’t have a status.podIP and don’t have a metadata.deletionTimestamp. When applied, this step sets the status.conditions, status.containerStatuses, status.initContainerStatuses, status.hostIP, and status.podIP fields for the host. It will also set the Phase and startTime fields to indicate that the node is up and running.

The pod-completed-for-job step applies to running nodes that do not have a metadata.deletionTimestamp and are owned by a job. When applied, this step updates the status.containerStatuses for the node by setting the ready and started fields to true and state.terminated to indicate that the node has terminated. It also sets the stage to Succeeded: it indicates that the node has completed its work.

Stage pod-delete applies to nodes with metadata.deletionTimestamp set. When applied, this step clears the metadata.finalizers field for the node, allowing it to be deleted and then the node itself.

Example link.