talking about the KWOK tool

Have you ever wondered how:
set up a cluster of thousands of nodes in seconds
simulate real nodes with low resource consumption
test the K8s controller at scale, but without spending a lot of infrastructure
If you answered “Yes” to most of the questions, then you will probably be interested in learning about KWOK, a tool that allows you to create a cluster of thousands of nodes.
What is it all about?
KWOK stands for Kubernetes WithOut Kubelet. KWOK includes two utilities:
kwok – responsible for modeling the life cycle of nodes, pods and other resources of the K8s API.
kwokctl is a command line tool designed to make it easy to create and manage clusters with nodes created by kwok
Why Use KWOK?
Speed: you can create and remove clusters and nodes almost instantly without waiting for a full load
Compatibility: KWOK works with any K8s API compatible tools or clients: kubectl, helm, kui, etc.
Portability: KWOK has no special requirements for hardware or software. You can run it using prebuilt Docker images, Nerdctl, or binaries.
Flexibility: You can customize node types, labels, clusters, and more, as well as customize pod behaviors to test different scenarios and edge cases.
Performance: You can create thousands of nodes without significant CPU or RAM consumption.
Why use KWOK
For learning: You can use KWOK to learn about the features and functions of Kubernetes without worrying about wasting resources or any other issues.
For Development: KWOK is good for creating new Kubernetes features or tools without needing to use a real cluster.
For testing. You can:
Measure how well your application scales with different number of nodes and pods.
Increase the load on the cluster by creating many pods or services with different resource requests or limits.
Simulate node failures or network partitions by changing node conditions or randomly deleting nodes.
Check how your controller interacts with other K8s components, use different versions of the API.
What are the restrictions?
KWOK is not intended to be a complete replacement for other tools: it has limitations.
Functionality: KWOK is not a kubelet, behavior may differ when managing pod lifecycles, mounting volumes and device plug-ins. Its main function is to simulate node and pod updates.
Accuracy: KWOK does not accurately represent the performance or behavior of real nodes under different workloads or environments. Instead, it approximates some behaviors using simple formulas.
Security: KWOK does not enforce any security policies or mechanisms.
Getting started with KWOK
Starting the cluster
docker run --rm -it -p 8080:8080 registry.k8s.io/kwok/cluster:v1.26.0
Creating cluster cluster=kwok
Starting cluster cluster=kwok
Cluster is created cluster=kwok elapsed=1s
You can now use your cluster with:
kubectl config use-context kwok-kwok
Thanks for using kwok!
###############################################################################
> kubectl -s :8080 version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:51:45Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
###############################################################################
# The following kubeconfig can be used to connect to the Kubernetes API server
apiVersion: v1
clusters:
- cluster:
server: http://127.0.0.1:8080
name: kwok
contexts:
- context:
cluster: kwok
name: kwok
current-context: kwok
kind: Config
preferences: {}
users: null
###############################################################################
> kubectl -s :8080 get ns
NAME STATUS AGE
default Active 0s
kube-node-lease Active 1s
kube-public Active 1s
kube-system Active 1s
###############################################################################
# The above example works if your host's port is the same as the container's,
# otherwise, change it to your host's port
Starting to serve on [::]:8080
How to install kwokctl and kwok:
On Linux/MacOS: using the brew command
brew install kwok
For binary versions:
# KWOK repository
KWOK_REPO=kubernetes-sigs/kwok
# Get latest
KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name')
wget -O kwokctl -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwokctl-$(go env GOOS)-$(go env GOARCH)"
chmod +x kwokctl
sudo mv kwokctl /usr/local/bin/kwokctl
wget -O kwok -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwok-$(go env GOOS)-$(go env GOARCH)"
chmod +x kwok
sudo mv kwok /usr/local/bin/kwok
How to deploy kwok in a cluster:
Prepare variables:
# Temporary directory
KWOK_WORK_DIR=$(mktemp -d)
# KWOK repository
KWOK_REPO=kubernetes-sigs/kwok
# Get latest
KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name')
Create a customization template YAML for the previously defined temporary directory:
cat <<EOF > "${KWOK_WORK_DIR}/kustomization.yaml"
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: registry.k8s.io/kwok/kwok
newTag: "${KWOK_LATEST_RELEASE}"
resources:
- "https://github.com/${KWOK_REPO}/kustomize/kwok?ref=${KWOK_LATEST_RELEASE}"
EOF
Then apply the customization template.
kubectl kustomize "${KWOK_WORK_DIR}" > "${KWOK_WORK_DIR}/kwok.yaml"
Now you can deploy kwok
kubectl apply -f "${KWOK_WORK_DIR}/kwok.yaml"
After that, you can start the cluster on the local computer and monitor its status:
kwok \
--kubeconfig=~/.kube/config \
--manage-all-nodes=false \
--manage-nodes-with-annotation-selector=kwok.x-k8s.io/node=fake \
--manage-nodes-with-label-selector= \
--disregard-status-with-annotation-selector=kwok.x-k8s.io/status=custom \
--disregard-status-with-label-selector= \
--cidr=10.0.0.1/24 \
--node-ip=10.0.0.1
How to run kwokctl to manage mock clusters:
Install kwokctl
Create a cluster
$ kwokctl create cluster --name=kwok
kwokctl create cluster
Creating cluster "kwok-kwok"
Starting cluster "kwok-kwok"
Cluster "kwok-kwok" is ready
You can now use your cluster with:
kubectl config use-context kwok-kwok
Thanks for using kwok!
Switching configurations
kubectl config use-context kwok-kwok
We transfer the cluster under the control of kwokctl
$ kwokctl get clusters
kwok
Delete the cluster
$ kwokctl delete cluster --name=kwok
Stopping cluster "kwok-kwok"
Deleting cluster "kwok-kwok"
Cluster "kwok-kwok" deleted
How to manage nodes and pods with kwok
kwok with arguments –manage-all-nodes=true
With the –manage-all-nodes=true argument, kwok will be in charge of all nodes in the cluster and keep them running on the API server. All nodes will behave like real nodes and remain in the Ready state.
kwok with arguments –manage-nodes-with-annotation-selector=kwok.x-k8s.io/node=fake
Here kwok will be responsible for all pods with an annotation kwok.x-k8s.io/node=fake. If the pod’s .spec.nodeName field has a value, kwok will leave them in the Running state.
With kwok, you can join an arbitrary node or nodes by simply creating a v1.Node object(s):
kubectl apply -f - <<EOF
apiVersion: v1
kind: Node
metadata:
annotations:
node.alpha.kubernetes.io/ttl: "0"
kwok.x-k8s.io/node: fake
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: kwok-node-0
kubernetes.io/os: linux
kubernetes.io/role: agent
node-role.kubernetes.io/agent: ""
type: kwok
name: kwok-node-0
spec:
taints: # Avoid scheduling actual running pods to fake Node
- effect: NoSchedule
key: kwok.x-k8s.io/node
value: fake
status:
allocatable:
cpu: 32
memory: 256Gi
pods: 110
capacity:
cpu: 32
memory: 256Gi
pods: 110
nodeInfo:
architecture: amd64
bootID: ""
containerRuntimeVersion: ""
kernelVersion: ""
kubeProxyVersion: fake
kubeletVersion: fake
machineID: ""
operatingSystem: linux
osImage: ""
systemUUID: ""
phase: Running
EOF
After the node is created, kwok will continue to run and maintain the resulting node.
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kwok-node-0 Ready agent 5s fake 196.168.0.1 <none> <unknown> <unknown> <unknown>
Creating a Pod
Now we create some pods to see if they can join the previously created nodes
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: fake-pod
namespace: default
spec:
replicas: 10
selector:
matchLabels:
app: fake-pod
template:
metadata:
labels:
app: fake-pod
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: In
values:
- kwok
# A taints was added to an automatically created Node.
# You can remove taints of Node or add this tolerations.
tolerations:
- key: "kwok.x-k8s.io/node"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: fake-container
image: fake-image
EOF
After the pod is created, we see that all pods are hosted on nodes and have a status of Running.
$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fake-pod-59bb47845f-4vl9f 1/1 Running 0 5s 10.0.0.5 kwok-node-0 <none> <none>
fake-pod-59bb47845f-bc49m 1/1 Running 0 5s 10.0.0.4 kwok-node-0 <none> <none>
fake-pod-59bb47845f-cnjsv 1/1 Running 0 5s 10.0.0.7 kwok-node-0 <none> <none>
fake-pod-59bb47845f-g29wz 1/1 Running 0 5s 10.0.0.2 kwok-node-0 <none> <none>
fake-pod-59bb47845f-gxq88 1/1 Running 0 5s 10.0.0.10 kwok-node-0 <none> <none>
fake-pod-59bb47845f-pnzmn 1/1 Running 0 5s 10.0.0.9 kwok-node-0 <none> <none>
fake-pod-59bb47845f-sfkk4 1/1 Running 0 5s 10.0.0.3 kwok-node-0 <none> <none>
fake-pod-59bb47845f-vl2z5 1/1 Running 0 5s 10.0.0.8 kwok-node-0 <none> <none>
fake-pod-59bb47845f-vpfhv 1/1 Running 0 5s 10.0.0.6 kwok-node-0 <none> <none>
fake-pod-59bb47845f-wxn4b 1/1 Running 0 5s 10.0.0.1 kwok-node-0 <none> <none>
Update specification
In kwok, nodes and pods are pure API objects, so you can change their specification to simulate or test.
How to save or restore a cluster using kwokctl
Setting up an audit policy
cat <<EOF > audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF
Create a cluster
kwokctl create cluster --kube-audit-policy audit-policy.yaml
Getting the audit log
kwokctl logs audit
Audit Log Example link
How to configure kwok/kwokctl functions
To configure, you will need to create a configuration file YAML . The minimum valid configuration file looks like this:
kind: KwokConfiguration
apiVersion: kwok.x-k8s.io/v1alpha1
options:
---
kind: KwokctlConfiguration
apiVersion: kwok.x-k8s.io/v1alpha1
options:
This configuration shows that we are configuring kwok/kwokctl using version v1alpha1( apiVersion: kwok.x-k8s.io/v1alpha1).
To set the correct parameters for the cluster, you must specify the version of kwok
To use this configuration, place the content in the ~/.kwok/kwok.yaml file, or run the –config=kwok.yaml command from the same directory.
A note about CLI flags, environment variables, and configuration files
Configuration priority order for kwok:
flags specified on the command line
environment variables (prefixed with KWOK_)
values specified in the configuration file ( –config= or ~/.kwok/kwok.yaml)
default values
Using kwok
When used, kwok takes its configuration from the appropriate file and ignores all other configurations.
Using kwokctl
kwokctl takes its configuration from the configuration file and passes it to kwok.
Stage Configuration
The Stage API is a kwok configuration that allows users to define and model various stages in the life cycle of Kubernetes resources: nodes and pods. Each Stage resource specifies a resourceRef field that specifies the type of resource to which it applies, and a selector field that specifies when the stage should be executed.
kind: Stage
apiVersion: kwok.x-k8s.io/v1alpha1
metadata:
name: <string>
spec:
resourceRef:
apiGroup: <string>
kind: <string>
selector:
matchLabels:
<string>: <string>
matchAnnotations:
<string>: <string>
matchExpressions:
- key: <expressions-string>
operator: <string>
values:
- <string>
delay:
durationMilliseconds: <int>
durationFrom:
expressionFrom: <expressions-string>
jitterDurationMilliseconds: <int>
jitterDurationFrom:
expressionFrom: <expressions-string>
next:
statusTemplate: <string>
finalizers:
add:
- value: <string>
remove:
- value: <string>
empty: <bool>
delete: <bool>
By setting the selector and next fields in the spec Stage, you can specify the conditions under which the configuration is applied, as well as the changes that will be made to the resource when the stage is applied. The next field allows users to define the new state of the resource using the statusTemplate field, modify the resource’s finalizers, or perform delete on the resource.
In addition, the delay field on the Stage resource allows users to specify a delay before applying the stage and make it float. This will help simulate real scenarios where events occur at different times.
By setting the delay, selector, and next fields, you can control when and how the stage is applied. This allows you to create complex and realistic models for testing, validation and experimentation, and gain insight into the behavior and performance of your applications and infrastructure.
Examples
Showing how to set up simple knots for kwok

The node-initialize stage applies to nodes that do not have any status.conditions set in their field. When applied, this step sets the status.conditions field for the host, as well as the status.addresses, status.allocatable, and status.capacity fields.
The node-heartbeat stage applies to nodes that have a Ready condition in their field. When applied, this stage maintains a field for the node.Truestatus.conditionsstatus.conditions
How to set up a pod

Stage pod-create-and-ready applies to pods that don’t have a status.podIP and don’t have a metadata.deletionTimestamp. When applied, this step sets the status.conditions, status.containerStatuses, status.initContainerStatuses, status.hostIP, and status.podIP fields for the host. It will also set the Phase and startTime fields to indicate that the node is up and running.
The pod-completed-for-job step applies to running nodes that do not have a metadata.deletionTimestamp and are owned by a job. When applied, this step updates the status.containerStatuses for the node by setting the ready and started fields to true and state.terminated to indicate that the node has terminated. It also sets the stage to Succeeded: it indicates that the node has completed its work.
Stage pod-delete applies to nodes with metadata.deletionTimestamp set. When applied, this step clears the metadata.finalizers field for the node, allowing it to be deleted and then the node itself.
Example link.
Get familiar with Kubernetes
you can in our courses: Kubernetes Base And Kubernetes for Developers.
There is a self-study format – lectures, practical work and certification will open for you, and Kubernetes Base we are launching a new stream, which starts on May 11th. This is a format where you will dive into the depths of K8s together with other students, and course speakers will answer questions about the instrument.
Learn more about the course Kubernetes for Developers and sign up for the stream Kubernetes Base can be on our website.