Kubeshark – Kuberneres monitoring and analysis
Introduction
Wireshark is a well-known packet capture, analysis, and troubleshooting tool. It can intercept live network traffic and analyze it in real time at a microscopic level, as well as read and process stored capture files. Wireshark can analyze and display many different protocols and has a powerful filtering system to narrow down the traffic of interest.
TCPDump/Wireshark gives us the ability to visualize and understand what is happening on our network at a microscopic level. Imagine if something like this is possible in K8s, if you can see exactly what happens when you deploy a workload, get pods, create a service account, and how different workloads interact with each other, etc.
What's missing now?
To see exactly what happens when you run the kubectl command, you can simply use the verbose flag with your kubectl command at some level of granularity, for example – kubectl get pods -v=6.
The level of detail starts at 0 and ends at 9, 0 is the minimum and 9 is the maximum level of detail. This is already present, but there is no clear visualization of the same flow.
Kubectl with the -v flag allows you to see L7 API calls, it will not allow you to monitor any network traffic per se.
Kubeshark does indeed capture L3 and L7, in fact it will have access to the entire L2 frame. This will create a dashboard/view to visualize the same, just like Wireshark.
Why is Kubeshark needed?
Monitor everything – be it monitoring Pod-Pod network traffic, API calls made, monitoring an individual node, service or a specific path, for example – /latest/meta-data or /health, etc.
Visualize – Kubeshark viewer is so simple yet so powerful that it allows you to view traffic in real time, filter, analyze and troubleshoot any problems. The query response information in the right panel is very clear.
PCAP storage – Store PCAP for future analysis, you can easily upload it to S3 or some other storage, download and visualize it later.
Simplify debugging – Whether it's debugging something or deepening your understanding of the communication between different components in a cluster, it makes it much easier to complete.
Real time alert – I can create an alert for K8s using the data that Kubeshark allows me to use. Think about failed get secrets calls, failed instance metadata calls, many other similar anomalies.
Installing Kubeshark
Let's install Kubeshark and see how it works. You can install KubeShark using this command:
sh <(curl -Ls https://kubeshark.co/install)
Launch Kubeshark:
kubeshark tap -A
tap -A affects all namespaces, you can specify -n to monitor a specific namespace.
kubeshark tap -n kube-system
Kubeshark will start monitoring the traffic and launch the viewer on port 8899 on your local hosting.
I have my test cluster deployed using KOPS, note that the Kubeshark binary looks up your current context from ~/.kube/config and uses the same credentials and cluster to create its resources and run monitoring.
You can create a local kind cluster to quickly get an idea of how Kubeshark works.
A little practice
Let's take a look at what we're tracking and what capabilities Kubeshark has in its dashboard. Here are some example filters I used to test kubeshark.
Single node monitoring:
Visualization using a service card:
Monitor the AWS instance credential metadata path:
Final Thoughts
Helps deepen my understanding of how the various components of a cluster interact and work together at the lowest possible level.
Debugging/troubleshooting becomes very easy by adding appropriate filters to the traffic.
Service maps provide a high-level overview of what my cluster looks like from a visual perspective.
Data can be expanded to provide real-time alerts, both operational and security.
It would be a good idea to run a performance test at a decent scale and see how this tool scales and impacts compute and storage given that traffic volume is growing exponentially on a massive scale.
Honestly, it's not fair to compare this to Sysdig falco or Cilium Tetragon, but the way it captures and sets up data makes me look at the real-time detection aspect and think about using it in some security aspects more related to K8s API anomalies.
Perhaps this is a tool I should use extensively to better understand the calls under the hood in order to write better runtime security policies that are more control plane related. This won't do much to detect any container anomalies or those related to the container runtime AFAIK.
Conclusion
Kubeshark is a powerful tool that was previously known as Mizu, developed by UP9 as an API traffic viewer and later made open source, now the name has changed to Kubeshark, which means it is Wireshark revived for the K8s.
While there are sophisticated K8s runtime security tools like Sysdig Falco, Cilium Tetragon which are built using eBPF capabilities, the documentation mentions that Kubeshark also uses eBPF but is more focused on providing a microscopic level visualization layer for better understanding of communications.
Link to Github – https://github.com/kubeshark/kubeshark
Link – https://kubeshark.co
