If, like me, you’ve come from a traditional sysadmin background then Kubernetes can be daunting to say the least, this doesn’t get much easier when it comes to trying to get to grips with how to debug networking issues. Kubernetes networking is VAST and supports a number of complex implementations that vary between the major Kubernetes-as-a-Service platforms (GKE, EKS, AKS) as well as many other options. The broad strokes are broken down in the official docs but as is so often the case the manual won’t do much to help you when it comes to crunch time and you have a networking issue to solve.
The rules for basic network debugging haven’t changes that much however and in this post I want to take a look at some simple methods for using some of the old standards for networking troubleshooting and how we can leverage them within a Kubernetes Cluster to make life easier.
Busybox and Netshoot – Weapons of Choice
Most admins are familiar at least with busybox, the Swiss Army Knife of Linux troubleshooting; it comes with dozens of binaries for debugging, several of which are handy for network issues, a much newer and more network-focused project however is netshoot which contains a HUGE amount of tools for debugging applications at just about every points of the stack. I won’t be looking at netshoot for the basics but we’ll come back to it for some more advanced stuff.
For our purposes we’ll be looking at some vital tools inside busybox:
- telnet
- tcpdump
- dig/nslookup
- traceroute
The first thing we’ll need to do before anything is get a working instance of busybox inside out Kubernetes Cluster, we can do this using kubectl run as below:
# Deploy to the default namespace kubectl run -i --tty mydebugpod --image=busybox --restart=Never -- sh # If you don't see a command prompt, try pressing enter. /# # Deploy to a specific namespace kubectl run -i --tty mydebugpod -n NAMESPACE --image=busybox --restart=Never -- sh # If you don't see a command prompt, try pressing enter. /#
This will deploy a pod named mydebugpod based on the official busybox image (which we will pull directly from DockerHub unless otherwise configured). The — sh argument will pass us directly in to the shell of the pod once it has started.
From here we can use the tools included within busybox.
Debugging Egress with Telnet
Now that we have busybox running within the same namespace as our pod, we can test egress to another node using telnet exactly as we would in a traditional environment:
telnet testserver.tinfoilcipher.co.uk 22 # connected to testserver.tinfoilcipher.co.uk. # Escape character is '^]'. # SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
From this we can see that we have egress from our namespace to our node over SSH, this same method can be used to debug TCP connectivity to any IP address or hostname and is a simple test to perform to test egress from a namespace.
Testing DNS Resolution
DNS resolution is another common issue which we can test with busybox using dig or nslookup:
# Forward lookup dig testserver.tinfoilcipher.co.uk nslookup testserver.tinfoilcipher.co.uk # Reverse lookup dig 10.0.1.10 nslookup 10.0.1.10
Route Testing
In order to confirm that traffic is observing the expected route through your Cluster (in the event you have a complex configuration) and after existing the cluster, busybox contain traceroute, we can use this to test that our traffic is going the right way:
traceroute testserver.tinfoilcipher.co.uk
Packet Captures – The Ultimate Tool
This is all good and well for very basic tests, however sooner or later the only thing that’s going to do for a complicated problem is a proper packet capture. As most admins know there’s three tools you really have at your disposal; Wireshark, tshark (the terminal driven deployment of Wireshark) and tcpdump.
Wireshark is the undisputed king of network troubleshooting tools, but getting it to play with Kubernetes is something of a challenge, luckily the packet captures from tcpdump can be exported and viewed in Wireshark, so our problem gets a little easier).
Ephemeral Debugging Containers
As of Kubernetes 1.16 a new feature (with a complicated sounding name) called Ephemeral Debug Containers has been released (as of the current stable Kubernetes, 1.19, this function is still in Alpha). I’ve only recently had a chance to play with it and it’s perfectly suited to this kind of task.
A Debug Container is created as a Sidecar container within an existing pod, in our case this means our busybox container will be created inside an existing Pod and allow us to execute tcpdump to capture traffic from the container running our application (or any other applications running inside our Pod); the network topology inside a Pod is completely flat, meaning that once we’re running inside we can capture whatever we like.
In order to do our capture let’s come back to our netshoot container that we mentioned earlier, but we’re going to get a little smarter and capture our traffic live in to Wireshark (yes that really works but there are a caveat).
In order to make this magic work, we’ll use a kubectl alpha feature called debug:
kubectl alpha debug -i tinfoilapp-497d0a86e0-nvhea -n tinfoiltest --image=nicolaka/netshoot --target=tinfoil -- tcpdump -i eth0 -w - | wireshark -k -i -
That’s a bit of a mouthful so let’s break it down…
We’re going to create a debug container for a pod named tinfoilapp-497d0a86e0-nvhea, in the namespace tinfoiltest and target the container in that pod named tinfoil. Once the debug container is created we’ll invoke tcpdump from within and start capturing all traffic on interface eth0 and then write out all captured packets (via the -w – argument) and then pipe all output to Wireshark on our local machine which we’ve instructed to start straight away and listen on all interfaces (via the -k -i – argument).
The end result of this is that Wireshark will instantly start to capture packets as if it were running inside the cluster and once we close it, the whole debug process should collapse itself.
This Doesn’t Work on Windows – But There’s an Answer!
So the caveat I alluded to earlier, this doesn’t work properly on Windows! That final pipe out to Wireshark doesn’t seem to work on Windows, Wireshark in Linux works great but as is so often the case, Windows puts a spanner in the works.
I have however found that if we use the excellent Windows Subsystem for Linux (WSL) and redirect the traffic to tshark we can capture the traffic in a single command still and then open the capture in Wireshark (assuming you don’t want to view it all in the terminal).
Assuming the same configuration as above, we can use the single command:
kubectl alpha debug -i tinfoilapp-497d0a86e0-nvhea -n tinfoiltest --image=nicolaka/netshoot --target=tinfoil -- tcpdump -i eth0 -w - | sudo tshark -i - -w /mnt/c/TEMP/testcapture.pcap
This will now pipe the output to tshark until a break is issued (via CTRL + C) and the completed pcap can be found within your Windows C:\TEMP.
Packet Captures – I Don’t Have Kubernetes 1.16!
So you don’t have Kubernetes 1.16 yet and you can’t upgrade for whatever reason? This removes the ability to use Ephemeral Debug Containers, there is another option but it comes with a few issues of it’s own. The problem can still be solved but right out of the gate I wouldn’t recommend this for anything but a development environment and certainly wouldn’t recommend it for production!
To get our capture, we can attempt to install tcpdump DIRECTLY in to a running container. This is a very variable process and relies on our container being built on a base image that has both an interactive shell and some package management (Alpine, Ubuntu etc.). Hardened “distroless” images aren’t a candidate for this process.
So you want to try this anyway? Let’s get an interactive shell on a running pod, in my example the container is based on Alpine Linux and uses the apk package manager:
kubectl exec --stdin --tty tinfoilapp-497d0a86e0-nvhea -c tinfoil -n tinfoiltest -- /bin/sh # If you don't see a command prompt, try pressing enter. /# apk add tcpdump /# exit
This will install tcpdump in to our running container (tinfoil). This is obviously far from perfect as we have modified a live container which we really shouldn’t be doing, but it will get us where we need to be for the purposes of emergency troubleshooting, from here our command to capture traffic changes slightly as we will need to use kubectl exec to execute tcpdump within our container and pass the traffic to either Wireshark or tshark (depending on our configuration):
#--Capture in Wireshark (Linux) kubectl exec tinfoilapp-497d0a86e0-nvhea -c tinfoil -n tinfoiltest -- tcpdump -i eth0 -w - | wireshark -k -i - #--Capture in tshark and export pcap (WSL on Windows) kubectl exec tinfoilapp-497d0a86e0-nvhea -c tinfoil -n tinfoiltest -- tcpdump -i eth0 -w - | sudo tshark -i - -w /mnt/c/TEMP/testcapture.pcap
Finally, make sure you clean up your running pod:
kubectl exec --stdin --tty tinfoilapp-497d0a86e0-nvhea -c tinfoil -n tinfoiltest -- /bin/sh # If you don't see a command prompt, try pressing enter. /# apk del tcpdump /# exit
It’s probably more prudent at this point, however, to simply re-provision the entire pod for safety, especially if you’ve had to do this in a production environment!
Conclusion
After a couple of weeks of digging in to the topic, I’ve found that getting to grips with the basics isn’t as bad as it looks. Hopefully this helps a few people be less intimidated by debugging Kubernetes networking issues, the basics never really change all that much when trying to solve simple networking issues.
Whilst Kubernetes networking is FAR from simple (and some deployments really do make my head spin), the small issues can often still be small and having the right tools to hand to try and observe and solve them them makes life much easier than trying to work out problems with a blindfold on :).
