Since my first tentative steps in to Kubernetes it’s been an interesting journey. For the most part I suspect the most common way to interact with Kubernetes is to use a managed platform through one of the main public cloud players (that’s certainly been my experience), but it doesn’t do a lot to understand the nuts and bolts of the platform. I’d been meaning to try and get stuck in to Kubernetes The Hard Way; Kelsey Hightower’s notorious guide to manually bootstrapping a Kubernetes Cluster on GCP with a view to gaining a better understanding the underlying mechanisms. So what did I get out of the process?
Linux Systems – Doing Linux Things
So the first thing that becomes apparent very quickly during the bootstrapping and building process is that a little bit of Linux knowledge goes a long way. What we build as part of the process is a series of Ubuntu Linux VMs and the configuration involves a lot of very familiar actions.
Bootstrapping Kubernetes and etcd doesn’t involve any new futuristic method, just good old fashioned systemd unit files, encrypted communication between nodes doesn’t revolve around a new method of encryption, just certificates and keys like we’ve been doing forever (though that might be enough to put people off)!
Accessing each of your nodes is as simple as SSH (though in my experience with EKS this is still possible on worker nodes using unmanaged node groups, we never have the ability to access to the master nodes in a managed deployment).
After a first run through, it is apparent why this type of implementation may be desirable; trading a pre-made implementation for the ability to tweak every aspect of your build, though this is of course relative to your needs.
Kubeconfigs – They’re Everywhere!
Up to this point, I’ve only ever become aware of kubeconfigs as a method to access the Kubernetes API from the perspective of either an administrator or for programmatic access from a configuration pipeline.
This is a casualty of only using managed Kubernetes deployments where the lions share of the config is already taken care of and there is no need to really configure communication or authentication between the Control and Data Planes. Following this work it is now apparent that the nodes now use their own kubeconfigs in order to communicate with each other.
Specifically, kubeconfigs are created for:
- kubelet and kube-proxy: Each Worker Node has these kubeconfigs allowing communication between Worker Node(s) and the Master Node(s) API Server
- kube-controller-manager and kube-scheduler: Each Master Node has these kubeconfigs allowing regulation of the Kubernetes Control Loops on all Nodes and the Scheduling of Pods to Nodes
etcd – Confirming Encrypted Secrets
etcd is a clustered, HA Key Value Store used by Kubernetes to store it’s data (including it’s secrets), in managed Kubernetes deployments, etcd is usually pre-baked and ready for your pleasure (AWS have a very interesting article on how etcd is deployed within the EKS managed Control Plane).
Much like building Kubernetes itself, bootstrapping etcd is a matter of traditional systemd unit files and TLS keys/certificates. What we get with a custom built deployment is the ability to interact with etcd and query our our Key Values; categorically confirming that they are properly encrypted at rest. The Smoke Test section of Kubernetes The Hard Way covers a hard and fast method of testing this function.
Extra Passes and Tweaks
After getting finished I realised that one pass isn’t going to get the job done, it’s going to take a few runs and some serious adaptations to get any real mastery. Think of Kubernetes the Hard Way as a jumping off point in to getting some real understanding. Whilst the guide is pretty perfect, it’s intentionally a highly manual process designed to make sure you understand everything you’re doing.
For me, there’s some steps that can be made more efficient when it comes to provisioning the entire environment. I don’t like to leave labs running in a public cloud if I can avoid it as they can become costly very quickly so the next steps are going to be to build out the entire configuration in a combination of Terraform and Ansible so I can spin up the entire ready-built solution at a few presses.
So should you do it? If you’re interested in Kubernetes, do it the hard way at least once and then start adding some tweaks.
