Single Node Kubernetes Control Plane Migration from docker to containerd

Photo by Matt Artz on Unsplash

To compliment my previous article about migrating kubelets to containerd, this will be about migrating a single node control plane cluster from docker to containerd. My homelab is built fromkubeadm init running on Ubuntu 20.04. Backups are stored on the kubernetes control node itself, but syncing these to a NAS or a cloud bucket would not be a bad idea. Keeping those backups safe is left as an exercise for the reader.

One thing to note, my homelab cluster doesn’t use any special flags. The move to containerd and the systemd cgroup driver requires a modified kubelet config according to documentation, which is installed through the kubeadm --config flag. You may need to adapt the steps around kubeadm config for your setup.

Backup etcd and the k8s control node pki data

Photo by Markus Winkler on Unsplash

This process has been adapted a bit from others below in the resource section. But the gist is, create a cronjob that backs up your etcd cluster periodically.

For example:

apiVersion: batch/v1beta1kind: CronJobmetadata:  name: etcd-backup  namespace: kube-systemspec:  schedule: "*/1 * * * *"  jobTemplate:    spec:      template:        spec:          tolerations:          - key: node-role.kubernetes.io/master            effect: NoSchedule          containers:          - name: backup            # Same image as in /etc/kubernetes/manifests/etcd.yaml            image: k8s.gcr.io/etcd:3.4.13-0            env:            - name: ETCDCTL_API              value: "3"            command: ["/bin/sh"]            args:            - -c            - "etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-snapshot-lateest.db"            volumeMounts:            - mountPath: /etc/kubernetes/pki/etcd              name: etcd-certs              readOnly: true            - mountPath: /backup              name: backup          nodeSelector:           kubernetes.io/hostname: k8s-control-node          restartPolicy: OnFailure          hostNetwork: true          volumes:          - name: etcd-certs            hostPath:              path: /etc/kubernetes/pki/etcd              type: Directory          - name: backup            hostPath:              path: /data/backup              type: DirectoryOrCreate

kubectl apply this config, and it will dump an etcd backup into /data/backup on your kubernetes control node every minute (don’t forget to change the kubernetes.io/hostname name as needed).

Backup the kubernetes cluster keys

sudo cp -r /etc/kubernetes/pki /data/backup/

Backup your kubeadm init config

If you are using a custom kubeadm config, back that up if you haven’t already. If you aren’t (I’m not), then run the following command and store the output in your backup directory.

kubeadm config print init-defaults >> kubeadm-init.yaml

You’ll have to edit this to set the advertiseAddress to be the ip address of your k8s control node.

Now for the fun parts.

Destroying your kubernetes cluster controller

Photo by Devin Avery on Unsplash

The next task is to clean up the k8s control node so you can change the container runtime. Run these commands as root, or with sudo.

systemctl stop kubelet        # stops the kubelet service
systemctl stop docker # stops the docker service
kubeadm reset # wipes the cluster state
**NOTE: THIS IS THE POINT OF NO RETURN**
apt remove kubelet kubeadm docker-ce # removes packages
rm -rf /etc/cni/net.d/* # removes now stale cni config
rm -rf /opt/cni # removes now stale cni binaries, etc
rm -rf /var/lib/docker # removes all old docker data
rm -rf /var/lib/dockershim # removes the dockershim

This leaves your control node in a pretty clean state, but if you have kubelets running still, as I do, you’re on a clock. So proceed quickly, but with care.

Configuring ContainerD container runtime

This process is documented on the Kubernetes Container Runtime documentation, but here’s a shell script that does most of the work. Save this as containerd-bootstrap.sh and make it executable.

#!/bin/bashcat <<EOF | sudo tee /etc/modules-load.d/containerd.confoverlaybr_netfilterEOFsudo modprobe overlaysudo modprobe br_netfilter# Setup required sysctl params, these persist across reboots.cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.confnet.bridge.bridge-nf-call-iptables  = 1net.ipv4.ip_forward                 = 1net.bridge.bridge-nf-call-ip6tables = 1EOF# Apply sysctl params without rebootsudo sysctl --system### Install packages to allow apt to use a repository over HTTPSsudo apt update && sudo apt install -y apt-transport-https ca-certificates curl software-properties-common## Install containerdsudo apt-get update && sudo apt-get install -y containerd# Configure containerdsudo mkdir -p /etc/containerdsudo containerd config default | sudo tee /etc/containerd/config.toml# Restart containerdsudo systemctl restart containerd

And then run it on your kubernetes control node. You’ll be prompted for the root password as necessary.

Now setup runc to use the systemd control group driver.

sudo vi /etc/containerd/config.toml

Search for the runc config section, and add these lines according to the kubernetes documentation:

[plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options]
SystemdCgroup = true

Now is a good time to reboot: sudo reboot

Reconnect once the reboot is complete.

Restoring the k8s control node

Photo by Colton Kresser on Unsplash

Steps for restoring the k8s control node PKI and etcd backup restoration are optional if you haven’t blown away the /etc/kubernetes/pki and /var/lib/etcd directories. I’m including them here in case something goes wrong.

Restore the k8s control node PKI

  • sudo cp -r /data/backup/pki /etc/kubernetes/

Install etcdctl

This script will install etcdctl to /usr/bin. It’s a slightly altered version of the standard release script for etcd on linux. Save this as install-etcd-tools.sh

#!/bin/bash# etcd 3.4.13 is installed by default with kubernetes 1.20, change this as neededETCD_VER=v3.4.13# choose either URLGOOGLE_URL=https://storage.googleapis.com/etcdGITHUB_URL=https://github.com/etcd-io/etcd/releases/downloadDOWNLOAD_URL=${GOOGLE_URL}rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gzrm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-testcurl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gztar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz/tmp/etcd-download-test/etcdctl versionsudo cp /tmp/etcd-download-test/etcdctl /usr/bin

and run it, sudo ./install-etcd-tools.sh

Restore the etcd backup

These commands assume you’re running from the /data/backup directory on the k8s control node as root.

  • export ETCDCTL_API=3
  • etcdctl snapshot restore /data/backup/etcd-snapshot-latest.db
  • mv ./default.etcd/member/ /var/lib/etcd/

Update the kubeadm config

Add this block to the end of the kubeadm-config.yaml file. This might be unnecessary if kubeadm does autodiscovery at this point. I wasn’t able to quickly find anything definitive though, so this got added according to the documentation.

---apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationcgroupDriver: systemd

Reinstall the kubeadm and kubelet packages

sudo apt install -y kubeadm kubelet

Reinitialize the k8s cluster

  • sudo kubeadm init — ignore-preflight-errors=DirAvailable — var-lib-etcd — config /data/backup/kubeadm-config.yaml

And if all went well, you should be able to see your cluster nodes and pods left as they were prior to shutdown.

If you haven’t already, migrate your kubelets as well. This was a fun weekend project.

As always, hope this helps. GL;HF

Resources

I'm an engineering leader who is passionate about reliability engineering and building sustainable/scalable teams.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store