Single Node Kubernetes Control Plane Migration from docker to containerd

Photo by Matt Artz on Unsplash

To compliment my previous article about migrating kubelets to containerd, this will be about migrating a single node control plane cluster from docker to containerd. My homelab is built fromkubeadm init running on Ubuntu 20.04. Backups are stored on the kubernetes control node itself, but syncing these to a NAS or a cloud bucket would not be a bad idea. Keeping those backups safe is left as an exercise for the reader.

One thing to note, my homelab cluster doesn’t use any special flags. The move to containerd and the systemd cgroup driver requires a modified kubelet config according to documentation, which is installed through the kubeadm --config flag. You may need to adapt the steps around kubeadm config for your setup.

Backup etcd and the k8s control node pki data

Photo by Markus Winkler on Unsplash

This process has been adapted a bit from others below in the resource section. But the gist is, create a cronjob that backs up your etcd cluster periodically.

For example:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          tolerations:
          - key: node-role.kubernetes.io/master
            effect: NoSchedule
          containers:
          - name: backup
            # Same image as in /etc/kubernetes/manifests/etcd.yaml
            image: k8s.gcr.io/etcd:3.4.13-0
            env:
            - name: ETCDCTL_API
              value: "3"
            command: ["/bin/sh"]
            args:
            - -c
            - "etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-snapshot-lateest.db"
            volumeMounts:
            - mountPath: /etc/kubernetes/pki/etcd
              name: etcd-certs
              readOnly: true
            - mountPath: /backup
              name: backup
          nodeSelector:
           kubernetes.io/hostname: k8s-control-node
          restartPolicy: OnFailure
          hostNetwork: true
          volumes:
          - name: etcd-certs
            hostPath:
              path: /etc/kubernetes/pki/etcd
              type: Directory
          - name: backup
            hostPath:
              path: /data/backup
              type: DirectoryOrCreate

kubectl apply this config, and it will dump an etcd backup into /data/backup on your kubernetes control node every minute (don’t forget to change the kubernetes.io/hostname name as needed).

Backup the kubernetes cluster keys

sudo cp -r /etc/kubernetes/pki /data/backup/

Backup your kubeadm init config

If you are using a custom kubeadm config, back that up if you haven’t already. If you aren’t (I’m not), then run the following command and store the output in your backup directory.

kubeadm config print init-defaults >> kubeadm-init.yaml

You’ll have to edit this to set the advertiseAddress to be the ip address of your k8s control node.

Now for the fun parts.

Destroying your kubernetes cluster controller

Photo by Devin Avery on Unsplash

The next task is to clean up the k8s control node so you can change the container runtime. Run these commands as root, or with sudo.

systemctl stop kubelet        # stops the kubelet service
systemctl stop docker         # stops the docker service
kubeadm reset                 # wipes the cluster state 
**NOTE: THIS IS THE POINT OF NO RETURN**
apt remove kubelet kubeadm docker-ce # removes packages
rm -rf /etc/cni/net.d/*       # removes now stale cni config
rm -rf /opt/cni               # removes now stale cni binaries, etc
rm -rf /var/lib/docker        # removes all old docker data
rm -rf /var/lib/dockershim    # removes the dockershim

This leaves your control node in a pretty clean state, but if you have kubelets running still, as I do, you’re on a clock. So proceed quickly, but with care.

Configuring ContainerD container runtime

This process is documented on the Kubernetes Container Runtime documentation, but here’s a shell script that does most of the work. Save this as containerd-bootstrap.sh and make it executable.

#!/bin/bash
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# Setup required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# Apply sysctl params without reboot
sudo sysctl --system
### Install packages to allow apt to use a repository over HTTPS
sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
## Install containerd
sudo apt-get update && sudo apt-get install -y containerd
# Configure containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
# Restart containerd
sudo systemctl restart containerd

And then run it on your kubernetes control node. You’ll be prompted for the root password as necessary.

Now setup runc to use the systemd control group driver.

sudo vi /etc/containerd/config.toml

Search for the runc config section, and add these lines according to the kubernetes documentation:

[plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options]
  SystemdCgroup = true

Now is a good time to reboot: sudo reboot

Reconnect once the reboot is complete.

Restoring the k8s control node

Photo by Colton Kresser on Unsplash

Steps for restoring the k8s control node PKI and etcd backup restoration are optional if you haven’t blown away the /etc/kubernetes/pki and /var/lib/etcd directories. I’m including them here in case something goes wrong.

Restore the k8s control node PKI

• sudo cp -r /data/backup/pki /etc/kubernetes/

Install etcdctl

This script will install etcdctl to /usr/bin. It’s a slightly altered version of the standard release script for etcd on linux. Save this as install-etcd-tools.sh

#!/bin/bash
# etcd 3.4.13 is installed by default with kubernetes 1.20, change this as needed
ETCD_VER=v3.4.13
# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
/tmp/etcd-download-test/etcdctl version
sudo cp /tmp/etcd-download-test/etcdctl /usr/bin

and run it, sudo ./install-etcd-tools.sh

Restore the etcd backup

These commands assume you’re running from the /data/backup directory on the k8s control node as root.

• export ETCDCTL_API=3
• etcdctl snapshot restore /data/backup/etcd-snapshot-latest.db
• mv ./default.etcd/member/ /var/lib/etcd/

Update the kubeadm config

Add this block to the end of the kubeadm-config.yaml file. This might be unnecessary if kubeadm does autodiscovery at this point. I wasn’t able to quickly find anything definitive though, so this got added according to the documentation.

---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Reinstall the kubeadm and kubelet packages

sudo apt install -y kubeadm kubelet

Reinitialize the k8s cluster

• sudo kubeadm init — ignore-preflight-errors=DirAvailable — var-lib-etcd — config /data/backup/kubeadm-config.yaml

And if all went well, you should be able to see your cluster nodes and pods left as they were prior to shutdown.

If you haven’t already, migrate your kubelets as well. This was a fun weekend project.

As always, hope this helps. GL;HF

Resources

• https://elastisys.com/backup-kubernetes-how-and-why/
• https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd
• https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd
• https://github.com/etcd-io/etcd/releases