Getting KubeVirt and K0s to play nicely on Ubuntu 20.04 LTS
One of the challenges of traditional metal providing companies seems to be the ability to make efficient use of hardware while still being able to dynamically schedule and orchestrate VMs and containers across a sea of compute resources. So I decided to learn more about it by getting it working on my homelab. Here’s how it worked.
Backup your cluster with Velero
The scariest bit about this adventure was decimating my kubeadm installed cluster which I’ve been nursing for the past several years. I’ve watched kubeadm go from a tool that would regularly destroy my cluster to a thing that is reliable and easily performs upgrades minus the occasional api deprecation. Enter Velero. This saved my bacon.
I have minio running on a synology diskstation, and used that to store the backup. To do the backup, you first have to install velero into your cluster using this command. You’ll need a file called creds-velero that has your minio access-id and secret-key in it.
velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.3.0 --bucket velero --secret-file ./creds-velero --use-volume-snapshots=false --backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url=http://minio:9000
Once the service is running in your cluster, perform the backup with this command:
velero backup create homelab-backup-$date
It only took a couple of minutes and the backup was complete.
Installing k0s using k0sctl
From here I could delete my old cluster. I ran a quick ansible playbook to run kubeadm reset -facross the nodes, and cleanup the CNI directory. Within a few minutes I had a fresh set of machines to work on.
First you need to create a k0sctl.yamlfile to describe your new k0s cluster. One of the cool things about k0s is the support for helm charts post install. I had quite a few, but then removed most of them so I could have flexibility over updates later. Some of them conflicted with the velero restore which comes later too so it’s good to be mindful. I’ve included a couple here to serve as a functional example.
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: k0s-cluster
spec:
hosts:
- ssh:
address: <ip_addr>
user: <user>
port: 22
keyPath: /home/<user>/.ssh/id_rsa
role: controller
- ssh:
address: <ip_addr>
user: <user>
port: 22
keyPath: /home/<user>/.ssh/id_rsa
role: worker
... do this for the rest of your worker nodes ...
k0s:
version: 1.22.3+k0s.0
config:
apiVersion: k0s.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: homelab-k8s
spec:
api:
address: <controller_ip_addr>
port: 6443
k0sApiPort: 9443
externalAddress: <external_dns_name>
sans:
- <conttroller_ip_addr>
storage:
type: etcd
etcd:
peerAddress: <controller_ip_addr>
network:
podCIDR: 10.244.0.0/16
serviceCIDR: 10.96.0.0/12
provider: kuberouter
kuberouter:
mtu: 0
peerRouterIPs: ""
peerRouterASNs: ""
autoMTU: true
kubeProxy:
disabled: false
mode: iptables
podSecurityPolicy:
defaultPolicy: 00-k0s-privileged
telemetry:
enabled: false
installConfig:
users:
etcdUser: etcd
kineUser: kube-apiserver
konnectivityUser: konnectivity-server
kubeAPIserverUser: kube-apiserver
kubeSchedulerUser: kube-scheduler
images:
konnectivity:
image: us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-agent
version: v0.0.24
metricsserver:
image: gcr.io/k8s-staging-metrics-server/metrics-server
version: v0.5.0
kubeproxy:
image: k8s.gcr.io/kube-proxy
version: v1.22.2
coredns:
image: docker.io/coredns/coredns
version: 1.7.0
kuberouter:
cni:
image: docker.io/cloudnativelabs/kube-router
version: v1.2.1
cniInstaller:
image: quay.io/k0sproject/cni-node
version: 0.1.0
default_pull_policy: IfNotPresent
konnectivity:
agentPort: 8132
adminPort: 8133
extensions:
helm:
repositories:
- name: descheduler
url: https://kubernetes-sigs.github.io/descheduler/
- name: nfs-subdir-external-provisioner
url: https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
charts:
- name: descheduler
chartname: descheduler/descheduler
version: "0.21.0"
values: |
cmdOptions:
v: 3
deschedulerPolicy:
strategies:
LowNodeUtilization:
enabled: true
params:
nodeResourceUtilizationThresholds:
targetThresholds:
cpu: 50
memory: 50
pods: 50
thresholds:
cpu: 20
memory: 20
pods: 20
RemoveDuplicates:
enabled: true
RemovePodsViolatingInterPodAntiAffinity:
enabled: true
RemovePodsViolatingNodeAffinity:
enabled: true
params:
nodeAffinityType:
- requiredDuringSchedulingIgnoredDuringExecution
RemovePodsViolatingNodeTaints:
enabled: true
fullnameOverride: ""
image:
pullPolicy: IfNotPresent
repository: k8s.gcr.io/descheduler/descheduler
tag: ""
nameOverride: ""
podSecurityPolicy:
create: true
priorityClassName: system-cluster-critical
rbac:
create: true
resources:
requests:
cpu: 500m
memory: 256Mi
limits:
cpu: 100m
memory: 256Mi
schedule: '*/2 * * * *'
serviceAccount:
create: true
namespace: kube-system
- name: nfs-subdir-external-provisioner
chartname: nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
version: "4.0.1"
values: |
affinity: {}
image:
pullPolicy: IfNotPresent
repository: gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner
tag: v4.0.2
leaderElection:
enabled: true
nfs:
path: /path/to/storage/
server: <nfs_server_ip_addr>
nodeSelector: {}
podSecurityPolicy:
enabled: false
rbac:
create: true
replicaCount: 1
resources: {}
serviceAccount:
create: true
storageClass:
accessModes: ReadWriteOnce
allowVolumeExpansion: true
archiveOnDelete: true
create: true
defaultClass: true
name: managed-nfs-storage
onDelete: null
pathPattern: null
reclaimPolicy: Retain
strategyType: Recreate
tolerations: []
namespace: kube-systemOnce the file is crafted, install is simple. Run the command k0sctl applyin the same directory as your k0sctl.yaml file and the command will ssh into hosts and install k0s command nodes first, then worker nodes. Worker nodes are automatically added to the cluster in the process. When all is said and done, you have a fully functional Kubernetes cluster.
To run commands against the new kubernetes cluster, use k0sctl kubeconfig > ~/.kube/configto give yourself admin rights.
KubeVirt Prereqs — installing KVM
To be honest, you should install kvm on your worker nodes before you install k0s, but I didn’t do that. In case you don’t either, install kvm by running the following commands:
sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utilsYou can verify kvm started by running the command
sudo systemctl status libvirtdAnd if it’s not in the running state, you might want to enable and start it.
sudo systemctl enable libvirtd
sudo systemctl start libvirtdApparmor can cause problems with kubevirt later. In order to work around that, I had to edit /etc/apparmor.d/usr.sbin/libvirtd to add the line /usr/libexec/qemu-kvm PUx, (include the comma). I added this around line 94. Then restart apparmor sudo systemctl reload apparmor.service to reload the new rules. This works around the fact that Ubuntu puts binaries in libexec and uses odd binary names from time to time. Thanks to this comment from github user aodinokov for pointing me in the right direction.
Installing Kubevirt
The actual install of kubevirt is easy, and well documented. I barely modified the instructions in this script:
#!/bin/bashexport VERSION=$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases | grep tag_name | grep -v -- '-rc' | sort -r | head -1 | awk -F': ' '{print $2}' | sed 's/,//' | xargs)echo -n "Using kubevirt version: "
echo $VERSIONecho "Installing kubevirt operator"
kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yamlecho "Installing CRDs"
kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml
You can watch the pods start in the kubevirt namespace. This installs the operator and a daemonset on the worker nodes to schedule VMs. To control what worker nodes are allowed to schedule VMs you can patch the kubevirt custom resource instance like so: kubectl patch -n kubevert kubevert kubevirt --type merge --patch '{"spec":{"workloads":{"nodePlacement":{"nodeSelector":{"host-platform":"metal"}}}}}'
This will only schedule the virt-handler app on nodes with the label host-platform=metal
Final fixes
Kubevirt expects the kubelet to live in /var/lib/kubelet but k0s installs it into /var/lib/k0s/kubelet so to reconcile, one last workaround is needed. You’ll need to run this set of commands on all your worker nodes (that you plan to run VMs on, but go ahead and do all of them for consistency). You should probably shut the kubelet down by stopping the k0s service on the worker node before doing this. I didn’t and it was fine, I did reboot though. Your mileage may vary. #yolo
sudo su -
cd /var/ilb/kubelet
cp -r * /var/lib/k0s/kubelet
cd ..
rm -rf kubelet
ln -s /var/lib/k0s/kubelet /var/lib/kubelet
rebootWhen the machine comes back up, you should be all set. There’s an ansible install option for k0s which could be extended to support all of this. Maybe that will be a later evolution of this process.
Launching a VM
Now you can test the system. I used an ubuntu container image from github user tedezed who has an amazing repo of kubevirt container images called kubevirt-images-generator. Create a file called vm-ubuntu.yaml like so:
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
name: ubuntu-bionic
spec:
running: false
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: ubuntu-bionic
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 2048M
networks:
- name: default
pod: {}
volumes:
- name: containervolume
containerDisk:
image: tedezed/ubuntu-container-disk:20.0
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:toor
expire: FalseThen apply the VM manifest with the command kubectl apply -f vm-ubuntu.yaml and verify it with the command kubectl get virtualmachines. To start an instance of this virtual machine, run the command virtctl start ubuntu-bionic and wait a bit. You can monitor the progress of the instance starting by watching kubectl get vmi.You can see the vm boot by running virtctl console ubuntu-bionic and once it’s completely booted, you can ssh in with the command virtctl ssh ubuntu-bionic. At this point you have a fully fledged kvm based virtual machine running from a kubernetes control plane using the same API design that kubernetes uses to manage containers. This is a very powerful concept.
Conclusion
Certainly there’s some trickery to getting ubuntu, k0s, and kubevirt to work together, but it works. It was fun to figure out, and there’s a lot more you can do with it from here. Having a single declarative API to schedule VMs that performs live migration, and leverages the orchestration, drift detection and management, role based access controls, and network level controls, is a powerful tool. You can eliminate the redundant costs and cognitive loads associated with other solutions, and create a path to modernization that many enterprises need. And it works across bare metal, private cloud, and public cloud providers. This is the middle ground of modernization and migration that brings the hybrid cloud to life. All of this in a weekend project! To quote two minute papers, “What a time to be alive!”
