一、目标
本文将安装kubernetes + containerd + kata/runc，发布容器时可以指定使用kata或runc来运行。
本文以 [how-to-use-k8s-with-containerd-and-kata](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-k8s-with-containerd-and-kata.md) 文章为参考主线。
二、环境准备
在windows上用vmware workstation 虚拟一台虚拟机，注意CPU要勾选开启 `虚拟化Intel VT-x/EPT 或 AMD-V/RVI(V)`。
操作系统内核为：centos-7.6 + kernel-5.4.228
三、安装容器运行时
3.1 安装kata
此 [链接](https://github.com/kata-containers/kata-containers/tree/main/docs/install) 给出了Kata相应的安装方式对比
我们使用的操作系统为CentOS-7.6，由于 [Using official distro packages](https://github.com/kata-containers/kata-containers/tree/main/docs/install#official-packages) 指引中，最低需要CentOS-8
所以这里我们选择 [Manual](https://github.com/kata-containers/kata-containers/tree/main/docs/install#manual-installation) 的安装方式。
从kata的[release](https://github.com/kata-containers/kata-containers/releases)页面，下载对应版本的包，这里我们下载最新版本 [kata-static-2.5.2-x86_64.tar.xz](https://github.com/kata-containers/kata-containers/releases/download/2.5.2/kata-static-2.5.2-x86_64.tar.xz)。下载下来后，解压得到`/opt`目录下（tar.xz包解压后会有opt目录，所以直接解压到根目录下）：
```
$ tar xvf kata-static-2.5.2-x86_64.tar.xz -C /
$ ls /opt/kata/bin/
cloud-hypervisor containerd-shim-kata-v2 firecracker jailer kata-collect-data.sh kata-monitor kata-runtime qemu-system-x86_64
```
然后需要把containerd-shim-kata-v2、kata-runtime、kata-collect-data.sh做一个软链接到PATH目录下：
```
$ ln -s /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2
$ ln -s /opt/kata/bin/kata-collect-data.sh /usr/local/bin/kata-collect-data.sh
$ ln -s /opt/kata/bin/kata-runtime /usr/local/bin/kata-runtime
```
然后检查一下kata版本，以及检查一下系统是否能跑kata容器：
```
$ kata-runtime --version
kata-runtime : 2.5.2
commit : 4b39dc0a390584d2ee21072cca7707f4ee7f56c5
OCI specs: 1.0.2-dev
$ kata-runtime kata-check
WARN[0000] Not running network checks as super user arch=amd64 name=kata-runtime pid=10121 source=runtime
System is capable of running Kata Containers
System can currently create Kata Containers
```
3.2 安装runc
我们参考containerd的 [getting-started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md#step-2-installing-runc) 一文，来安装runc。
我们从runc的[github release](https://github.com/opencontainers/runc/releases)页面，下载最新版本的runc二进制文件。这里我们下载 1.1.4版本的 [runc.amd64](https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64)。然后，把它安装到对应目录下：
```
$ install -m 755 runc.amd64 /usr/local/bin/runc
$ runc -v
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d1
spec: 1.0.2-dev
go: go1.17.10
libseccomp: 2.5.4
```
四、安装containerd
containerd的安装我们参考containerd的 [getting-started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md#option-1-from-the-official-binaries) 文章，使用二进制进行安装。[kata文章中的containerd安装指引](https://github.com/kata-containers/kata-containers/blob/main/docs/install/container-manager/containerd/containerd-install.md#install-containerd) 其实类似，不过它是把kata作为默认运行时。
下载最新版本的 [containerd-1.6.15-linux-amd64.tar.gz](https://github.com/containerd/containerd/releases/download/v1.6.15/containerd-1.6.15-linux-amd64.tar.gz)，解压到`/usr/local/`目录下：
```
$ tar xvf containerd-1.6.15-linux-amd64.tar.gz -C /usr/local/
bin/
bin/containerd-stress
bin/containerd-shim
bin/containerd-shim-runc-v1
bin/containerd-shim-runc-v2
bin/containerd
bin/ctr
$ containerd --version
containerd github.com/containerd/containerd v1.6.15 5b842e528e99d4d4c1686467debf2bd4b88ecd86
```
下载[containerd.service](https://raw.githubusercontent.com/containerd/containerd/main/containerd.service)文件，放到`/etc/systemd/system/`目录下。
接着，我们首先使用命令生成containerd的默认配置文件`/etc/containerd/config.toml`：
```
$ mkdir /etc/containerd
$ containerd config default > /etc/containerd/config.toml
```
默认的配置文件中，默认的容器运行时为runc。我们在该文件中的对应位置：（1）更改sanbox imgage为国内镜像，以免拉不下来（2）添加如下kata运行时的如下三行内容（内容可参考该样例配置[cri/config.md](https://github.com/containerd/containerd/blob/main/docs/cri/config.md)和[kata此文](https://github.com/kata-containers/kata-containers/blob/main/docs/install/container-manager/containerd/containerd-install.md#install-containerd)）
```
...
[plugins]
...
[plugins."io.containerd.grpc.v1.cri"]
...
# 更改为国内的镜像
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
...
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
# 添加如下三行
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
privileged_without_host_devices = false
...
```
启动containerd
```
$ systemctl daemon-reload
$ systemctl start containerd
$ systemctl enable containerd
```
接着我们来运行一个runc容器和kata容器试试，参考[kata-container/containerd-install/test-installation](https://github.com/kata-containers/kata-containers/blob/main/docs/install/container-manager/containerd/containerd-install.md#test-the-installation)
```
$ ctr image pull docker.io/library/busybox:latest
# 运行一个容器，未指定runtime，默认为runc，内核为主机内核
$ ctr run --rm -t docker.io/library/busybox:latest test-kata uname -r
5.4.228-1.el7.elrepo.x86_64
# 运行一个容器，指定runtime为kata，内核与主机内核不一样
$ ctr run --runtime "io.containerd.kata.v2" --rm -t docker.io/library/busybox:latest test-kata uname -r
5.19.2
```
五、安装kubernetes
创建`/etc/yum.repos.d/aliyun-kubernetes.repo`文件，内容如下：
```
[aliyun-kubernetes]
name=aliyun-kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0
enabled=1
```
然后执行命令安装指定版本的kubectl kubeadm kubelet
```
$ yum -y install kubelet-1.26.0 kubeadm-1.26.0 kubectl-1.26.0
```
接着，执行命令安装集群：
```
$ kubeadm init --cri-socket unix:///run/containerd/containerd.sock --kubernetes-version v1.26.0 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr 172.26.0.0/16
```
然后去掉污点（注意污点和以前的版本不一样了，以前是`node-role.kubernetes.io/master`）：
```
$ kubectl taint nodes --all node-role.kubernetes.io/control-plane-
```
然后，下载 3.24版本的 [calico.yaml](
https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/calico.yaml) 文件，无须做任何更改（已经不用再改PodCIDR了），安装：
```
$ kubectl apply -f calico.yaml
```
然后查看容器状态，这些容器都是用runc运行的：
```
$ kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-7bdbfc669-lt995 1/1 Running 0 2m18s 172.26.167.131 kata01 <none> <none>
calico-node-qtllt 1/1 Running 0 2m18s 192.168.92.101 kata01 <none> <none>
coredns-5bbd96d687-824mc 1/1 Running 0 113m 172.26.167.129 kata01 <none> <none>
coredns-5bbd96d687-zb44d 1/1 Running 0 113m 172.26.167.130 kata01 <none> <none>
etcd-kata01 1/1 Running 0 113m 192.168.92.101 kata01 <none> <none>
kube-apiserver-kata01 1/1 Running 0 113m 192.168.92.101 kata01 <none> <none>
kube-controller-manager-kata01 1/1 Running 0 113m 192.168.92.101 kata01 <none> <none>
kube-proxy-5r994 1/1 Running 0 113m 192.168.92.101 kata01 <none> <none>
kube-scheduler-kata01 1/1 Running 0 113m 192.168.92.101 kata01 <none> <none>
```
六、发布kata容器
创建runtime-kata.yaml文件，内容如下：
```
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: kata
handler: kata
```
然后创建这个资源对象：
```
$ kubectl apply -f runtime-kata.yaml
```
接着，我们创建文件busybox.yaml，内容如下：
```
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
runtimeClassName: kata
containers:
- name: busybox
image: docker.io/library/busybox:latest
```
创建它，并查看Pod的状态：
```
$ kubectl apply -f busybox.yaml
$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 2m3s 172.26.167.132 kata01 <none> <none>
```
查看该容器的内核：
```
$ kubectl exec busybox -- uname -r
5.19.2
```
七、参考
* https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-k8s-with-containerd-and-kata.md
* https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/containerd-kata.md
* https://github.com/kata-containers/kata-containers/tree/main/docs/install
* https://github.com/kata-containers/kata-containers/blob/main/docs/install/container-manager/containerd/containerd-install.md
* https://katacontainers.io/docs/
