Yahoo Developer Network

Latest Blogs

May 28, 2020

Kata Containers in Screwdriver

Screwdriver is a scalable CI/CD solution which uses Kubernetes to manage user builds. Screwdriver build workers interfaces with Kubernetes using either “<a href="https://github.com/screwdriver-cd/executor-k8s">executor-k8s</a>” or “<a href="https://github.com/screwdriver-cd/executor-k8s-vm/">executor-k8s-vm</a>” depending on required build isolation. <a href="https://github.com/screwdriver-cd/executor-k8s">executor-k8s</a> runs builds directly as Kubernetes pods while <a href="https://github.com/screwdriver-cd/executor-k8s-vm/">executor-k8s-vm</a> uses <a href="https://github.com/hyperhq/hyperd">HyperContainers</a> along with Kubernetes for stricter build isolation with containerized Virtual Machines (VMs). This setup was ideal for running builds in an isolated, ephemeral, and lightweight environment. However, <a href="https://github.com/hyperhq/hyperd">HyperContainer</a> is now deprecated, has no support, is based on an older Docker runtime and it also required non-native Kubernetes setup for build execution. Therefore, it was time to find a new solution.<h2> </h2><h2>Why Kata Containers ? </h2><a href="https://katacontainers.io/">Kata Containers</a> is an open source project and community that builds a standard implementation of lightweight virtual machines (VMs) that perform like containers, but provide the workload isolation and security advantages of VMs. It combines the benefits of using a hypervisor, such as enhanced security, along with container orchestration capabilities provided by Kubernetes. It is the same team behind HyperD where they successfully merged the best parts of Intel Clear Containers with Hyper.sh RunV. As a Kubernetes runtime, Kata enables us to deprecate <a href="https://github.com/screwdriver-cd/executor-k8s-vm/">executor-k8s-vm</a> and use <a href="https://github.com/screwdriver-cd/executor-k8s">executor-k8s</a> exclusively for all Kubernetes based builds. <h2> </h2><h2>Screwdriver journey to Kata</h2>As we faced a growing number of instabilities with the current HyperD - like network and devicemapper issues and IP cleanup workarounds, we started our initial evaluation of Kata in early 2019 ( <a href="https://github.com/screwdriver-cd/screwdriver/issues/818#issuecomment-482239236">https://github.com/screwdriver-cd/screwdriver/issues/818#issuecomment-482239236</a>) and identified two major blockers to move ahead with Kata:1. Security concern for privileged mode (required to run docker daemon in kata)2. Disk performance. We recently started reevaluating Kata in early 2020 based on a fix to “add flag to overload default privileged host device behaviour” provided by Containerd/cri (<a href="https://github.com/containerd/cri/pull/1225">https://github.com/containerd/cri/pull/1225</a>), but still we faced issues with disk performance and switched from overlayfs to devicemapper, which yielded significant improvement. With our two major blockers resolved and initial tests with Kata looking promising, we moved ahead with Kata.<h2>Screwdriver Build Architecture</h2>Replacing Hyper with Kata led to a simpler build architecture. We were able to remove the custom build setup scripts to launch Hyper VM and rely on native Kubernetes setup. <figure class="tmblr-full" data-orig-height="1508" data-orig-width="2096"><img src="https://64.media.tumblr.com/1f90ef88dcadeb8cb2a38e7d7d1aeeff/a74849879cd376a2-8e/s540x810/7277a8c414aa4bf7cff2f17da535d4402508aee8.png" data-orig-height="1508" data-orig-width="2096"/></figure><div> </div><h2>Setup </h2>To use Kata containers for running user builds in a Screwdriver Kubernetes build cluster, a cluster admin needs to configure Kubernetes to use Containerd container runtime with Cri-plugin.<blockquote>Components:Screwdriver build Kubernetes cluster (minimum version: 1.14+) nodes must have the following components set up for using Kata containers for user builds. Containerd:Containerd is a container runtime that helps with management of the complete lifecycle of the container.Reference: <a href="https://containerd.io/docs/getting-started/">https://containerd.io/docs/getting-started/</a> CRI-Containerd plugin:Cri-Containerd is a containerd plugin which implements Kubernetes container runtime interface. CRI plugin interacts with containerd to manage the containers.Reference: <a href="https://github.com/containerd/cri">https://github.com/containerd/cri</a> </blockquote><figure data-orig-width="1076" data-orig-height="204" class="tmblr-full"><img src="https://64.media.tumblr.com/03500a7dc20b400cd0e6f1fd71c13164/a74849879cd376a2-72/s540x810/f069b15e83dc8943ade421c5199972de78449bab.png" alt="image" data-orig-width="1076" data-orig-height="204"/></figure><blockquote>Image credit: <a href="https://github.com/containerd/cri/blob/master/docs/cri.png">containerd / cri</a>. Photo licensed under <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY-4.0</a>. Architecture:</blockquote><figure data-orig-width="1242" data-orig-height="484" class="tmblr-full"><img src="https://64.media.tumblr.com/fe8ab79a371657e4e1f52100e68245d8/a74849879cd376a2-e7/s540x810/2559f2427d95a5bda765821d358b7562d9e8bc2b.png" alt="image" data-orig-width="1242" data-orig-height="484"/></figure><blockquote>Image credit: <a href="https://github.com/containerd/cri/blob/master/docs/cri.png">containerd / cri</a>. Photo licensed under <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY-4.0</a>. Installation: Reference: <a href="https://github.com/containerd/cri/blob/master/docs/installation.md">https://github.com/containerd/cri/blob/master/docs/installation.md</a>, <a href="https://github.com/containerd/containerd/blob/master/docs/ops.md">https://github.com/containerd/containerd/blob/master/docs/ops.md</a>Crictl: To debug, inspect, and manage their pods, containers, and container images. Reference: <a href="https://github.com/containerd/cri/blob/master/docs/crictl.md">https://github.com/containerd/cri/blob/master/docs/crictl.md</a> Kata:Builds lightweight virtual machines that seamlessly plugin to the containers ecosystem. Architecture:</blockquote><figure data-orig-width="1242" data-orig-height="542" class="tmblr-full"><img src="https://64.media.tumblr.com/5e9b997dd94546aae737826be37b1fc5/a74849879cd376a2-ca/s540x810/e2d49003fbe2bd3c92590e668d67d1daa8f3239f.png" alt="image" data-orig-width="1242" data-orig-height="542"/></figure><blockquote>Image credit: <a href="https://github.com/kata-containers/documentation/blob/master/design/arch-images/shimv2.svg">kata-containers</a> Project licensed under <a href="https://opensource.org/licenses/Apache-2.0">Apache License Version 2.0</a> Installation:</blockquote><ul><li><a href="https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes">https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes</a></li><li><a href="https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md">https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md</a></li><li><a href="https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md">https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md</a></li><li><a href="https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md#kubernetes-runtimeclass">https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md#kubernetes-runtimeclass</a></li><li><a href="https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md#configuration">https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md#configuration</a></li></ul> <h2>Routing builds to Kata in Screwdriver build cluster</h2>Screwdriver uses <a href="https://kubernetes.io/docs/concepts/containers/runtime-class/">Runtime Class</a> to route builds to Kata nodes in Screwdriver build clusters. The Screwdriver plugin <a href="https://github.com/screwdriver-cd/executor-k8s">executor-k8s</a> config handles this based on: 1. Pod configuration:apiVersion: v1 kind: Pod metadata: name: kata-pod namespace: sd-build-namespace labels: sdbuild: “sd-kata-build” app: screwdriver tier: builds spec: runtimeClassName: kata containers: - name: “sd-build-container” image: <<image>> imagePullPolicy: IfNotPresent 2. Update the plugin to use k8s in your <a href="https://github.com/screwdriver-cd/buildcluster-queue-worker/">buildcluster-queue-worker</a> configuration <a href="https://github.com/screwdriver-cd/buildcluster-queue-worker/blob/master/config/custom-environment-variables.yaml#L7"></a><a href="https://github.com/screwdriver-cd/buildcluster-queue-worker/blob/master/config/custom-environment-variables.yaml#L4-L83">https://github.com/screwdriver-cd/buildcluster-queue-worker/blob/master/config/custom-environment-variables.yaml#L4-L83</a> <h2>Performance</h2>The below tables compare build setup and overall execution time for Kata and Hyper when the image is pre-cached or not cached.<figure data-orig-width="1128" data-orig-height="564" class="tmblr-full"><img src="https://64.media.tumblr.com/4fd90d8cfa4f7a6e886d616c232fd9b7/a74849879cd376a2-ff/s540x810/521baf90865560d2270972699f6cf0633f6aca15.png" alt="image" data-orig-width="1128" data-orig-height="564"/></figure><h2> </h2><h2>Known problems</h2>While the new Kata implementation offers a lot of advantages, there are some known problems we are aware of with fixes or workarounds:<ul><li>Run images based on Rhel6 containers don’t start and immediately exit </li><li>Pre-2.15 glibc: Enabled kernel_params = “vsyscall=emulate” refer kata issue <a href="https://github.com/kata-containers/runtime/issues/1916">https://github.com/kata-containers/runtime/issues/1916</a> if trouble running pre-2.15 glibc. </li><li>Yum install will hang forever: Enabled kernel_params = “init=/usr/bin/kata-agent” refer kata issue <a href="https://github.com/kata-containers/runtime/issues/1916">https://github.com/kata-containers/runtime/issues/1916</a> to get a better boot time, small footprint. </li><li>32-bit executable cannot be loaded refer kata issue <a href="https://github.com/kata-containers/runtime/issues/886">https://github.com/kata-containers/runtime/issues/886</a>: To workaround/mitigate we maintain a container exclusion list and route to current hyperd setup and we have plans to eol these containers by Q4 of this year.</li><li>Containerd IO snapshotter - Overlayfs vs devicemapper for storage driver: Devicemapper gives better performance. Overlayfs took 19.325605 seconds to write 1GB, but Devicemapper only took 5.860671 seconds.</li></ul> <h2>Compatibility List</h2>In order to use this feature, you will need these minimum versions:<ul><li><a href="https://hub.docker.com/r/screwdrivercd/screwdriver">API</a> - v0.5.902</li><li><a href="https://hub.docker.com/r/screwdrivercd/ui">UI</a> - v1.0.515</li><li><a href="https://hub.docker.com/r/screwdrivercd/buildcluster-queue-worker">Build Cluster Queue Worker</a> - v1.18.0</li><li><a href="https://hub.docker.com/r/screwdrivercd/launcher">Launcher</a> - v6.0.71</li></ul><h2> </h2><h2>Contributors</h2>Thanks to the following contributors for making this feature possible:<ul><li><a href="https://github.com/parthasl">Lakshminarasimhan Parthasarathy</a></li><li><a href="https://github.com/sureshvis"> Suresh Visvanathan</a> </li><li><a href="https://github.com/vnandha">Nandhakumar Venkatachalam</a> </li><li><a href="https://github.com/pritamstyz4ever">Pritam Paul</a></li><li><a href="https://github.com/chestery">Chester Yuan</a></li><li><a href="https://github.com/minz1027">Min Zhang</a></li></ul><h2> </h2><h2>Questions & Suggestions</h2>We’d love to hear from you. If you have any questions, please feel free to reach out <a href="https://docs.screwdriver.cd/about/support">here</a>. You can also visit us on <a href="https://github.com/screwdriver-cd">Github</a> and <a href="https://slack.screwdriver.cd">Slack</a>.