{"id":64397,"date":"2022-05-09T10:33:45","date_gmt":"2022-05-09T01:33:45","guid":{"rendered":"https:\/\/avinton.com\/?p=64397"},"modified":"2023-08-03T10:09:30","modified_gmt":"2023-08-03T01:09:30","slug":"how-to-run-kubernetes-on-the-edge","status":"publish","type":"post","link":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/","title":{"rendered":"How to Run Kubernetes on the Edge"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Setup\" >Setup<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Analysis\" >Analysis<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Deployment_and_management\" >Deployment and management<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Edge_Network\" >Edge Network<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Resource_and_performance\" >Resource and performance<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Customization\" >Customization<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Documentation_and_community\" >Documentation and community<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Summary_table\" >Summary table<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#Conclusion\" >Conclusion<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#References\" >References<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<div class=\"wpb-content-wrapper\"><p>[vc_row][vc_column][vc_column_text]<em>Kubernetes<\/em> itself is becoming more popular for many workloads, especially on the cloud or on-premise. On the other hand, there are some emerged solutions for running <em>Kubernetes<\/em> on edge and IoT devices. Edge computing has the benefit of low latency, low costs, small space, etc. Combining edge devices and cloud-native solutions, we will be able to enjoy many benefits.<\/p>\n<ul dir=\"auto\" data-sourcepos=\"5:1-11:0\">\n<li data-sourcepos=\"5:1-5:36\">Ease of deployment and management<\/li>\n<li data-sourcepos=\"6:1-6:13\">Automation<\/li>\n<li data-sourcepos=\"7:1-7:44\">Modern and cloud-native way of developing<\/li>\n<li data-sourcepos=\"8:1-8:15\">Observability<\/li>\n<li data-sourcepos=\"9:1-9:13\">Resilience<\/li>\n<li data-sourcepos=\"10:1-11:0\">Scalability<\/li>\n<\/ul>\n<p dir=\"auto\" data-sourcepos=\"12:1-12:646\">Avinton also provides <a href=\"https:\/\/avinton.com\/services\/edge-ai-camera\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\"><em>Edge AI Camera<\/em> solutions<\/a> for many use-cases which require real-time inference. But what happens if we want to deploy 100+ devices at the edge? If you have like 20+ devices to be deployed on the edge, setting up the devices would be very painful even with the provisioning or auto-installation methods. Also, the management costs of running devices would be larger and larger if you have more devices. By adopting <em>Kubernetes<\/em> into edge workload, we can reduce many manual tasks. The more devices we have at the edge, the more costs we can reduce by leveraging <em>Kubernetes<\/em>.<\/p>\n<p dir=\"auto\" data-sourcepos=\"14:1-14:204\">There are some available open-source <em>Kubernetes<\/em> distributions that are designed for edge or IoT use-cases. We did a test of these distros and compared the features, performance, deployment costs, etc.<\/p>\n<p dir=\"auto\" data-sourcepos=\"14:1-14:204\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-64435\" src=\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/kubernetes-on-the-edge-logos.png\" alt=\"How to Run Kubernetes on the Edge (k0s, k3s, microk8s, KubeEdge)\" width=\"878\" height=\"80\" srcset=\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/kubernetes-on-the-edge-logos.png 878w, https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/kubernetes-on-the-edge-logos-300x27.png 300w, https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/kubernetes-on-the-edge-logos-355x32.png 355w\" sizes=\"(max-width: 878px) 100vw, 878px\" \/><\/p>\n<ul dir=\"auto\" data-sourcepos=\"16:1-20:0\">\n<li data-sourcepos=\"16:1-16:87\"><a href=\"https:\/\/k0sproject.io\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">k0s<\/a>: The Simple, Solid &amp; Certified Kubernetes Distribution<\/li>\n<li data-sourcepos=\"17:1-17:95\"><a href=\"https:\/\/k3s.io\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">k3s<\/a>: The certified Kubernetes distribution built for IoT &amp; Edge computing<\/li>\n<li data-sourcepos=\"18:1-18:130\"><a href=\"https:\/\/microk8s.io\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">microk8s<\/a>: Low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT.<\/li>\n<li data-sourcepos=\"19:1-20:0\"><a href=\"https:\/\/kubeedge.io\/en\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">KubeEdge<\/a>: A Kubernetes Native Edge Computing Framework<\/li>\n<\/ul>\n<h2 class=\"header2\" dir=\"auto\" data-sourcepos=\"21:1-21:8\"><span class=\"ez-toc-section\" id=\"Setup\"><\/span>Setup<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p dir=\"auto\" data-sourcepos=\"22:1-22:41\">Our device configuration is as follows:<\/p>\n<ul dir=\"auto\" data-sourcepos=\"23:1-29:0\">\n<li data-sourcepos=\"23:1-23:30\">3 mini PCs as a worker node<\/li>\n<li data-sourcepos=\"24:1-24:28\">1 mini PC as a master node<\/li>\n<li data-sourcepos=\"25:1-25:42\">3 IP cameras for inference applications<\/li>\n<li data-sourcepos=\"26:1-26:37\">Network switch for interconnecting<\/li>\n<li data-sourcepos=\"27:1-27:28\">PoE switch for IP cameras<\/li>\n<li data-sourcepos=\"28:1-29:0\">Management VM &#8211; Running <em>Prometheus<\/em> and <em>Grafana<\/em> instances for monitoring, <em>ELK<\/em> stack for monitoring<\/li>\n<\/ul>\n<p dir=\"auto\" data-sourcepos=\"30:1-30:32\">The Mini PC&#8217;s spec is as below:<\/p>\n<ul dir=\"auto\" data-sourcepos=\"31:1-36:0\">\n<li data-sourcepos=\"31:1-31:13\">4 core CPU<\/li>\n<li data-sourcepos=\"32:1-32:14\">4GB Memory<\/li>\n<li data-sourcepos=\"33:1-33:14\">60GB Storage<\/li>\n<li data-sourcepos=\"34:1-34:22\">Ethernet connection<\/li>\n<li data-sourcepos=\"35:1-36:0\">Wifi available<\/li>\n<\/ul>\n<h2 dir=\"auto\" data-sourcepos=\"39:1-39:11\"><img decoding=\"async\" class=\"aligncenter wp-image-64403 size-full\" src=\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/Run-Kubernetes-on-Edge.png\" alt=\"How To Run Kubernetes on Edge\" width=\"637\" height=\"480\" srcset=\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/Run-Kubernetes-on-Edge.png 637w, https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/Run-Kubernetes-on-Edge-300x226.png 300w, https:\/\/avinton.com\/wp-content\/uploads\/2022\/04\/Run-Kubernetes-on-Edge-222x167.png 222w\" sizes=\"(max-width: 637px) 100vw, 637px\" \/><\/h2>\n<h2 class=\"header2\" dir=\"auto\" data-sourcepos=\"39:1-39:11\"><span class=\"ez-toc-section\" id=\"Analysis\"><\/span>Analysis<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p dir=\"auto\" data-sourcepos=\"40:1-40:72\">We analyzed and compared those distributions from these perspectives:<\/p>\n<ul dir=\"auto\" data-sourcepos=\"41:1-46:0\">\n<li data-sourcepos=\"41:1-41:35\">Deployment and management costs<\/li>\n<li data-sourcepos=\"42:1-42:15\">Edge Network<\/li>\n<li data-sourcepos=\"43:1-43:16\">Customization<\/li>\n<li data-sourcepos=\"44:1-44:27\">Resource and performance<\/li>\n<li data-sourcepos=\"45:1-46:0\">Documentation and community<\/li>\n<\/ul>\n<h3 dir=\"auto\" data-sourcepos=\"47:1-47:29\"><span class=\"ez-toc-section\" id=\"Deployment_and_management\"><\/span>Deployment and management<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p dir=\"auto\" data-sourcepos=\"48:1-50:75\"><em>k0s<\/em> and <em>k3s<\/em> can be deployed very easily and quickly.<br \/>\nThey are single binary that contains all of the necessary packages (kubelet, runc, k8s-api, etcd, etc).<br \/>\nVery easy to configure the cluster and uninstall step is also very clean.<\/p>\n<p dir=\"auto\" data-sourcepos=\"52:1-54:80\"><em>microk8s<\/em> is also a single package of <em>snap<\/em> but the processes run separately as systemd services.<br \/>\nConfigurations need to be applied per each component and restart respective systemd services.<br \/>\nThe uninstall command took very long and that is reported by the community too.<\/p>\n<p dir=\"auto\" data-sourcepos=\"56:1-59:84\"><em>kubeedge<\/em> can be installed by the command <code>keadm<\/code> which is a similar command to <code>kubeadm<\/code>.<br \/>\n<code>keadm<\/code> command can install both <em>cloudcore<\/em> or <em>edgecore<\/em>, <em>cloudcore<\/em> or <em>edgecore<\/em> run as a systemd process on the control plane or edge. However, the control plane nodes have to be created as native <em>Kubernetes<\/em> clusters before installing <em>kubeedge<\/em>.<br \/>\nThe latest version support running <em>cloudcore<\/em> as <em>Kubernetes<\/em> resources. (See <a href=\"https:\/\/github.com\/kubeedge\/kubeedge\/tree\/master\/build\/cloud\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">here<\/a>)<br \/>\nThe management cost is larger and installation steps are not as simpler as others.<\/p>\n<h3 dir=\"auto\" data-sourcepos=\"61:1-61:16\"><span class=\"ez-toc-section\" id=\"Edge_Network\"><\/span>Edge Network<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p dir=\"auto\" data-sourcepos=\"62:1-63:80\">Most of them support various CNIs so any favorite CNI can be installed.<br \/>\nSome CNIs (like Calico) require access from the edge node to <em>Kubernetes<\/em> API.<\/p>\n<p dir=\"auto\" data-sourcepos=\"65:1-66:157\">To run nodes in a different network (like the dynamic public IP address of LTE), <em>konnectivity<\/em> seems to be a good option.<br \/>\nkonnectivity is a server-client architecture, so the <em>konnectivity<\/em> agent on the edge device connects to the <em>konnectivity<\/em> server on the controller nodes.<\/p>\n<p dir=\"auto\" data-sourcepos=\"68:1-69:286\"><em>k0s<\/em> support <em>konnectivity<\/em> by default for a worker-to-controller communication. (See <a href=\"https:\/\/docs.k0sproject.io\/v1.21.2+k0s.1\/networking\/#controller-worker-communication\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">here<\/a>)<br \/>\nAlso, <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/architecture\/control-plane-node-communication\/#apiserver-to-nodes-pods-and-services\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">here<\/a> is the description of <em>konnectivity<\/em> from the <em>Kubernetes<\/em> official documentation. <em>konnectivity<\/em> can be enabled manually on any <em>Kubernetes<\/em> cluster.<\/p>\n<p dir=\"auto\" data-sourcepos=\"71:1-72:107\"><em>kubeedge<\/em> adopts a different kind of networking between controller and worker: <a href=\"https:\/\/kubeedge.io\/en\/docs\/architecture\/cloud\/cloudhub\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">cloudhub<\/a> and <a href=\"https:\/\/kubeedge.io\/docs\/architecture\/edge\/edgehub\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">edgehub<\/a><br \/>\nThis is a server-client type of architecture too and it allows to running of nodes in a different network.<\/p>\n<h3 dir=\"auto\" data-sourcepos=\"74:1-74:28\"><span class=\"ez-toc-section\" id=\"Resource_and_performance\"><\/span>Resource and performance<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p dir=\"auto\" data-sourcepos=\"75:1-75:111\">For <em>k0s<\/em> and <em>k3s<\/em>, the resource usage is reasonable enough. ~440MB memory usage on worker nodes when idle.<\/p>\n<p dir=\"auto\" data-sourcepos=\"77:1-77:56\">For <em>microk8s<\/em>, the memory usage is ~ 700MB when idle.<\/p>\n<p dir=\"auto\" data-sourcepos=\"79:1-79:57\">For <em>kubeedge<\/em>, the memory usage is ~ 300 MB when idle.<\/p>\n<p dir=\"auto\" data-sourcepos=\"81:1-82:215\">This is a result of perf-tests by using <a href=\"https:\/\/github.com\/kubernetes\/perf-tests\/tree\/master\/clusterloader2\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">the cluster loader script<\/a>.<br \/>\nOnly running simple test cases from <a href=\"https:\/\/github.com\/kubernetes\/perf-tests\/blob\/master\/clusterloader2\/docs\/GETTING_STARTED.md\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">getting-started guide<\/a>. This script creates a deployment and scales to 10 replicas.<\/p>\n<table dir=\"auto\" data-sourcepos=\"84:1-92:100\">\n<thead>\n<tr data-sourcepos=\"84:1-84:50\">\n<th data-sourcepos=\"84:2-84:13\">test cases<\/th>\n<th data-sourcepos=\"84:15-84:26\"><em>kubeedge<\/em><\/th>\n<th data-sourcepos=\"84:28-84:32\">k0s<\/th>\n<th data-sourcepos=\"84:34-84:38\">k3s<\/th>\n<th data-sourcepos=\"84:40-84:49\">microk8s<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr data-sourcepos=\"86:1-86:74\">\n<td data-sourcepos=\"86:2-86:21\">Start measurements<\/td>\n<td data-sourcepos=\"86:23-86:34\">0.101133 s<\/td>\n<td data-sourcepos=\"86:36-86:46\">0.10159 s<\/td>\n<td data-sourcepos=\"86:48-86:60\">0.101366 s<\/td>\n<td data-sourcepos=\"86:62-86:73\">0.101339 s<\/td>\n<\/tr>\n<tr data-sourcepos=\"87:1-87:68\">\n<td data-sourcepos=\"87:2-87:14\">Cluster API<\/td>\n<td data-sourcepos=\"87:16-87:27\">0.000221 s<\/td>\n<td data-sourcepos=\"87:29-87:41\">0.000172 s<\/td>\n<td data-sourcepos=\"87:43-87:54\">0.000165s<\/td>\n<td data-sourcepos=\"87:56-87:67\">0.000239 s<\/td>\n<\/tr>\n<tr data-sourcepos=\"88:1-88:75\">\n<td data-sourcepos=\"88:2-88:22\">Cluster performance<\/td>\n<td data-sourcepos=\"88:24-88:35\">0.000419 s<\/td>\n<td data-sourcepos=\"88:37-88:48\">0.000317 s<\/td>\n<td data-sourcepos=\"88:50-88:61\">0.000313 s<\/td>\n<td data-sourcepos=\"88:63-88:74\">0.000267 s<\/td>\n<\/tr>\n<tr data-sourcepos=\"89:1-89:73\">\n<td data-sourcepos=\"89:2-89:20\">Create deployment<\/td>\n<td data-sourcepos=\"89:22-89:33\">1.002712 s<\/td>\n<td data-sourcepos=\"89:35-89:46\">1.003559 s<\/td>\n<td data-sourcepos=\"89:48-89:59\">1.001963 s<\/td>\n<td data-sourcepos=\"89:61-89:72\">1.008023 s<\/td>\n<\/tr>\n<tr data-sourcepos=\"90:1-90:75\">\n<td data-sourcepos=\"90:2-90:30\">Wait for pods to be running<\/td>\n<td data-sourcepos=\"90:32-90:48\"><strong>14.079484 s<\/strong><\/td>\n<td data-sourcepos=\"90:50-90:61\">5.013258 s<\/td>\n<td data-sourcepos=\"90:63-90:73\">5.01466 s<\/td>\n<td data-sourcepos=\"90:0-90:0\"><\/td>\n<\/tr>\n<tr data-sourcepos=\"91:1-91:83\">\n<td data-sourcepos=\"91:2-91:30\">Measure pod startup latency<\/td>\n<td data-sourcepos=\"91:32-91:43\">0.019507 s<\/td>\n<td data-sourcepos=\"91:45-91:56\">0.018685 s<\/td>\n<td data-sourcepos=\"91:58-91:69\">0.029368 s<\/td>\n<td data-sourcepos=\"91:71-91:82\">0.021512 s<\/td>\n<\/tr>\n<tr data-sourcepos=\"92:1-92:100\">\n<td data-sourcepos=\"92:2-92:45\">test overall (including clean up duration)<\/td>\n<td data-sourcepos=\"92:47-92:60\"><strong>35.261 s<\/strong><\/td>\n<td data-sourcepos=\"92:62-92:74\">21.188928 s<\/td>\n<td data-sourcepos=\"92:76-92:88\">21.231549 s<\/td>\n<td data-sourcepos=\"92:90-92:99\">21.256 s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p dir=\"auto\" data-sourcepos=\"94:1-94:104\">From the result, we did not find many things, but only <em>kubeedge<\/em> scores some delay in pod deployment.<\/p>\n<h3 dir=\"auto\" data-sourcepos=\"96:1-96:17\"><span class=\"ez-toc-section\" id=\"Customization\"><\/span>Customization<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p dir=\"auto\" data-sourcepos=\"97:1-97:70\">Most of the distros allow much variety of choices for CRI, CNI, CSI.<\/p>\n<h4 dir=\"auto\" data-sourcepos=\"99:1-99:8\">CRI<\/h4>\n<p dir=\"auto\" data-sourcepos=\"100:1-101:73\"><em>containerd<\/em> is supported as default for all 4 distros.<br \/>\n<em>k0s<\/em>, <em>k3s<\/em>, <em>microk8s<\/em> install <em>containerd<\/em> automatically by default.<\/p>\n<p dir=\"auto\" data-sourcepos=\"103:1-104:51\"><em>cri-o<\/em> is also supported except by <em>microk8s<\/em> but some manual steps are required.<br \/>\nThey provide enough documentation to enable them.<\/p>\n<p dir=\"auto\" data-sourcepos=\"106:1-108:96\"><code>nvidia-container-runtime<\/code> can be enabled on <em>k0s<\/em> and <em>kubeege<\/em> manually.<br \/>\n<em>microk8s<\/em> provides an add-on for gpu.<br \/>\n<em>k3s<\/em> seems not officially support gpu but we can see some workaround on the web to enable it.<\/p>\n<h4 dir=\"auto\" data-sourcepos=\"110:1-110:8\">CNI<\/h4>\n<p dir=\"auto\" data-sourcepos=\"111:1-115:118\">Most of CNIs can be enabled manually.<br \/>\n<em>k0s<\/em> support <a href=\"https:\/\/github.com\/projectcalico\/calico\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">Calico<\/a> or <a href=\"https:\/\/github.com\/cloudnativelabs\/kube-router\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">kube-router<\/a> as default.<br \/>\n<em>k3s<\/em> support <a href=\"https:\/\/github.com\/flannel-io\/flannel\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">Flannel<\/a> as default CNI.<br \/>\n<em>microk8s<\/em> enables Flannel in non-HA-mode and enables Calico in HA-mode.<br \/>\n<em>Kubeege<\/em> provides <a href=\"https:\/\/edgemesh.netlify.app\/guide\/#dependencies\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">edgemesh<\/a> for default CNI.<\/p>\n<h4 dir=\"auto\" data-sourcepos=\"117:1-117:8\">CSI<\/h4>\n<p dir=\"auto\" data-sourcepos=\"118:1-121:100\">Most of CSIs can be enabled manually.<br \/>\n<em>k0s<\/em> offers <a href=\"https:\/\/openebs.io\/docs\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">OpenEBS<\/a> as a part of addons. This allows to leverage host path or device as a volume backend.<br \/>\n<em>k3s<\/em> support <a href=\"https:\/\/github.com\/longhorn\/longhorn\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">Longhorn<\/a> (which also originated in Rancher).<\/p>\n<h4 dir=\"auto\" data-sourcepos=\"123:1-123:7\">HA<\/h4>\n<p dir=\"auto\" data-sourcepos=\"124:1-124:64\">High availability configuration can be enabled on all distros.<\/p>\n<h3 dir=\"auto\" data-sourcepos=\"126:1-126:31\"><span class=\"ez-toc-section\" id=\"Documentation_and_community\"><\/span>Documentation and community<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p dir=\"auto\" data-sourcepos=\"127:1-128:76\">For <em>k0s<\/em> and <em>k3s<\/em>, documentation is well structured and completed.<br \/>\nAlso, the community on Github is active and the release cycle is frequent.<\/p>\n<p dir=\"auto\" data-sourcepos=\"130:1-130:44\">Documentations for <em>microk8s<\/em> are good too.<\/p>\n<p dir=\"auto\" data-sourcepos=\"132:1-133:65\"><em>kubeedge<\/em>&#8216;s documentation is a bit hard to follow.<br \/>\nSome pages are stale and do not reflect the latest information.<\/p>\n<h2 class=\"header2\" dir=\"auto\" data-sourcepos=\"135:1-135:16\"><span class=\"ez-toc-section\" id=\"Summary_table\"><\/span>Summary table<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p dir=\"auto\" data-sourcepos=\"137:1-137:72\">This is the summary table of basic features at this moment (Apr 2022).<\/p>\n<table dir=\"auto\" data-sourcepos=\"139:1-146:142\">\n<thead>\n<tr data-sourcepos=\"139:1-139:92\">\n<th data-sourcepos=\"139:2-139:5\"><\/th>\n<th data-sourcepos=\"139:7-139:42\"><em>kubeedge<\/em><\/th>\n<th data-sourcepos=\"139:44-139:51\">k3s<\/th>\n<th data-sourcepos=\"139:53-139:66\">k0s<\/th>\n<th data-sourcepos=\"139:68-139:91\">microk8s<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr data-sourcepos=\"141:1-141:112\">\n<td data-sourcepos=\"141:2-141:28\">Deployment costs<\/td>\n<td data-sourcepos=\"141:30-141:62\">complex<\/td>\n<td data-sourcepos=\"141:64-141:71\">simple<\/td>\n<td data-sourcepos=\"141:73-141:86\">simple<\/td>\n<td data-sourcepos=\"141:88-141:111\">simple<\/td>\n<\/tr>\n<tr data-sourcepos=\"142:1-142:112\">\n<td data-sourcepos=\"142:2-142:28\">Configuraion<\/td>\n<td data-sourcepos=\"142:30-142:62\">complex<\/td>\n<td data-sourcepos=\"142:64-142:71\">simple<\/td>\n<td data-sourcepos=\"142:73-142:86\">simple<\/td>\n<td data-sourcepos=\"142:88-142:111\">separated config files<\/td>\n<\/tr>\n<tr data-sourcepos=\"143:1-143:112\">\n<td data-sourcepos=\"143:2-143:28\">Documentation<\/td>\n<td data-sourcepos=\"143:30-143:62\">available but not sufficient<\/td>\n<td data-sourcepos=\"143:64-143:71\">good<\/td>\n<td data-sourcepos=\"143:73-143:86\">good<\/td>\n<td data-sourcepos=\"143:88-143:111\">good<\/td>\n<\/tr>\n<tr data-sourcepos=\"144:1-144:114\">\n<td data-sourcepos=\"144:2-144:28\">Isolated network<\/td>\n<td data-sourcepos=\"144:30-144:62\">cloudhub &#8211; edgehub<\/td>\n<td data-sourcepos=\"144:64-144:71\">&#8211;<\/td>\n<td data-sourcepos=\"144:73-144:88\"><em>konnectivity<\/em><\/td>\n<td data-sourcepos=\"144:90-144:113\">&#8211;<\/td>\n<\/tr>\n<tr data-sourcepos=\"145:1-145:112\">\n<td data-sourcepos=\"145:2-145:28\">Idle Resource utilization<\/td>\n<td data-sourcepos=\"145:30-145:62\">~300MB<\/td>\n<td data-sourcepos=\"145:64-145:71\">~440MB<\/td>\n<td data-sourcepos=\"145:73-145:86\">~440MB<\/td>\n<td data-sourcepos=\"145:88-145:111\">~700MB<\/td>\n<\/tr>\n<tr data-sourcepos=\"146:1-146:142\">\n<td data-sourcepos=\"146:2-146:28\">Performance<\/td>\n<td data-sourcepos=\"146:30-146:62\">a bit slow for pod initializing<\/td>\n<td data-sourcepos=\"146:64-146:71\">fast<\/td>\n<td data-sourcepos=\"146:73-146:85\">fast<\/td>\n<td data-sourcepos=\"146:87-146:141\">fast but slow for stopping the node<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2 class=\"header2\" dir=\"auto\" data-sourcepos=\"148:1-148:13\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p dir=\"auto\" data-sourcepos=\"149:1-149:210\"><em>Kubernetes<\/em> now can be running on the edge devices which have fewer resources. For edge solutions, <em>Kubernetes<\/em> will help to reduce the management and deployment costs in case we have more and more devices.<\/p>\n<p dir=\"auto\" data-sourcepos=\"151:1-151:370\">For edge use-cases, <em>k0s<\/em> seems to be the best for its simplicity and features at this moment (Apr 2022). It&#8217;s easy to deploy and manage and has only one binary. So even application developers without much <em>Kubernetes<\/em> experience can operate clusters easily by themselves. k0s allows managing a cluster from the isolated control plane nodes with default <em>konnectivity<\/em>.<\/p>\n<p dir=\"auto\" data-sourcepos=\"153:1-153:493\">Although the complexity of the operation and the documentation, <em>kubeedge<\/em> also supports running nodes in a different network and provides many flexible features for edge use-case (device CRD, queuing system, etc). Since it is an extension of native <em>Kubernetes<\/em> and the control plane is a native <em>Kubernetes<\/em> cluster, we can easily integrate with existing Kubernetes-based solutions\/applications. This requires some <em>Kubernetes<\/em> engineers and some more work than other solutions like <em>k0s<\/em>.<\/p>\n<p dir=\"auto\" data-sourcepos=\"155:1-155:184\">Through this test, we can see the community is trying to run more workloads on the edge. We believe this area grows up fast, and new interesting solutions emerge in the coming years.<\/p>\n<h3 dir=\"auto\" data-sourcepos=\"157:1-157:14\"><span class=\"ez-toc-section\" id=\"References\"><\/span>References<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul dir=\"auto\" data-sourcepos=\"158:1-160:233\">\n<li data-sourcepos=\"158:1-158:126\"><a href=\"https:\/\/developer.nvidia.com\/blog\/the-future-of-edge-ai-is-cloud-native\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">The Future of Edge AI is Cloud-Native<\/a> by Nvidia<\/li>\n<li data-sourcepos=\"159:1-159:219\"><a href=\"https:\/\/cloud.google.com\/blog\/topics\/developers-practitioners\/small-footprint-big-impact-running-cloud-connected-kubernetes-edge\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">Small footprint, big impact: running cloud-connected <em>Kubernetes<\/em> at the edge<\/a> by GCP<\/li>\n<li data-sourcepos=\"160:1-160:233\"><a href=\"https:\/\/www.cncf.io\/blog\/2021\/05\/04\/kubernetes-at-the-edge-organizations-are-using-edge-technologies-but-there-is-room-to-grow\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">Kubernetes at the Edge: Organizations are using edge technologies, but there is room to grow<\/a> by CNCF<\/li>\n<\/ul>\n<p>[\/vc_column_text][\/vc_column][\/vc_row]<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>[vc_row][vc_column][vc_column_text]Kubernetes itself is becoming more popular for many workloads, especially on the cloud or on-premise. On the other hand, there are some emerged solutions for running Kubernetes on edge and IoT devices. Edge computing has the benefit of low latency, low costs, small space, etc. Combining edge devices and cloud-native solutions, we will be able<br \/><a href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/\" class=\"more\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":65108,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,1830,1864],"tags":[1727,1764,1832],"class_list":["post-64397","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","category--en","category-technical","tag-1727","tag-edge-ai","tag-kubernetes-en"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to Run Kubernetes on the Edge - Avinton Japan<\/title>\n<meta name=\"description\" content=\"The more devices we have at the edge, the more costs we can reduce by leveraging Kubernetes. There are some available distributions that are designed for edge or IoT use-cases. We compared them.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Run Kubernetes on the Edge - Avinton Japan\" \/>\n<meta property=\"og:description\" content=\"The more devices we have at the edge, the more costs we can reduce by leveraging Kubernetes. There are some available distributions that are designed for edge or IoT use-cases. We compared them.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/\" \/>\n<meta property=\"og:site_name\" content=\"Avinton Japan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Avintons\/\" \/>\n<meta property=\"article:published_time\" content=\"2022-05-09T01:33:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-08-03T01:09:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1338\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"James Cauchi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AvintonJapan\" \/>\n<meta name=\"twitter:site\" content=\"@AvintonJapan\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"James Cauchi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/\",\"url\":\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/\",\"name\":\"How to Run Kubernetes on the Edge - Avinton Japan\",\"isPartOf\":{\"@id\":\"https:\/\/avinton.com\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg\",\"datePublished\":\"2022-05-09T01:33:45+00:00\",\"dateModified\":\"2023-08-03T01:09:30+00:00\",\"author\":{\"@id\":\"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b\"},\"description\":\"The more devices we have at the edge, the more costs we can reduce by leveraging Kubernetes. There are some available distributions that are designed for edge or IoT use-cases. We compared them.\",\"breadcrumb\":{\"@id\":\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#primaryimage\",\"url\":\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg\",\"contentUrl\":\"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg\",\"width\":1338,\"height\":630,\"caption\":\"How to Run Kubernetes on the Edge\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/avinton.com\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Run Kubernetes on the Edge\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/avinton.com\/en\/#website\",\"url\":\"https:\/\/avinton.com\/en\/\",\"name\":\"Avinton Japan\",\"description\":\"Tailored Solutions and Consulting in AI and Big Data\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/avinton.com\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b\",\"name\":\"James Cauchi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/avinton.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g\",\"caption\":\"James Cauchi\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Run Kubernetes on the Edge - Avinton Japan","description":"The more devices we have at the edge, the more costs we can reduce by leveraging Kubernetes. There are some available distributions that are designed for edge or IoT use-cases. We compared them.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/","og_locale":"en_US","og_type":"article","og_title":"How to Run Kubernetes on the Edge - Avinton Japan","og_description":"The more devices we have at the edge, the more costs we can reduce by leveraging Kubernetes. There are some available distributions that are designed for edge or IoT use-cases. We compared them.","og_url":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/","og_site_name":"Avinton Japan","article_publisher":"https:\/\/www.facebook.com\/Avintons\/","article_published_time":"2022-05-09T01:33:45+00:00","article_modified_time":"2023-08-03T01:09:30+00:00","og_image":[{"width":1338,"height":630,"url":"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg","type":"image\/jpeg"}],"author":"James Cauchi","twitter_card":"summary_large_image","twitter_creator":"@AvintonJapan","twitter_site":"@AvintonJapan","twitter_misc":{"Written by":"James Cauchi","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/","url":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/","name":"How to Run Kubernetes on the Edge - Avinton Japan","isPartOf":{"@id":"https:\/\/avinton.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#primaryimage"},"image":{"@id":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#primaryimage"},"thumbnailUrl":"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg","datePublished":"2022-05-09T01:33:45+00:00","dateModified":"2023-08-03T01:09:30+00:00","author":{"@id":"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b"},"description":"The more devices we have at the edge, the more costs we can reduce by leveraging Kubernetes. There are some available distributions that are designed for edge or IoT use-cases. We compared them.","breadcrumb":{"@id":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#primaryimage","url":"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg","contentUrl":"https:\/\/avinton.com\/wp-content\/uploads\/2022\/05\/kubernetes-on-the-edge-en.jpg","width":1338,"height":630,"caption":"How to Run Kubernetes on the Edge"},{"@type":"BreadcrumbList","@id":"https:\/\/avinton.com\/en\/blog\/2022\/05\/how-to-run-kubernetes-on-the-edge\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/avinton.com\/en\/"},{"@type":"ListItem","position":2,"name":"How to Run Kubernetes on the Edge"}]},{"@type":"WebSite","@id":"https:\/\/avinton.com\/en\/#website","url":"https:\/\/avinton.com\/en\/","name":"Avinton Japan","description":"Tailored Solutions and Consulting in AI and Big Data","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/avinton.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b","name":"James Cauchi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/avinton.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g","caption":"James Cauchi"}}]}},"_links":{"self":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts\/64397","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/comments?post=64397"}],"version-history":[{"count":9,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts\/64397\/revisions"}],"predecessor-version":[{"id":84155,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts\/64397\/revisions\/84155"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/media\/65108"}],"wp:attachment":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/media?parent=64397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/categories?post=64397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/tags?post=64397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}