Fresh/Brewed

K3os, a promising start

Published: Dec 22, 2019 by Isaac Johnson

Rancher, best known for their flagship product Rancher, also makes a couple of distributions of kubernetes.  The one we’ve covered the most here is k3s, but they also make RKE (Rancher Kubernetes Engine) and more recently, k3os.  

Unlike RKE which is oft installed on RancherOS, K3os is designed to be solely managed from within k8s (which means logging into the host is really logging into a container).  All aspects are to be managed from the container in order to “remove as much OS maintenance as possible” from a managed cluster.

Installing K3OS:

All the details you need are in the github repo:
https://github.com/rancher/k3os/blob/master/README.md

To install, you need to provide a URL the host can see to a cloud-config.yaml file - this could be on a private nginx in-house or for ease,  you could use pastebin as well.

I used pastebin under my google for 1hour file and then used the “raw” link in the installer.

ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDJcNg9N4Ndtr3SZdfI1BWoVYY+/R1pXkwh+/5Gvc0nKMZQhIWnXCEw4DQe8Q6l1MUyi+pTHJKlA1Fiivrky8mofPAO5U/yIHWWhwXdypdzVtvBxRq/aBdCLmI/coXFCTGz0aYNqumP08SIk66PGggnIsTqYYgKaFLnAJw2K9faWyiTrmEO7ggu1fesL5VR6MGAqpgfyHEMv2x21oZD/WmtecsURxbJUHyH9yvXfXSVbpXI9kHWpLEc228xi72ZyIQTTHSuNGHzE7KTyOn93aFhbr4VoMkCWiq2Yxh6Xzzyysmu49sHs5DOX975jhmYSmwJqXMDRg5FnGw8GeZc8kV
- github:idjohnson
write_files:
- encoding: ""
  content: |-
    #!/bin/bash
    echo hi
  owner: root
  path: /etc/rc.local
  permissions: '0755'
hostname: linuxkit-54bf6465f26d.ent.core.medtronic.com
run_cmd:
- "echo hi && echo bye"
boot_cmd:
- "echo hi && echo bye"
init_cmd:
- "echo hi && echo bye"

k3os:
  data_sources:
  - aws
  - cdrom
  modules:
  - kvm
  - nvme
  dns_nameservers:
  - 8.8.8.8
  - 1.1.1.1
  - 10.1.1.1
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  password: r4nch3r1
  server_url: https://linuxkit-54bf6465f26d.house.comcast.net:6443
  token: d69c1e6e89409e1b4e2262b5a08452ad
  labels:
    region: us-west-1
    environment: dev

Note, if you want a private DNS (like the one inside your network or router, put that in the dns_nameservers list, eg. 10.1.1.1 above).

Download the ISO from the releases page. In our case, we’ll use the amd64 iso which I then burned to a thumb drive and installed via that.

However… The above config failed.. The token does funny things, from what I experienced, so I removed that.

Also, if you don’t set a server URL, it will just use the IP which is fine. In the end, i used something like this for the cloud-config:

ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDJcNg9N4Ndtr3SZdfI1BWoVYY+/R1pXkwh+/5Gvc0nKMZQhIWnXCEw4DQe8Q6l1MUyi+pTHJKlA1Fiivrky8mofPAO5U/yIHWWhwXdypdzVtvBxRq/aBdCLmI/coXFCTGz0aYNqumP08SIk66PGggnIsTqYYgKaFLnAJw2K9faWyiTrmEO7ggu1fesL5VR6MGAqpgfyHEMv2x21oZD/WmtecsURxbJUHyH9yvXfXSVbpXI9kHWpLEc228xi72ZyIQTTHSuNGHzE7KTyOn93aFhbr4VoMkCWiq2Yxh6Xzzyysmu49sHs5DOX975jhmYSmwJqXMDRg5FnGw8GeZc8kV
- github:idjohnson

k3os:
  data_sources:
  - aws
  - cdrom
  modules:
  - kvm
  - nvme
  dns_nameservers:
  - 8.8.8.8
  - 1.1.1.1
  - 10.48.246.170
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  password: r4nch3r1

This then created a k3s with a kube config such as this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJWekNCL3FBREFnRUNBZ0VBTUFvR0NDcUdTTTQ5QkFNQ01DTXhJVEFmQmdOVkJBTU1HR3N6Y3kxelpYSjIKWlhJdFkyRkFNVFUzTmpJME5qQTBOREFlRncweE9URXlNVE14TkRBM01qUmFGdzB5T1RFeU1UQXhOREEzTWpSYQpNQ014SVRBZkJnTlZCQU1NR0dzemN5MXpaWEoyWlhJdFkyRkFNVFUzTmpJME5qQTBOREJaTUJNR0J5cUdTTTQ5CkFnRUdDQ3FHU000OUF3RUhBMElBQkFSWFp4UmtSRFVIby9CaHVmNm45dUxaZjJrOENMYzVFL0lldDVxVWNBOS8KSkJoRkhYa2FLVkZzbmtJWkRxalhNU3lpeDcwUVcwRGxHRWFMbkVJQXdLQ2pJekFoTUE0R0ExVWREd0VCL3dRRQpBd0lDcERBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUFvR0NDcUdTTTQ5QkFNQ0EwZ0FNRVVDSUdGNlBuU0gra2wyCkxRV1Iyb0kxUUFFNk8xNjM4U2I3TzBMUlBwVm4wcDlEQWlFQTJjT3g3Rm54TEhiL1g4RVBHUjVVRlRZRm5WR0sKSWkwZEdwSlhDT3NZdWFRPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://10.1.2.3:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    password: d8bd1577621b5436892159c592137ebf
    username: admin

I want to bring up a quick point - the k3os system, upon boot will show an Ip on the screen. Something like “docker-sys: 172….”.  I kept thinking that was the IP of the host until it dawned on me that the range didn’t even line up with my local network topography.  So you either need to know ahead of time what the local IP is or use some other method to figure it out.  I recalled from booting to linux what the local IP was (10.x.x.x) and was able to SSH that way.

The reason this matters is:

  1. You’ll need to external facing IP to use the kubeconfig
  2. You’ll likely want to SSH back to this host to get the kubeconfig (or you’ll have to sftp it off the host)

Recall again, the login shell is actually a container.  (perhaps if i were to describe nodes, i would get the external IP. I’ll need to test that on the next pass)

Update: The newest version seems to solve this.  The UI has changed as well showing eth0:

Notes on usage:

In testing Elastic on our stack, we encountered an issue with ulimits (https://github.com/rancher/k3os/issues/87) when installing EFK.  This really became a deal breaker.  If your stack doesn’t have Java or has no need for large java containers, you likely won’t be blocked.

Installing into Windows 10

Download the iso from the releases page. In our case, the amd64 iso and launch hyper-v manager:

Click “Quick create…”

Then choose installation source and pick the iso:

Uncheck the “this virtual machine will run Windows” and then “Create Virtual Machine”.

When done, we should get the complete window:

Click connect then start…

When booted, the default is the Live install and Install option.  You can login with just “rancher”, no password needed.

At this point, you indeed have a functional k3s box.  However, upon reboot it will reset.

For testing, this might be fine.  But ideally we want to actually use this in a repeatable way.

Next steps, lets install a real server to use.. First, in windows, gen a ssh keypair

C:\Users\isaac>ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\isaac/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\isaac/.ssh/id_rsa.
Your public key has been saved in C:\Users\isaac/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Br4Jr2AzqhbxdZ1inCyfVs4fDyexmoizGiFFo5ympIY isaac@DESKTOP-2SQ9NQM
The key's randomart image is:
+---[RSA 2048]----+
| o |
|. + . |
| * . + o . |
|*.. + O + . |
|Eoo.o * S o |
|.....o B o = . |
| *. * . + * |
| + +.+ . o . . |
|= .o.o |
+----[SHA256]-----+

We can vim the .ssh/id_rsa.pub file to get it’s contents (or open in the editor of your choice):

Next, we go to pastebin.com to create a paste.  I like to login so that i can make it just last an hour.

ssh_authorized_keys:
- ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDihWpPWuhb/J/hB3D3jPVoUCG4gjaWknbnZOVgOKszV5UTgpAB2BnvOfMZMjWr43M/bb9tmBxCCw6Mw0m8phuaskHP6rrsCHKBQw7CXAmDkKgH8+6ul3MYhyiEC4tjuWQXd17oWo7RGbA/EE5hIJzx5fDBbe2qjPdVJKFUq8TEwzdHCT4LRDGjCUaeu1qhxmJszCsQaAJqH7T1ah8HvnM+x++pux0MXIMu3p7Ay098lYuO9RxHvcXW1IH5RrUV+cgWZcW2JZSnIiRn1KXfyUvZf/fpLK9nUvnSYW+Q2WPdJVhLUXQ6OWk2D6Z3M0Mv1r839g4V62gryBj3hKtHGkwj
- github:idjohnson

k3os:
  data_sources:
  - aws
  - cdrom
  modules:
  - kvm
  - nvme
  dns_nameservers:
  - 8.8.8.8
  - 1.1.1.1
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  password: r4nch3r1

Once pasted, you can click the “raw” button to show the raw files

i.e. https://pastebin.com/raw/nHCtY88y

We then used on a hyper-v however it did not come up after reboot.

Installing via Oracle VirtualBox

Create a new Debian based VM:

We can create as a VMDK (in case we want to move to an AMI)

Lastly, point the CD to the k3os ISO image:

Once booted, we can install with sudo os-config:

You will want to change the network to bridged mode so you can use the k3os cluster:

“ifconfig more”  to show the eth0 ip

Then you can SSH to it:

$ ssh rancher@192.168.1.179
The authenticity of host '192.168.1.179 (192.168.1.179)' can't be established.
ECDSA key fingerprint is SHA256:r0Li6X2ihLSDBuz1ZfUwv1EoeQ8t9ZX/gYIuVfTOpjk.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.179' (ECDSA) to the list of known hosts.
Welcome to k3OS!

Refer to https://github.com/rancher/k3os for README and issues

By default mode of k3OS is to run a single node cluster. Use "kubectl"
to access it. The node token in /var/lib/rancher/k3s/server/node-token
can be used to join agents to this server.

linuxkit-0800271f59d8 [~]$ ulimit -Hn
4096

As you can see, the kubeconfig we used did set the value, it just wasn’t respected:

linuxkit-0800271f59d8 [~]$ sudo cat /etc/sysctl.d/60-filemax.conf 
fs.file-max = 65000

Installing into Azure

Spin up a basic Ubuntu 18.04 VM.

Once up and logged in, we can look at our /etc/resolv.conf

nameserver 127.0.0.53
options edns0
search ys1jtr24esyujaxemedvzpbgqb.bx.internal.cloudapp.net

We’ll use this for the cloud config file:

In this case, we aren’t using pastebin, but rather storing locally

First, let’s add a couple settings (the sysctl block):

ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDJcNg9N4Ndtr3SZdfI1BWoVYY+/R1pXkwh+/5Gvc0nKMZQhIWnXCEw4DQe8Q6l1MUyi+pTHJKlA1Fiivrky8mofPAO5U/yIHWWhwXdypdzVtvBxRq/aBdCLmI/coXFCTGz0aYNqumP08SIk66PGggnIsTqYYgKaFLnAJw2K9faWyiTrmEO7ggu1fesL5VR6MGAqpgfyHEMv2x21oZD/WmtecsURxbJUHyH9yvXfXSVbpXI9kHWpLEc228xi72ZyIQTTHSuNGHzE7KTyOn93aFhbr4VoMkCWiq2Yxh6Xzzyysmu49sHs5DOX975jhmYSmwJqXMDRg5FnGw8GeZc8kV
- github:idjohnson

k3os:
  sysctl:
    kernel.printk: 4 4 1 7
    fs.file-max: 100000
  data_sources:
  - aws
  - cdrom
  modules:
  - kvm
  - nvme
  dns_nameservers:
  - 8.8.8.8
  - 1.1.1.1
  - 127.0.0.53
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  password: r4nch3r1

The way we installed k3os into Azure was to use a standard Azure VM with Ubuntu and then used the “k3os takeover” approach which uses the installer to install over top of an already existing linux node.

Once the node is up, ssh into it and run the following:

johnsi10@idj-k3os-001:~$ wget https://raw.githubusercontent.com/rancher/k3os/master/install.sh
--2019-12-16 13:14:27-- https://raw.githubusercontent.com/rancher/k3os/master/install.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.248.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.248.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8034 (7.8K) [text/plain]
Saving to: ‘install.sh’

install.sh 100%[=======================>] 7.85K --.-KB/s in 0s      

2019-12-16 13:14:27 (62.4 MB/s) - ‘install.sh’ saved [8034/8034]

johnsi10@idj-k3os-001:~$ chmod 755 ./install.sh 
johnsi10@idj-k3os-001:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 796M 660K 795M 1% /run
/dev/sda1 29G 1.2G 28G 5% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda15 105M 3.6M 101M 4% /boot/efi
/dev/sdb1 16G 45M 15G 1% /mnt
tmpfs 796M 0 796M 0% /run/user/1000
johnsi10@idj-k3os-001:~$ sudo ./install.sh --takeover --debug --tty ttyS0 --config /tmp/config.yaml --no-format /dev/sda1 https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso
...snip...
+ cleanup
+ EXIT=0
+ cleanup2
+ return 0
johnsi10@idj-k3os-001:~$ sudo reboot
Connection to 40.71.228.61 closed by remote host.
Connection to 40.71.228.61 closed.

Note: i use df -h to figure out the mount point (/dev/sda1) for /.

After the reboot, i kept trying to reach it:

$ ssh johnsi10@40.71.228.61
ssh: connect to host 40.71.228.61 port 22: Operation timed out
$ ssh johnsi10@40.71.228.61
ssh: connect to host 40.71.228.61 port 22: Operation timed out
$ ssh johnsi10@40.71.228.61
ssh: connect to host 40.71.228.61 port 22: Operation timed out
$ ssh johnsi10@40.71.228.61
ssh: connect to host 40.71.228.61 port 22: Operation timed out
$ ssh johnsi10@40.71.228.61
ssh: connect to host 40.71.228.61 port 22: Operation timed out
$ ssh johnsi10@40.71.228.61

When it didnt come back up, i did a force reset in the Azure portal and that solved it.

However

k3os-2272 [~]$ ulimit -Hn
4096
k3os-2272 [~]$ ulimit -Sn
1024

Trying to solve ulimits myself:

TL/DR; i didn’t. But I thought you might want to know what i did try.

Let’s try setting inline and rebooting…

k3os-2272 [~]$ sudo vi /etc/sysctl.d/00-alpine.conf 
k3os-2272 [~]$ sudo reboot
k3os-2272 [~]$ Connection to 40.71.228.61 closed by remote host.
Connection to 40.71.228.61 closed.

I also tried all the steps documented in these blogs as well:

  1. https://serverfault.com/questions/165316/how-to-configure-linux-file-descriptor-limit-with-fs-file-max-and-ulimit/166037
  2. https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a

Trying to Use Multipass for k3os (fail)

Here are the steps I used

$ multipass launch -c 2 -m 2G -n foo5
Launched: foo5                                                                  

#########################################################################
A new Multipass version 1.0.1 is available!
Find out more: https://github.com/canonical/multipass/releases/tag/v1.0.1
#########################################################################

$ multipass list
Name State IPv4 Image
foo1 Stopped -- Ubuntu 18.04 LTS
foo2 Stopped -- Ubuntu 18.04 LTS
foo5 Running 192.168.64.23 Ubuntu 18.04 LTS
foo3 Stopped -- Ubuntu 18.04 LTS
foo4 Stopped -- Ubuntu 18.04 LTS

$ multipass shell foo5
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-72-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Sat Dec 21 18:51:45 CST 2019

  System load: 0.04 Processes: 126
  Usage of /: 20.9% of 4.67GB Users logged in: 0
  Memory usage: 6% IP address for enp0s2: 192.168.64.23
  Swap usage: 0%


0 packages can be updated.
0 updates are security updates.


To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

multipass@foo5:~$ 
multipass@foo5:~$ cat /tmp/config.yaml 
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDJcNg9N4Ndtr3SZdfI1BWoVYY+/R1pXkwh+/5Gvc0nKMZQhIWnXCEw4DQe8Q6l1MUyi+pTHJKlA1Fiivrky8mofPAO5U/yIHWWhwXdypdzVtvBxRq/aBdCLmI/coXFCTGz0aYNqumP08SIk66PGggnIsTqYYgKaFLnAJw2K9faWyiTrmEO7ggu1fesL5VR6MGAqpgfyHEMv2x21oZD/WmtecsURxbJUHyH9yvXfXSVbpXI9kHWpLEc228xi72ZyIQTTHSuNGHzE7KTyOn93aFhbr4VoMkCWiq2Yxh6Xzzyysmu49sHs5DOX975jhmYSmwJqXMDRg5FnGw8GeZc8kV
- github:idjohnson

k3os:
  sysctl:
    kernel.printk: 4 4 1 7
    fs.file-max: 65000
  data_sources:
  - aws
  - cdrom
  modules:
  - kvm
  - nvme
  dns_nameservers:
  - 8.8.8.8
  - 1.1.1.1
  - 127.0.0.53
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  password: r4nch3r1

multipass@foo5:~$ wget https://raw.githubusercontent.com/rancher/k3os/master/install.sh && chmod 755 ./install.sh 
--2019-12-21 18:55:33-- https://raw.githubusercontent.com/rancher/k3os/master/install.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.184.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.184.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8034 (7.8K) [text/plain]
Saving to: ‘install.sh’

install.sh 100%[========================================================================================>] 7.85K --.-KB/s in 0.001s  

2019-12-21 18:55:34 (11.1 MB/s) - ‘install.sh’ saved [8034/8034]

multipass@foo5:~$ sudo ./install.sh --takeover --debug --tty ttyS0 --config /tmp/config.yaml --no-format /dev/sda1 https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso
+ K3OS_INSTALL_DEBUG=true
+ shift 1
+ '[' 7 -gt 0 ']'
+ case $1 in
+ shift 1
+ K3OS_INSTALL_TTY=ttyS0
+ shift 1
+ '[' 5 -gt 0 ']'
+ case $1 in
+ shift 1
+ K3OS_INSTALL_CONFIG_URL=/tmp/config.yaml
+ shift 1
+ '[' 3 -gt 0 ']'
+ case $1 in
+ K3OS_INSTALL_NO_FORMAT=true
+ shift 1
+ '[' 2 -gt 0 ']'
+ case $1 in
+ '[' 2 -gt 2 ']'
+ INTERACTIVE=true
+ K3OS_INSTALL_DEVICE=/dev/sda1
+ K3OS_INSTALL_ISO_URL=https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso
+ break
+ '[' -e /etc/environment ']'
+ source /etc/environment
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ '[' -e /etc/os-release ']'
+ source /etc/os-release
++ NAME=Ubuntu
++ VERSION='18.04.3 LTS (Bionic Beaver)'
++ ID=ubuntu
++ ID_LIKE=debian
++ PRETTY_NAME='Ubuntu 18.04.3 LTS'
++ VERSION_ID=18.04
++ HOME_URL=https://www.ubuntu.com/
++ SUPPORT_URL=https://help.ubuntu.com/
++ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
++ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
++ VERSION_CODENAME=bionic
++ UBUNTU_CODENAME=bionic
+ '[' -z https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso ']'
+ '[' -z /dev/sda1 ']'
+ validate_progs
+ for i in $PROGS
++ which dd
+ '[' '!' -x /bin/dd ']'
+ for i in $PROGS
++ which curl
+ '[' '!' -x /usr/bin/curl ']'
+ for i in $PROGS
++ which mkfs.ext4
+ '[' '!' -x /sbin/mkfs.ext4 ']'
+ for i in $PROGS
++ which mkfs.vfat
+ '[' '!' -x /sbin/mkfs.vfat ']'
+ for i in $PROGS
++ which fatlabel
+ '[' '!' -x /sbin/fatlabel ']'
+ for i in $PROGS
++ which parted
+ '[' '!' -x /sbin/parted ']'
+ for i in $PROGS
++ which partprobe
+ '[' '!' -x /sbin/partprobe ']'
+ for i in $PROGS
++ which grub-install
+ '[' '!' -x /usr/sbin/grub-install ']'
+ '[' -n '' ']'
+ validate_device
+ DEVICE=/dev/sda1
+ '[' '!' -b /dev/sda1 ']'
+ trap cleanup exit
+ get_iso
++ blkid -L K3OS
++ true
+ ISO_DEVICE=
+ '[' -z '' ']'
++ lsblk -o NAME,TYPE -n
++ awk '{print $1}'
++ grep -w disk
+ for i in $(lsblk -o NAME,TYPE -n | grep -w disk | awk '{print $1}')
+ mkdir -p /run/k3os/iso
+ mount -t iso9660 -o ro /dev/sda /run/k3os/iso
mount: /run/k3os/iso: /dev/sda already mounted or mount point busy.
+ '[' -z '' ']'
+ '[' -n https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso ']'
++ mktemp k3os.XXXXXXXX.iso
+ TEMP_FILE=k3os.VQCYb4MI.iso
+ get_url https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso k3os.VQCYb4MI.iso
+ FROM=https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso
+ TO=k3os.VQCYb4MI.iso
+ case $FROM in
+ curl -o k3os.VQCYb4MI.iso -fL https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-amd64.iso
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 605 0 605 0 0 1957 0 --:--:-- --:--:-- --:--:-- 1957
100 433M 100 433M 0 0 15.4M 0 0:00:27 0:00:27 --:--:-- 15.7M
++ losetup --show -f k3os.VQCYb4MI.iso
+ ISO_DEVICE=/dev/loop0
+ rm -f k3os.VQCYb4MI.iso
+ '[' -z /dev/loop0 ']'
+ setup_style
+ '[' '' = true ']'
+ '[' -e /sys/firmware/efi ']'
+ PARTTABLE=msdos
+ BOOTFLAG=boot
+ do_format
+ '[' true = true ']'
++ blkid -L K3OS_STATE
++ true
+ STATE=
+ '[' -z '' ']'
+ '[' -n /dev/sda1 ']'
+ tune2fs -L K3OS_STATE /dev/sda1
tune2fs 1.44.1 (24-Mar-2018)
++ blkid -L K3OS_STATE
+ STATE=/dev/sda1
+ return 0
+ do_mount
+ TARGET=/run/k3os/target
+ mkdir -p /run/k3os/target
+ mount /dev/sda1 /run/k3os/target
+ mkdir -p /run/k3os/target/boot
+ '[' -n '' ']'
+ mkdir -p /run/k3os/iso
+ mount -t iso9660 -o ro /dev/loop0 /run/k3os/iso
+ do_copy
+ tar cf - -C /run/k3os/iso k3os
+ tar xvf - -C /run/k3os/target
k3os/
k3os/system/
k3os/system/config.yaml
k3os/system/k3os/
k3os/system/k3os/current
k3os/system/k3os/v0.8.0/
k3os/system/k3os/v0.8.0/k3os
k3os/system/k3os/v0.8.0/k3os-install.sh
k3os/system/k3s/
k3os/system/k3s/current
k3os/system/k3s/v1.0.0/
k3os/system/k3s/v1.0.0/k3s
k3os/system/k3s/v1.0.0/k3s-install.sh
k3os/system/kernel/
k3os/system/kernel/4.15.0-72-generic/
k3os/system/kernel/4.15.0-72-generic/initrd
k3os/system/kernel/4.15.0-72-generic/kernel.squashfs
k3os/system/kernel/current
+ '[' -n '' ']'
+ '[' -n /tmp/config.yaml ']'
+ get_url /tmp/config.yaml /run/k3os/target/k3os/system/config.yaml
+ FROM=/tmp/config.yaml
+ TO=/run/k3os/target/k3os/system/config.yaml
+ case $FROM in
+ cp -f /tmp/config.yaml /run/k3os/target/k3os/system/config.yaml
+ chmod 600 /run/k3os/target/k3os/system/config.yaml
+ '[' true = true ']'
+ touch /run/k3os/target/k3os/system/takeover
+ install_grub
+ '[' true ']'
+ GRUB_DEBUG=k3os.debug
+ mkdir -p /run/k3os/target/boot/grub
+ cat
+ '[' -z ttyS0 ']'
+ TTY=ttyS0
+ '[' -e /dev/ttyS0 ']'
+ '[' ttyS0 '!=' tty1 ']'
+ '[' -n ttyS0 ']'
+ sed -i 's!console=tty1!console=tty1 console=ttyS0!g' /run/k3os/target/boot/grub/grub.cfg
+ '[' true = true ']'
+ return 0
+ create_opt
+ mkdir -p /run/k3os/target/k3os/data/opt
+ '[' -n true ']'
+ exit 0
+ cleanup
+ EXIT=0
+ cleanup2
+ return 0
multipass@foo5:~$ 
multipass@foo5:~$ exit
logout
JOHNSI10-M1:altair-hwx johnsi10$ multipass stop foo5
JOHNSI10-M1:altair-hwx johnsi10$ multipass start foo5                           
Starting foo5 /

I can say i tried a few ways of rebooting , but it did not seem the inline install worked at all.  I did a snap install of kubectl and localhost was refused:

multipass@foo5:~$ sudo snap install kubectl --classic
2019-12-21T19:02:54-06:00 INFO Waiting for restart...
kubectl 1.17.0 from Canonical✓ installed
multipass@foo5:~$ kubectl get pods
The connection to the server localhost:8080 was refused - did you specify the right host or port?

I decided i would try one more host with takeover - this time a straight debian 8 instance on Linode.  It too was a fail:

note: i did need to install “apt install dosfstools parted” to get the install.sh to run

Summary

In the end, the issue about hard and soft file limits blocked us.  We simply couldn’t use ELK charts and frankly, any hefty java based container would be truly hampered by a soft 1024 and hard 4096 file limit.  

If your stack does not have large file counter concerns, one could use k3os.  If they resolve this issue, i’ll certainly circle back and update this blog post.

In the project we were considering k3os for, we instead used RKE via Rancher/RancherOS which handled our ELK needs just fine.  

ulimits in RancherOS (for RKE)

That said, k3os, with its meager requirements is a very promising option for creating a stack with lighter or older hardware.

k3s linode

Have something to add? Feedback? You can use the feedback form

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes