We've explored Rancher's K3s before as well as K3OS.  It's a pretty handy lightweight kubernetes environment.  However, past attempts to make my Pi3b's proper containerized build agent pools ultimately fell down due it's limited RAM (1Gb).  The Pi4 has a 4Gb option which puts it in the running.. So how can we take what is (presently) about a $65 hobbiest board and do something cool with it?  Let's dig in - you might be suprised how capable it really is.

Setting up an initial Azure DevOps agent

We assume you’ve set up the Pi4 to have a functional instance of Raspbian running on it.

Next you'll need to create the necessary files for the Azure DevOps docker agent (start.sh and Dockerfile)

(from https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops) but with a key difference being the image platform (linux-arm instead of linux-x64)

pi@raspberrypi:~/dockeragent $ cat start.sh 
#!/bin/bash
set -e

if [ -z "$AZP_URL" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -z "$AZP_TOKEN_FILE" ]; then
  if [ -z "$AZP_TOKEN" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE=/azp/.token
  echo -n $AZP_TOKEN > "$AZP_TOKEN_FILE"
fi

unset AZP_TOKEN

if [ -n "$AZP_WORK" ]; then
  mkdir -p "$AZP_WORK"
fi

rm -rf /azp/agent
mkdir /azp/agent
cd /azp/agent

export AGENT_ALLOW_RUNASROOT="1"

cleanup() {
  if [ -e config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    ./config.sh remove --unattended \
      --auth PAT \
      --token $(cat "$AZP_TOKEN_FILE")
  fi
}

print_header() {
  lightcyan='\033[1;36m'
  nocolor='\033[0m'
  echo -e "${lightcyan}$1${nocolor}"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE=AZP_TOKEN,AZP_TOKEN_FILE

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_RESPONSE=$(curl -LsS \
  -u user:$(cat "$AZP_TOKEN_FILE") \
  -H 'Accept:application/json;api-version=3.0-preview' \
  "$AZP_URL/_apis/distributedtask/packages/agent?platform=linux-arm")

if echo "$AZP_AGENT_RESPONSE" | jq . >/dev/null 2>&1; then
  AZP_AGENTPACKAGE_URL=$(echo "$AZP_AGENT_RESPONSE" \
    | jq -r '.value | map([.version.major,.version.minor,.version.patch,.downloadUrl]) | sort | .[length-1] | .[3]')
fi

if [ -z "$AZP_AGENTPACKAGE_URL" -o "$AZP_AGENTPACKAGE_URL" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent - check that account '$AZP_URL' is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and installing Azure Pipelines agent..."

curl -LsS $AZP_AGENTPACKAGE_URL | tar -xz & wait $!

source ./env.sh

trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "$AZP_URL" \
  --auth PAT \
  --token $(cat "$AZP_TOKEN_FILE") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

# remove the administrative token before accepting work
rm $AZP_TOKEN_FILE

print_header "4. Running Azure Pipelines agent..."

# `exec` the node runtime so it's aware of TERM and INT signals
# AgentService.js understands how to handle agent self-update and restart
exec ./externals/node/bin/node ./bin/AgentService.js interactive

pi@raspberrypi:~/dockeragent $ cat Dockerfile 
FROM ubuntu:16.04

# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        jq \
        git \
        iputils-ping \
        libcurl3 \
        libicu55 \
        libunwind8 \
        netcat

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

CMD ["./start.sh"]

Installing docker on pi


Next, let’s install docker on the pi.  We can generally follow steps from https://linuxize.com/post/how-to-install-and-use-docker-on-raspberry-pi/

pi@raspberrypi:~ $ curl -fsSL https://get.docker.com -o get-docker.sh
pi@raspberrypi:~ $ sh get-docker.sh 
# Executing docker install script, commit: f45d7c11389849ff46a6b4d94e0dd1ffebca32c1
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sudo -E sh -c curl -fsSL "https://download.docker.com/linux/raspbian/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sudo -E sh -c echo "deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable" > /etc/apt/sources.list.d/docker.list
+ sudo -E sh -c apt-get update -qq >/dev/null
+ [ -n  ]
+ sudo -E sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
+ sudo -E sh -c docker version
Client: Docker Engine - Community
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.16
 Git commit:        369ce74
 Built:             Thu Feb 13 01:37:07 2020
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.16
  Git commit:       369ce74
  Built:            Thu Feb 13 01:31:06 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:

  sudo usermod -aG docker pi

Remember that you will have to log out and back in for this to take effect!

WARNING: Adding a user to the "docker" group will grant the ability to run
         containers which can be used to obtain root privileges on the
         docker host.
         Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
         for more information.

Pro-tip ; if you get an access error, you likely just need to usermod the pi user

pi@raspberrypi:~/dockeragent $ docker build -t dockeragent:latest .
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=nl12q1vysdeyq6pbvnv16ou4m&shmsize=0&t=dockeragent%3Alatest&target=&ulimits=null&version=1: dial unix /var/run/docker.sock: connect: permission denied
pi@raspberrypi:~/dockeragent $ sudo usermod -aG docker pi
pi@raspberrypi:~/dockeragent $ exit
logout
Connection to 192.168.1.207 closed.

johnsi10$ ssh pi@192.168.1.207
pi@192.168.1.207's password: 
Linux raspberrypi 4.19.93-v7l+ #1290 SMP Fri Jan 10 16:45:11 GMT 2020 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Feb 14 20:56:53 2020 from 192.168.1.113
pi@raspberrypi:~ $ cd dockeragent/
pi@raspberrypi:~/dockeragent $ docker build -t dockeragent:latest .

We should now be able to build the image

pi@raspberrypi:~/dockeragent $ docker build -t dockeragent:latest .
Sending build context to Docker daemon  5.632kB
Step 1/8 : FROM ubuntu:16.04
16.04: Pulling from library/ubuntu
e60df59fb597: Pull complete 
99ba0251fafa: Pull complete 
cff14f7ed750: Pull complete 
638c86faa077: Pull complete 
Digest: sha256:3f3ee50cb89bc12028bab7d1e187ae57f12b957135b91648702e835c37c6c971
Status: Downloaded newer image for ubuntu:16.04
 ---> 771a2e2e1f23
Step 2/8 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Running in b43e7f22dcc3
Removing intermediate container b43e7f22dcc3
 ---> 8196f523375c
Step 3/8 : RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
 ---> Running in f01e9b3f9c90
Removing intermediate container f01e9b3f9c90
 ---> fe563fb6aec0
Step 4/8 : RUN apt-get update && apt-get install -y --no-install-recommends         ca-certificates         curl         jq         git         iputils-ping         libcurl3         libicu55         libunwind8         netcat
 ---> Running in e1ae490f1220
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial InRelease [247 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [109 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease [107 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease [109 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf Packages [1486 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports xenial/restricted armhf Packages [8491 B]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial/universe armhf Packages [9531 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial/multiverse armhf Packages [149 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf Packages [1045 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports xenial-updates/restricted armhf Packages [8488 B]
Get:11 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe armhf Packages [905 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports xenial-updates/multiverse armhf Packages [13.6 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports xenial-backports/main armhf Packages [7936 B]
Get:14 http://ports.ubuntu.com/ubuntu-ports xenial-backports/universe armhf Packages [8424 B]
Get:15 http://ports.ubuntu.com/ubuntu-ports xenial-security/main armhf Packages [714 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports xenial-security/restricted armhf Packages [8480 B]
Get:17 http://ports.ubuntu.com/ubuntu-ports xenial-security/universe armhf Packages [533 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports xenial-security/multiverse armhf Packages [3157 B]
Fetched 15.0 MB in 4s (3193 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  git-man libasn1-8-heimdal libcurl3-gnutls liberror-perl libexpat1 libffi6
  libgdbm3 libgmp10 libgnutls-openssl27 libgnutls30 libgssapi-krb5-2
  libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
  libheimntlm0-heimdal libhogweed4 libhx509-5-heimdal libidn11 libk5crypto3
  libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2
  libnettle6 libonig2 libp11-kit0 libperl5.22 libroken18-heimdal librtmp1
  libsasl2-2 libsasl2-modules-db libsqlite3-0 libssl1.0.0 libtasn1-6
  libwind0-heimdal netcat-traditional openssl perl perl-modules-5.22
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
  git-gui gitk gitweb git-arch git-cvs git-mediawiki git-svn gnutls-bin
  krb5-doc krb5-user perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl make
Recommended packages:
  patch less rsync ssh-client krb5-locales libsasl2-modules netbase rename
The following NEW packages will be installed:
  ca-certificates curl git git-man iputils-ping jq libasn1-8-heimdal libcurl3
  libcurl3-gnutls liberror-perl libexpat1 libffi6 libgdbm3 libgmp10
  libgnutls-openssl27 libgnutls30 libgssapi-krb5-2 libgssapi3-heimdal
  libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhogweed4
  libhx509-5-heimdal libicu55 libidn11 libk5crypto3 libkeyutils1
  libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libnettle6
  libonig2 libp11-kit0 libperl5.22 libroken18-heimdal librtmp1 libsasl2-2
  libsasl2-modules-db libsqlite3-0 libssl1.0.0 libtasn1-6 libunwind8
  libwind0-heimdal netcat netcat-traditional openssl perl perl-modules-5.22
0 upgraded, 49 newly installed, 0 to remove and 17 not upgraded.
Need to get 21.0 MB of archives.
After this operation, 91.9 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libgdbm3 armhf 1.8.3-13.1 [15.3 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf perl-modules-5.22 all 5.22.1-9ubuntu0.6 [2629 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libperl5.22 armhf 5.22.1-9ubuntu0.6 [2728 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf perl armhf 5.22.1-9ubuntu0.6 [237 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libgmp10 armhf 2:6.1.0+dfsg-2 [184 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libnettle6 armhf 3.2-1ubuntu0.16.04.1 [111 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libhogweed4 armhf 3.2-1ubuntu0.16.04.1 [126 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libidn11 armhf 1.32-3ubuntu1.2 [43.1 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libffi6 armhf 3.2.1-4 [16.2 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libp11-kit0 armhf 0.23.2-5~ubuntu16.04.1 [91.0 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libtasn1-6 armhf 4.7-3ubuntu0.16.04.3 [37.9 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgnutls30 armhf 3.4.10-4ubuntu1.7 [485 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgnutls-openssl27 armhf 3.4.10-4ubuntu1.7 [19.0 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf iputils-ping armhf 3:20121221-5ubuntu2 [50.1 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libexpat1 armhf 2.1.0-7ubuntu0.16.04.5 [53.2 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libsqlite3-0 armhf 3.11.0-1ubuntu1.3 [338 kB]
Get:17 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libssl1.0.0 armhf 1.0.2g-1ubuntu4.15 [711 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf openssl armhf 1.0.2g-1ubuntu4.15 [485 kB]
Get:19 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf ca-certificates all 20170717~16.04.2 [167 kB]
Get:20 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libroken18-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [34.1 kB]
Get:21 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libasn1-8-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [138 kB]
Get:22 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libkrb5support0 armhf 1.13.2+dfsg-5ubuntu2.1 [27.5 kB]
Get:23 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libk5crypto3 armhf 1.13.2+dfsg-5ubuntu2.1 [77.6 kB]
Get:24 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libkeyutils1 armhf 1.5.9-8ubuntu1 [9192 B]
Get:25 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libkrb5-3 armhf 1.13.2+dfsg-5ubuntu2.1 [230 kB]
Get:26 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgssapi-krb5-2 armhf 1.13.2+dfsg-5ubuntu2.1 [98.8 kB]
Get:27 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libhcrypto4-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [75.7 kB]
Get:28 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libheimbase1-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [24.0 kB]
Get:29 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libwind0-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [47.0 kB]
Get:30 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libhx509-5-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [88.4 kB]
Get:31 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libkrb5-26-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [164 kB]
Get:32 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libheimntlm0-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [13.4 kB]
Get:33 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgssapi3-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [78.5 kB]
Get:34 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libsasl2-modules-db armhf 2.1.26.dfsg1-14ubuntu0.2 [13.0 kB]
Get:35 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libsasl2-2 armhf 2.1.26.dfsg1-14ubuntu0.2 [42.0 kB]
Get:36 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libldap-2.4-2 armhf 2.4.42+dfsg-2ubuntu3.7 [137 kB]
Get:37 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf librtmp1 armhf 2.4+20151223.gitfa8646d-1ubuntu0.1 [49.4 kB]
Get:38 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libcurl3-gnutls armhf 7.47.0-1ubuntu2.14 [159 kB]
Get:39 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libicu55 armhf 55.1-7ubuntu0.4 [7404 kB]
Get:40 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf curl armhf 7.47.0-1ubuntu2.14 [135 kB]
Get:41 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf liberror-perl all 0.17-1.2 [19.6 kB]
Get:42 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf git-man all 1:2.7.4-0ubuntu1.7 [736 kB]
Get:43 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf git armhf 1:2.7.4-0ubuntu1.7 [2182 kB]
Get:44 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe armhf libonig2 armhf 5.9.6-1ubuntu0.1 [72.5 kB]
Get:45 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe armhf jq armhf 1.5+dfsg-1ubuntu0.1 [144 kB]
Get:46 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libcurl3 armhf 7.47.0-1ubuntu2.14 [162 kB]
Get:47 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libunwind8 armhf 1.1-4.1 [43.6 kB]
Get:48 http://ports.ubuntu.com/ubuntu-ports xenial/universe armhf netcat-traditional armhf 1.10-41 [59.2 kB]
Get:49 http://ports.ubuntu.com/ubuntu-ports xenial/universe armhf netcat all 1.10-41 [3438 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 21.0 MB in 2s (8109 kB/s)
Selecting previously unselected package libgdbm3:armhf.
(Reading database ... 4771 files and directories currently installed.)
Preparing to unpack .../libgdbm3_1.8.3-13.1_armhf.deb ...
Unpacking libgdbm3:armhf (1.8.3-13.1) ...
Selecting previously unselected package perl-modules-5.22.
Preparing to unpack .../perl-modules-5.22_5.22.1-9ubuntu0.6_all.deb ...
Unpacking perl-modules-5.22 (5.22.1-9ubuntu0.6) ...
Selecting previously unselected package libperl5.22:armhf.
Preparing to unpack .../libperl5.22_5.22.1-9ubuntu0.6_armhf.deb ...
Unpacking libperl5.22:armhf (5.22.1-9ubuntu0.6) ...
Selecting previously unselected package perl.
Preparing to unpack .../perl_5.22.1-9ubuntu0.6_armhf.deb ...
Unpacking perl (5.22.1-9ubuntu0.6) ...
Selecting previously unselected package libgmp10:armhf.
Preparing to unpack .../libgmp10_2%3a6.1.0+dfsg-2_armhf.deb ...
Unpacking libgmp10:armhf (2:6.1.0+dfsg-2) ...
Selecting previously unselected package libnettle6:armhf.
Preparing to unpack .../libnettle6_3.2-1ubuntu0.16.04.1_armhf.deb ...
Unpacking libnettle6:armhf (3.2-1ubuntu0.16.04.1) ...
Selecting previously unselected package libhogweed4:armhf.
Preparing to unpack .../libhogweed4_3.2-1ubuntu0.16.04.1_armhf.deb ...
Unpacking libhogweed4:armhf (3.2-1ubuntu0.16.04.1) ...
Selecting previously unselected package libidn11:armhf.
Preparing to unpack .../libidn11_1.32-3ubuntu1.2_armhf.deb ...
Unpacking libidn11:armhf (1.32-3ubuntu1.2) ...
Selecting previously unselected package libffi6:armhf.
Preparing to unpack .../libffi6_3.2.1-4_armhf.deb ...
Unpacking libffi6:armhf (3.2.1-4) ...
Selecting previously unselected package libp11-kit0:armhf.
Preparing to unpack .../libp11-kit0_0.23.2-5~ubuntu16.04.1_armhf.deb ...
Unpacking libp11-kit0:armhf (0.23.2-5~ubuntu16.04.1) ...
Selecting previously unselected package libtasn1-6:armhf.
Preparing to unpack .../libtasn1-6_4.7-3ubuntu0.16.04.3_armhf.deb ...
Unpacking libtasn1-6:armhf (4.7-3ubuntu0.16.04.3) ...
Selecting previously unselected package libgnutls30:armhf.
Preparing to unpack .../libgnutls30_3.4.10-4ubuntu1.7_armhf.deb ...
Unpacking libgnutls30:armhf (3.4.10-4ubuntu1.7) ...
Selecting previously unselected package libgnutls-openssl27:armhf.
Preparing to unpack .../libgnutls-openssl27_3.4.10-4ubuntu1.7_armhf.deb ...
Unpacking libgnutls-openssl27:armhf (3.4.10-4ubuntu1.7) ...
Selecting previously unselected package iputils-ping.
Preparing to unpack .../iputils-ping_3%3a20121221-5ubuntu2_armhf.deb ...
Unpacking iputils-ping (3:20121221-5ubuntu2) ...
Selecting previously unselected package libexpat1:armhf.
Preparing to unpack .../libexpat1_2.1.0-7ubuntu0.16.04.5_armhf.deb ...
Unpacking libexpat1:armhf (2.1.0-7ubuntu0.16.04.5) ...
Selecting previously unselected package libsqlite3-0:armhf.
Preparing to unpack .../libsqlite3-0_3.11.0-1ubuntu1.3_armhf.deb ...
Unpacking libsqlite3-0:armhf (3.11.0-1ubuntu1.3) ...
Selecting previously unselected package libssl1.0.0:armhf.
Preparing to unpack .../libssl1.0.0_1.0.2g-1ubuntu4.15_armhf.deb ...
Unpacking libssl1.0.0:armhf (1.0.2g-1ubuntu4.15) ...
Selecting previously unselected package openssl.
Preparing to unpack .../openssl_1.0.2g-1ubuntu4.15_armhf.deb ...
Unpacking openssl (1.0.2g-1ubuntu4.15) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../ca-certificates_20170717~16.04.2_all.deb ...
Unpacking ca-certificates (20170717~16.04.2) ...
Selecting previously unselected package libroken18-heimdal:armhf.
Preparing to unpack .../libroken18-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libroken18-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libasn1-8-heimdal:armhf.
Preparing to unpack .../libasn1-8-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libasn1-8-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libkrb5support0:armhf.
Preparing to unpack .../libkrb5support0_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libkrb5support0:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libk5crypto3:armhf.
Preparing to unpack .../libk5crypto3_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libk5crypto3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libkeyutils1:armhf.
Preparing to unpack .../libkeyutils1_1.5.9-8ubuntu1_armhf.deb ...
Unpacking libkeyutils1:armhf (1.5.9-8ubuntu1) ...
Selecting previously unselected package libkrb5-3:armhf.
Preparing to unpack .../libkrb5-3_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libkrb5-3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libgssapi-krb5-2:armhf.
Preparing to unpack .../libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libgssapi-krb5-2:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libhcrypto4-heimdal:armhf.
Preparing to unpack .../libhcrypto4-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libhcrypto4-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libheimbase1-heimdal:armhf.
Preparing to unpack .../libheimbase1-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libheimbase1-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libwind0-heimdal:armhf.
Preparing to unpack .../libwind0-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libwind0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libhx509-5-heimdal:armhf.
Preparing to unpack .../libhx509-5-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libhx509-5-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libkrb5-26-heimdal:armhf.
Preparing to unpack .../libkrb5-26-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libkrb5-26-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libheimntlm0-heimdal:armhf.
Preparing to unpack .../libheimntlm0-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libheimntlm0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libgssapi3-heimdal:armhf.
Preparing to unpack .../libgssapi3-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libgssapi3-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libsasl2-modules-db:armhf.
Preparing to unpack .../libsasl2-modules-db_2.1.26.dfsg1-14ubuntu0.2_armhf.deb ...
Unpacking libsasl2-modules-db:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Selecting previously unselected package libsasl2-2:armhf.
Preparing to unpack .../libsasl2-2_2.1.26.dfsg1-14ubuntu0.2_armhf.deb ...
Unpacking libsasl2-2:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Selecting previously unselected package libldap-2.4-2:armhf.
Preparing to unpack .../libldap-2.4-2_2.4.42+dfsg-2ubuntu3.7_armhf.deb ...
Unpacking libldap-2.4-2:armhf (2.4.42+dfsg-2ubuntu3.7) ...
Selecting previously unselected package librtmp1:armhf.
Preparing to unpack .../librtmp1_2.4+20151223.gitfa8646d-1ubuntu0.1_armhf.deb ...
Unpacking librtmp1:armhf (2.4+20151223.gitfa8646d-1ubuntu0.1) ...
Selecting previously unselected package libcurl3-gnutls:armhf.
Preparing to unpack .../libcurl3-gnutls_7.47.0-1ubuntu2.14_armhf.deb ...
Unpacking libcurl3-gnutls:armhf (7.47.0-1ubuntu2.14) ...
Selecting previously unselected package libicu55:armhf.
Preparing to unpack .../libicu55_55.1-7ubuntu0.4_armhf.deb ...
Unpacking libicu55:armhf (55.1-7ubuntu0.4) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.47.0-1ubuntu2.14_armhf.deb ...
Unpacking curl (7.47.0-1ubuntu2.14) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../liberror-perl_0.17-1.2_all.deb ...
Unpacking liberror-perl (0.17-1.2) ...
Selecting previously unselected package git-man.
Preparing to unpack .../git-man_1%3a2.7.4-0ubuntu1.7_all.deb ...
Unpacking git-man (1:2.7.4-0ubuntu1.7) ...
Selecting previously unselected package git.
Preparing to unpack .../git_1%3a2.7.4-0ubuntu1.7_armhf.deb ...
Unpacking git (1:2.7.4-0ubuntu1.7) ...
Selecting previously unselected package libonig2:armhf.
Preparing to unpack .../libonig2_5.9.6-1ubuntu0.1_armhf.deb ...
Unpacking libonig2:armhf (5.9.6-1ubuntu0.1) ...
Selecting previously unselected package jq.
Preparing to unpack .../jq_1.5+dfsg-1ubuntu0.1_armhf.deb ...
Unpacking jq (1.5+dfsg-1ubuntu0.1) ...
Selecting previously unselected package libcurl3:armhf.
Preparing to unpack .../libcurl3_7.47.0-1ubuntu2.14_armhf.deb ...
Unpacking libcurl3:armhf (7.47.0-1ubuntu2.14) ...
Selecting previously unselected package libunwind8.
Preparing to unpack .../libunwind8_1.1-4.1_armhf.deb ...
Unpacking libunwind8 (1.1-4.1) ...
Selecting previously unselected package netcat-traditional.
Preparing to unpack .../netcat-traditional_1.10-41_armhf.deb ...
Unpacking netcat-traditional (1.10-41) ...
Selecting previously unselected package netcat.
Preparing to unpack .../netcat_1.10-41_all.deb ...
Unpacking netcat (1.10-41) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libgdbm3:armhf (1.8.3-13.1) ...
Setting up perl-modules-5.22 (5.22.1-9ubuntu0.6) ...
Setting up libperl5.22:armhf (5.22.1-9ubuntu0.6) ...
Setting up perl (5.22.1-9ubuntu0.6) ...
update-alternatives: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode
Setting up libgmp10:armhf (2:6.1.0+dfsg-2) ...
Setting up libnettle6:armhf (3.2-1ubuntu0.16.04.1) ...
Setting up libhogweed4:armhf (3.2-1ubuntu0.16.04.1) ...
Setting up libidn11:armhf (1.32-3ubuntu1.2) ...
Setting up libffi6:armhf (3.2.1-4) ...
Setting up libp11-kit0:armhf (0.23.2-5~ubuntu16.04.1) ...
Setting up libtasn1-6:armhf (4.7-3ubuntu0.16.04.3) ...
Setting up libgnutls30:armhf (3.4.10-4ubuntu1.7) ...
Setting up libgnutls-openssl27:armhf (3.4.10-4ubuntu1.7) ...
Setting up iputils-ping (3:20121221-5ubuntu2) ...
Setcap is not installed, falling back to setuid
Setting up libexpat1:armhf (2.1.0-7ubuntu0.16.04.5) ...
Setting up libsqlite3-0:armhf (3.11.0-1ubuntu1.3) ...
Setting up libssl1.0.0:armhf (1.0.2g-1ubuntu4.15) ...
Setting up openssl (1.0.2g-1ubuntu4.15) ...
Setting up ca-certificates (20170717~16.04.2) ...
Setting up libroken18-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libasn1-8-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libkrb5support0:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libk5crypto3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libkeyutils1:armhf (1.5.9-8ubuntu1) ...
Setting up libkrb5-3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libgssapi-krb5-2:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libhcrypto4-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libheimbase1-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libwind0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libhx509-5-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libkrb5-26-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libheimntlm0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libgssapi3-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libsasl2-modules-db:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Setting up libsasl2-2:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Setting up libldap-2.4-2:armhf (2.4.42+dfsg-2ubuntu3.7) ...
Setting up librtmp1:armhf (2.4+20151223.gitfa8646d-1ubuntu0.1) ...
Setting up libcurl3-gnutls:armhf (7.47.0-1ubuntu2.14) ...
Setting up libicu55:armhf (55.1-7ubuntu0.4) ...
Setting up curl (7.47.0-1ubuntu2.14) ...
Setting up liberror-perl (0.17-1.2) ...
Setting up git-man (1:2.7.4-0ubuntu1.7) ...
Setting up git (1:2.7.4-0ubuntu1.7) ...
Setting up libonig2:armhf (5.9.6-1ubuntu0.1) ...
Setting up jq (1.5+dfsg-1ubuntu0.1) ...
Setting up libcurl3:armhf (7.47.0-1ubuntu2.14) ...
Setting up libunwind8 (1.1-4.1) ...
Setting up netcat-traditional (1.10-41) ...
update-alternatives: using /bin/nc.traditional to provide /bin/nc (nc) in auto mode
Setting up netcat (1.10-41) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for ca-certificates (20170717~16.04.2) ...
Updating certificates in /etc/ssl/certs...
148 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Removing intermediate container e1ae490f1220
 ---> 436f1a0a555b
Step 5/8 : WORKDIR /azp
 ---> Running in a6cd328afd5f
Removing intermediate container a6cd328afd5f
 ---> 8cd9408c3d2a
Step 6/8 : COPY ./start.sh .
 ---> a0106165900f
Step 7/8 : RUN chmod +x start.sh
 ---> Running in 7de0217ce723
Removing intermediate container 7de0217ce723
 ---> c741cd4e9480
Step 8/8 : CMD ["./start.sh"]
 ---> Running in 4b335dc618da
Removing intermediate container 4b335dc618da
 ---> eac4c6b0709e
Successfully built eac4c6b0709e
Successfully tagged dockeragent:latest

We can then push to dockerhub (or the registry of our choice):

pi@raspberrypi:~/dockeragent $ docker tag dockeragent:latest idjohnson/pivstsagent:latest
pi@raspberrypi:~/dockeragent $ docker push idjohnson/pivstsagent:latest
The push refers to repository [docker.io/idjohnson/pivstsagent]
299c2b3ce4f0: Pushed 
bb599863d08a: Pushed 
889a29fac21d: Pushed 
c58c5431d648: Pushed 
3fc1655ba429: Pushed 
8fdf48a435b0: Mounted from library/ubuntu 
89961002fb88: Mounted from library/ubuntu 
160f8b9f32ca: Mounted from library/ubuntu 
c03e0758467b: Pushed 
latest: digest: sha256:3bd8022c881ef97ad2a527a3d644709bd8185616f43e3116336ff1637b9cde6e size: 2192

In my case, you’ll see the arm based Azure DevOps agent here:

https://hub.docker.com/repository/registry-1.docker.io/idjohnson/pivstsagent/tags?page=1

(and you can use it if you want: docker pull idjohnson/pivstsagent:latest)

launching an agent in our k3s cluster

Create the yaml that will hold our deployment, configmap and secret yaml.  You can use base64 to encode your Azure DevOps PAT (token) for the secret file (e.g. echo mytoken | base64).

vsts-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: isaac-vsts-agent-deployment
  labels:
    app: isaac-vsts-agent-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: isaac-vsts-agent-deployment
  template:
    metadata:
      labels:
        app: isaac-vsts-agent-deployment
    spec:
      containers:
      - name: dockeragent
        image: idjohnson/pivstsagent:latest
        env:
          - name: AZP_TOKEN
            valueFrom:
              secretKeyRef:
                name: azdo-secret-config
                key: AZP_TOKEN
          - name: AZP_URL
            valueFrom:
              configMapKeyRef:
                name: azdo-environment-config
                key: AZP_URL
          - name: AZP_AGENT_NAME
            valueFrom:
              configMapKeyRef:
                name: azdo-environment-config
                key: AZP_AGENT_NAME
          - name: testing.variable
            value: "hello world"
---
apiVersion: v1
kind: Secret
metadata:
  name: azdo-secret-config
data:
  AZP_TOKEN: bm9wZS1ub3QtbXktdG9rZW4tYnV0LWdvb2QtdHJ5IQo=
---
apiVersion: v1 
kind: ConfigMap 
metadata:
  name: azdo-environment-config
data:
  AZP_URL: https://dev.azure.com/princessking/
  AZP_AGENT_NAME: pi4agent

Now let’s launch it:

$ kubectl apply -f vsts-deployment.yaml 
deployment.apps/isaac-vsts-agent-deployment created
secret/azdo-secret-config created
configmap/azdo-environment-config created

If we head over to our agent pools, we can see that it launched.

We can create a quick build pipeline to verify that indeed it’s running…

Making it more useful

This is great, but having some build tools will make this image far more useful in pipelines.

Let’s update the Dockerfile to include golang and java;

$ cat Dockerfile 
FROM ubuntu:16.04

# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        jq \
        git \
        iputils-ping \
        libcurl3 \
        libicu55 \
        libunwind8 \
        netcat

# Common build tools
RUN apt-get update \ 
&& apt-get install -y openjdk-8-jdk

RUN curl -O https://dl.google.com/go/go1.12.17.linux-armv6l.tar.gz \
&& tar -C /usr/local -xzf go1.12.17.linux-armv6l.tar.gz \
&& rm go1.12.17.linux-armv6l.tar.gz

RUN echo "PATH=$PATH:/usr/local/go/bin" >>  ~/.profile

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

CMD ["./start.sh"]

In fact, if we want to get fancy, we can create a large docker container with Node, Go and Java!

$ cat Dockerfile
FROM ubuntu:16.04

# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        jq \
        git \
        iputils-ping \
        libcurl3 \
        libicu55 \
        libunwind8 \
        netcat

# Common build tools
RUN apt-get update \ 
&& apt-get install -y openjdk-8-jdk

RUN curl -O https://dl.google.com/go/go1.12.17.linux-armv6l.tar.gz \
&& tar -C /usr/local -xzf go1.12.17.linux-armv6l.tar.gz \
&& rm go1.12.17.linux-armv6l.tar.gz

RUN echo "PATH=$PATH:/usr/local/go/bin" >>  ~/.profile

# replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh

# update the repository sources list
# and install dependencies
RUN apt-get update \
    && apt-get install -y curl \
    && apt-get -y autoclean

# nvm environment variables
ENV NVM_DIR /usr/local/nvm
ENV NODE_VERSION 4.4.7

# install nvm
# https://github.com/creationix/nvm#install-script
# instructions from https://gist.github.com/remarkablemark/aacf14c29b3f01d6900d13137b21db3a
RUN curl --silent -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.2/install.sh | bash

# install node and npm
RUN source $NVM_DIR/nvm.sh \
    && nvm install $NODE_VERSION \
    && nvm alias default $NODE_VERSION \
    && nvm use default

# add node and npm to path so the commands are available
ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules
ENV PATH $NVM_DIR/versions/node/v$NODE_VERSION/bin:$PATH

# confirm installation
RUN node -v
RUN npm -v

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

CMD ["./start.sh"]

After we do a docker build, we can see some of our images as well as tag and push to docker.io

pi@raspberrypi:~/dockeragent $ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
dockeragent             latest              244a5838d922        5 seconds ago       749MB
<none>                  <none>              22c719477057        3 minutes ago       711MB
<none>                  <none>              c7b91824198a        8 hours ago         711MB
idjohnson/pivstsagent   latest              57ecdd599a28        9 hours ago         209MB
idjohnson/pivstsagent   latest2             57ecdd599a28        9 hours ago         209MB
idjohnson/pivstsagent   tagname             eac4c6b0709e        20 hours ago        209MB
ubuntu                  16.04               771a2e2e1f23        4 weeks ago         98.2MB
pi@raspberrypi:~/dockeragent $ docker tag dockeragent:latest idjohnson/pivstsagent:nodegojava
pi@raspberrypi:~/dockeragent $ docker push idjohnson/pivstsagent:nodegojava
The push refers to repository [docker.io/idjohnson/pivstsagent]
e4771494a2fc: Pushed 
45ab68c49c1c: Pushed 
091b58c7f80e: Pushed 
08dcde28dc90: Pushed 
15e5922ff065: Pushed 
3e0ae270bcbd: Pushed 
43c4d5af1be0: Pushed 
2026ebeace84: Pushed 
7af300e93a53: Pushing  181.7MB/259.8MB
13a8ad2a479f: Pushing  201.6MB/241.4MB
c58c5431d648: Layer already exists 
3fc1655ba429: Layer already exists 
8fdf48a435b0: Layer already exists 
89961002fb88: Layer already exists 
160f8b9f32ca: Layer already exists 

As you can see, it’s a hefty image.  Adding docker itself to it pushes it over the 1gb level.

pi@raspberrypi:~/dockeragent $ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
dockeragent             latest              00e6a46d1747        30 minutes ago      1.01GB
pi@raspberrypi:~/dockeragent $ docker tag dockeragent:latest idjohnson/pivstsagent:ndgjvdocker
pi@raspberrypi:~/dockeragent $ docker push idjohnson/pivstsagent:ndgjvdocker
The push refers to repository [docker.io/idjohnson/pivstsagent]
091fccd8f7ba: Pushed 
180edef0eb30: Pushed 
d5db2fd9c90b: Pushed 
4b7562f44ee8: Pushed 
08dcde28dc90: Layer already exists 
15e5922ff065: Layer already exists 
3e0ae270bcbd: Layer already exists 
43c4d5af1be0: Layer already exists 
2026ebeace84: Layer already exists 
7af300e93a53: Layer already exists 
13a8ad2a479f: Layer already exists 
c58c5431d648: Layer already exists 
3fc1655ba429: Layer already exists 
8fdf48a435b0: Layer already exists 
89961002fb88: Layer already exists 
160f8b9f32ca: Layer already exists 
c03e0758467b: Layer already exists 
ndgjvdocker: digest: sha256:5c72b0c6b5a673c6b75148fd04f4bdea02c47189193a730b44f35d5dfe22ffa5 size: 3876

One thing we can do, so that we can actually leverage multiple agents from the deployment, is to dynamically set the agent name - this allows us to have a replicaset scaled up to meet our needs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: isaac-vsts-agent-deployment
  labels:
    app: isaac-vsts-agent-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: isaac-vsts-agent-deployment
  template:
    metadata:
      labels:
        app: isaac-vsts-agent-deployment
    spec:
      containers:
      - name: dockeragent
        image: idjohnson/pivstsagent:ndgjvdocker
        env:
          - name: AZP_TOKEN
            valueFrom:
              secretKeyRef:
                name: azdo-secret-config
                key: AZP_TOKEN
          - name: AZP_URL
            valueFrom:
              configMapKeyRef:
                name: azdo-environment-config
                key: AZP_URL
          - name: AZP_AGENT_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
---
apiVersion: v1
kind: Secret
metadata:
  name: azdo-secret-config
data:
  AZP_TOKEN: bm9wZS1ub3QtcmVhbGx5LW15LXRva2VuLWJ1dC10aGFua3MtZm9yLXJlYWRpbmctdGhlLWJsb2cK=

You'll notice I dropped the configmap as the name is now pulled from 'metadata.name'

      - name: AZP_AGENT_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name

The result, when applied, is agents are now added with their pod names:

Scaling events or updating the image will cause the old agents to go offline in AzDO, which is expected:

We can also confirm the name came from metadata this time by looking at the pod details:

$ kubectl describe pod isaac-vsts-agent-deployment-645bcb968c-qgxbh
Name:           isaac-vsts-agent-deployment-645bcb968c-qgxbh
Namespace:      default
Priority:       0
Node:           raspberrypi/192.168.1.207
Start Time:     Sun, 16 Feb 2020 07:19:09 -0600
Labels:         app=isaac-vsts-agent-deployment
                pod-template-hash=645bcb968c
Annotations:    <none>
Status:         Running
IP:             10.42.0.18
Controlled By:  ReplicaSet/isaac-vsts-agent-deployment-645bcb968c
Containers:
  dockeragent:
    Container ID:   containerd://87c9478fe8b7f46104cda7017512c6a38b340489c4c9ba9c7d91c63c8edb457c
    Image:          idjohnson/pivstsagent:ndgjvdocker
    Image ID:       docker.io/idjohnson/pivstsagent@sha256:5c72b0c6b5a673c6b75148fd04f4bdea02c47189193a730b44f35d5dfe22ffa5
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sun, 16 Feb 2020 07:19:42 -0600
    Ready:          True
    Restart Count:  0
    Environment:
      AZP_TOKEN:       <set to the key 'AZP_TOKEN' in secret 'azdo-secret-config'>         Optional: false
      AZP_URL:         <set to the key 'AZP_URL' of config map 'azdo-environment-config'>  Optional: false
      AZP_AGENT_NAME:  isaac-vsts-agent-deployment-645bcb968c-qgxbh (v1:metadata.name)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xm6zh (ro)
...

While the docker binary will be there, it wont actually be able to work (because we are missing a few key things like iptables and lxc).  While digging into this, another blogger pointed out that we really aren't looking to run docker in docker, merely automate CI/CD from within a containerized agent.  So he suggested exposing the docker sock of the host, which makes a lot more sense.

First, get the groupid of the dockersock;

pi@raspberrypi:~/dockeragent $ ls -l /var/run/docker.sock 
srw-rw---- 1 root docker 0 Feb 14 22:25 /var/run/docker.sock
pi@raspberrypi:~/dockeragent $ cat /etc/group | grep docker
docker:x:995:pi

We can then apply it in the yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: isaac-vsts-agent-deployment
  labels:
    app: isaac-vsts-agent-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: isaac-vsts-agent-deployment
  template:
    metadata:
      labels:
        app: isaac-vsts-agent-deployment
    spec:
      securityContext:
        fsGroup: 995    # Group ID of docker group on k2s node.
      containers:
      - name: dockeragent
        image: idjohnson/pivstsagent:ndgjvdocker
        env:
          - name: AZP_TOKEN
            valueFrom:
              secretKeyRef:
                name: azdo-secret-config
                key: AZP_TOKEN
          - name: AZP_URL
            valueFrom:
              configMapKeyRef:
                name: azdo-environment-config
                key: AZP_URL
          - name: AZP_AGENT_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        volumeMounts:
          - name: dockersock
            mountPath: "/var/run/docker.sock"
      volumes:
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
---
apiVersion: v1
kind: Secret
metadata:
  name: azdo-secret-config
data:
  AZP_TOKEN: bm9wZS1ub3QtcmVhbGx5LW15LXRva2VuLWJ1dC10aGFua3MtZm9yLXJlYWRpbmctdGhlLWJsb2cK=
---
apiVersion: v1 
kind: ConfigMap 
metadata:
  name: azdo-environment-config
data:
  AZP_URL: https://dev.azure.com/princessking/
  AZP_AGENT_NAME: pi4agent

And after applying the yaml:

$ kubectl apply -f vsts-deployment.yaml
deployment.apps/isaac-vsts-agent-deployment configured
secret/azdo-secret-config unchanged
configmap/azdo-environment-config unchanged

We can login and see that docker is working:

$ kubectl exec -it isaac-vsts-agent-deployment-576bd97bf9-9xj2c -- /bin/bash
groups: cannot find name for group ID 995
root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
idjohnson/pivstsagent   ndgjvdocker         00e6a46d1747        2 hours ago         1.01GB

On pushing

As it stands, we can take and pull, but when we try and push, we will get blocked:

root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker push idjohnson/ubuntu:test
The push refers to repository [docker.io/idjohnson/ubuntu]
8fdf48a435b0: Preparing 
89961002fb88: Preparing 
160f8b9f32ca: Preparing 
c03e0758467b: Preparing 
denied: requested access to the resource is denied

This is because the credentials are stored locally and not via the socket.

If we login to docker.io (or the registry of our choice), we should be able to push just fine:

root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker login -u idjohnson
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker push idjohnson/ubuntu:test
The push refers to repository [docker.io/idjohnson/ubuntu]
8fdf48a435b0: Mounted from idjohnson/pivstsagent 
89961002fb88: Mounted from idjohnson/pivstsagent 
160f8b9f32ca: Mounted from idjohnson/pivstsagent 
c03e0758467b: Mounted from idjohnson/pivstsagent 
test: digest: sha256:cd400513ac48600238b03cb2506fe433467c167faf4cd5694a6dd8b445e8b8e9 size: 1150

Putting it all together - CI / Pipelines

Let’s test this out.  I’ll create a new git repo with an index.html, Dockerfile and initialized azure-pipelines.yaml that uses our Default node pool.  I did make a Service Connection (called dockerhub) with my docker.io credentials in advance that this pipeline will be able to leverage.


$ cat azure-pipelines.yml 
# Docker
# Build a Docker image 
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- master

resources:
- repo: self

variables:
  tag: '$(Build.BuildId)'

stages:
- stage: Build
  displayName: Build image
  jobs:  
  - job: Build
    pool: Default
    displayName: Build
    steps:
    - task: Docker@2
      inputs:
        containerRegistry: 'dockerhub'
        repository: 'idjohnson/helloworld'
        command: 'buildAndPush'
        Dockerfile: '$(Build.SourcesDirectory)/Dockerfile'

$ cat index.html 
<HTML>
<HEAD>
<TITLE>Hello World</TITLE>
</HEAD>
<BODY>
I'm on a Pi!
</BODY>
</HTML>

$ cat Dockerfile 
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html

Running the pipeline, we see it build and push the image:

And we can see the resultant image on dockerhub: https://hub.docker.com/repository/docker/idjohnson/helloworld

We can even see the tags the build is using and verify that indeed the architecture is arm:

And if you are thinking, great, now how do we use, let’s finish this up and deploy to our k3s cluster to see it in action:

$ cat my-hello-world.yaml 
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: isaac-hello-world
spec:
  selector:
    matchLabels:
      app: isaac-hello-world
  replicas: 1
  template:
    metadata:
      labels:
        app: isaac-hello-world
    spec:
      containers:
      - name: isaac-hello-world
        image: idjohnson/helloworld:414
        ports:
        - containerPort: 80
$ kubectl apply -f my-hello-world.yaml 
deployment.apps/isaac-hello-world created

$ kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
isaac-vsts-agent-deployment-576bd97bf9-9xj2c   1/1     Running   0          50m
isaac-vsts-agent-deployment-576bd97bf9-b84x4   1/1     Running   0          50m
isaac-hello-world-579d5b9796-h9pxx             1/1     Running   0          27s

Testing is as easy as a quick port forward (from our laptop, not the pi, of course)

$ kubectl port-forward isaac-hello-world-579d5b9796-h9pxx 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080

Summary

The Raspberry Pi4 is a very functional little box.  I spent $100 on Amazon (https://www.amazon.com/gp/product/B07YRSYR3M) to get a full kit - everything you see above was run on it.  And if you have most of the bits, Amazon has Pi4’s with 4Gb as low as $62 right now.  

K3s seems to have improved from our last go running incredibly well on the Pi4.  We created several containers and verified we could push them from the pi4 to docker.io.  Then we created an Azure DevOps build agent on the Pi, loaded with tools and lastly verified we could create an alpine based nginx container to host a helloworld app (which we verified by running on the same pi).