K3S, Pi4, and Azure DevOps
We've explored Rancher's K3s before as well as K3OS. It's a pretty handy lightweight kubernetes environment. However, past attempts to make my Pi3b's proper containerized build agent pools ultimately fell down due it's limited RAM (1Gb). The Pi4 has a 4Gb option which puts it in the running.. So how can we take what is (presently) about a $65 hobbiest board and do something cool with it? Let's dig in - you might be suprised how capable it really is.
Setting up an initial Azure DevOps agent
We assume you’ve set up the Pi4 to have a functional instance of Raspbian running on it.
Next you'll need to create the necessary files for the Azure DevOps docker agent (start.sh and Dockerfile)
(from https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops) but with a key difference being the image platform (linux-arm instead of linux-x64)
pi@raspberrypi:~/dockeragent $ cat start.sh
#!/bin/bash
set -e
if [ -z "$AZP_URL" ]; then
echo 1>&2 "error: missing AZP_URL environment variable"
exit 1
fi
if [ -z "$AZP_TOKEN_FILE" ]; then
if [ -z "$AZP_TOKEN" ]; then
echo 1>&2 "error: missing AZP_TOKEN environment variable"
exit 1
fi
AZP_TOKEN_FILE=/azp/.token
echo -n $AZP_TOKEN > "$AZP_TOKEN_FILE"
fi
unset AZP_TOKEN
if [ -n "$AZP_WORK" ]; then
mkdir -p "$AZP_WORK"
fi
rm -rf /azp/agent
mkdir /azp/agent
cd /azp/agent
export AGENT_ALLOW_RUNASROOT="1"
cleanup() {
if [ -e config.sh ]; then
print_header "Cleanup. Removing Azure Pipelines agent..."
./config.sh remove --unattended \
--auth PAT \
--token $(cat "$AZP_TOKEN_FILE")
fi
}
print_header() {
lightcyan='\033[1;36m'
nocolor='\033[0m'
echo -e "${lightcyan}$1${nocolor}"
}
# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE=AZP_TOKEN,AZP_TOKEN_FILE
print_header "1. Determining matching Azure Pipelines agent..."
AZP_AGENT_RESPONSE=$(curl -LsS \
-u user:$(cat "$AZP_TOKEN_FILE") \
-H 'Accept:application/json;api-version=3.0-preview' \
"$AZP_URL/_apis/distributedtask/packages/agent?platform=linux-arm")
if echo "$AZP_AGENT_RESPONSE" | jq . >/dev/null 2>&1; then
AZP_AGENTPACKAGE_URL=$(echo "$AZP_AGENT_RESPONSE" \
| jq -r '.value | map([.version.major,.version.minor,.version.patch,.downloadUrl]) | sort | .[length-1] | .[3]')
fi
if [ -z "$AZP_AGENTPACKAGE_URL" -o "$AZP_AGENTPACKAGE_URL" == "null" ]; then
echo 1>&2 "error: could not determine a matching Azure Pipelines agent - check that account '$AZP_URL' is correct and the token is valid for that account"
exit 1
fi
print_header "2. Downloading and installing Azure Pipelines agent..."
curl -LsS $AZP_AGENTPACKAGE_URL | tar -xz & wait $!
source ./env.sh
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM
print_header "3. Configuring Azure Pipelines agent..."
./config.sh --unattended \
--agent "${AZP_AGENT_NAME:-$(hostname)}" \
--url "$AZP_URL" \
--auth PAT \
--token $(cat "$AZP_TOKEN_FILE") \
--pool "${AZP_POOL:-Default}" \
--work "${AZP_WORK:-_work}" \
--replace \
--acceptTeeEula & wait $!
# remove the administrative token before accepting work
rm $AZP_TOKEN_FILE
print_header "4. Running Azure Pipelines agent..."
# `exec` the node runtime so it's aware of TERM and INT signals
# AgentService.js understands how to handle agent self-update and restart
exec ./externals/node/bin/node ./bin/AgentService.js interactive
pi@raspberrypi:~/dockeragent $ cat Dockerfile
FROM ubuntu:16.04
# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl3 \
libicu55 \
libunwind8 \
netcat
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh"]
Installing docker on pi
Next, let’s install docker on the pi. We can generally follow steps from https://linuxize.com/post/how-to-install-and-use-docker-on-raspberry-pi/
pi@raspberrypi:~ $ curl -fsSL https://get.docker.com -o get-docker.sh
pi@raspberrypi:~ $ sh get-docker.sh
# Executing docker install script, commit: f45d7c11389849ff46a6b4d94e0dd1ffebca32c1
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sudo -E sh -c curl -fsSL "https://download.docker.com/linux/raspbian/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sudo -E sh -c echo "deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable" > /etc/apt/sources.list.d/docker.list
+ sudo -E sh -c apt-get update -qq >/dev/null
+ [ -n ]
+ sudo -E sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
+ sudo -E sh -c docker version
Client: Docker Engine - Community
Version: 19.03.6
API version: 1.40
Go version: go1.12.16
Git commit: 369ce74
Built: Thu Feb 13 01:37:07 2020
OS/Arch: linux/arm
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.6
API version: 1.40 (minimum version 1.12)
Go version: go1.12.16
Git commit: 369ce74
Built: Thu Feb 13 01:31:06 2020
OS/Arch: linux/arm
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:
sudo usermod -aG docker pi
Remember that you will have to log out and back in for this to take effect!
WARNING: Adding a user to the "docker" group will grant the ability to run
containers which can be used to obtain root privileges on the
docker host.
Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
for more information.
Pro-tip ; if you get an access error, you likely just need to usermod the pi user
pi@raspberrypi:~/dockeragent $ docker build -t dockeragent:latest .
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=nl12q1vysdeyq6pbvnv16ou4m&shmsize=0&t=dockeragent%3Alatest&target=&ulimits=null&version=1: dial unix /var/run/docker.sock: connect: permission denied
pi@raspberrypi:~/dockeragent $ sudo usermod -aG docker pi
pi@raspberrypi:~/dockeragent $ exit
logout
Connection to 192.168.1.207 closed.
johnsi10$ ssh pi@192.168.1.207
pi@192.168.1.207's password:
Linux raspberrypi 4.19.93-v7l+ #1290 SMP Fri Jan 10 16:45:11 GMT 2020 armv7l
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Feb 14 20:56:53 2020 from 192.168.1.113
pi@raspberrypi:~ $ cd dockeragent/
pi@raspberrypi:~/dockeragent $ docker build -t dockeragent:latest .
We should now be able to build the image
pi@raspberrypi:~/dockeragent $ docker build -t dockeragent:latest .
Sending build context to Docker daemon 5.632kB
Step 1/8 : FROM ubuntu:16.04
16.04: Pulling from library/ubuntu
e60df59fb597: Pull complete
99ba0251fafa: Pull complete
cff14f7ed750: Pull complete
638c86faa077: Pull complete
Digest: sha256:3f3ee50cb89bc12028bab7d1e187ae57f12b957135b91648702e835c37c6c971
Status: Downloaded newer image for ubuntu:16.04
---> 771a2e2e1f23
Step 2/8 : ENV DEBIAN_FRONTEND=noninteractive
---> Running in b43e7f22dcc3
Removing intermediate container b43e7f22dcc3
---> 8196f523375c
Step 3/8 : RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
---> Running in f01e9b3f9c90
Removing intermediate container f01e9b3f9c90
---> fe563fb6aec0
Step 4/8 : RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl jq git iputils-ping libcurl3 libicu55 libunwind8 netcat
---> Running in e1ae490f1220
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial InRelease [247 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [109 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease [107 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease [109 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf Packages [1486 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports xenial/restricted armhf Packages [8491 B]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial/universe armhf Packages [9531 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial/multiverse armhf Packages [149 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf Packages [1045 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports xenial-updates/restricted armhf Packages [8488 B]
Get:11 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe armhf Packages [905 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports xenial-updates/multiverse armhf Packages [13.6 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports xenial-backports/main armhf Packages [7936 B]
Get:14 http://ports.ubuntu.com/ubuntu-ports xenial-backports/universe armhf Packages [8424 B]
Get:15 http://ports.ubuntu.com/ubuntu-ports xenial-security/main armhf Packages [714 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports xenial-security/restricted armhf Packages [8480 B]
Get:17 http://ports.ubuntu.com/ubuntu-ports xenial-security/universe armhf Packages [533 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports xenial-security/multiverse armhf Packages [3157 B]
Fetched 15.0 MB in 4s (3193 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
git-man libasn1-8-heimdal libcurl3-gnutls liberror-perl libexpat1 libffi6
libgdbm3 libgmp10 libgnutls-openssl27 libgnutls30 libgssapi-krb5-2
libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
libheimntlm0-heimdal libhogweed4 libhx509-5-heimdal libidn11 libk5crypto3
libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2
libnettle6 libonig2 libp11-kit0 libperl5.22 libroken18-heimdal librtmp1
libsasl2-2 libsasl2-modules-db libsqlite3-0 libssl1.0.0 libtasn1-6
libwind0-heimdal netcat-traditional openssl perl perl-modules-5.22
Suggested packages:
gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
git-gui gitk gitweb git-arch git-cvs git-mediawiki git-svn gnutls-bin
krb5-doc krb5-user perl-doc libterm-readline-gnu-perl
| libterm-readline-perl-perl make
Recommended packages:
patch less rsync ssh-client krb5-locales libsasl2-modules netbase rename
The following NEW packages will be installed:
ca-certificates curl git git-man iputils-ping jq libasn1-8-heimdal libcurl3
libcurl3-gnutls liberror-perl libexpat1 libffi6 libgdbm3 libgmp10
libgnutls-openssl27 libgnutls30 libgssapi-krb5-2 libgssapi3-heimdal
libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhogweed4
libhx509-5-heimdal libicu55 libidn11 libk5crypto3 libkeyutils1
libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libnettle6
libonig2 libp11-kit0 libperl5.22 libroken18-heimdal librtmp1 libsasl2-2
libsasl2-modules-db libsqlite3-0 libssl1.0.0 libtasn1-6 libunwind8
libwind0-heimdal netcat netcat-traditional openssl perl perl-modules-5.22
0 upgraded, 49 newly installed, 0 to remove and 17 not upgraded.
Need to get 21.0 MB of archives.
After this operation, 91.9 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libgdbm3 armhf 1.8.3-13.1 [15.3 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf perl-modules-5.22 all 5.22.1-9ubuntu0.6 [2629 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libperl5.22 armhf 5.22.1-9ubuntu0.6 [2728 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf perl armhf 5.22.1-9ubuntu0.6 [237 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libgmp10 armhf 2:6.1.0+dfsg-2 [184 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libnettle6 armhf 3.2-1ubuntu0.16.04.1 [111 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libhogweed4 armhf 3.2-1ubuntu0.16.04.1 [126 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libidn11 armhf 1.32-3ubuntu1.2 [43.1 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libffi6 armhf 3.2.1-4 [16.2 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libp11-kit0 armhf 0.23.2-5~ubuntu16.04.1 [91.0 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libtasn1-6 armhf 4.7-3ubuntu0.16.04.3 [37.9 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgnutls30 armhf 3.4.10-4ubuntu1.7 [485 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgnutls-openssl27 armhf 3.4.10-4ubuntu1.7 [19.0 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf iputils-ping armhf 3:20121221-5ubuntu2 [50.1 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libexpat1 armhf 2.1.0-7ubuntu0.16.04.5 [53.2 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libsqlite3-0 armhf 3.11.0-1ubuntu1.3 [338 kB]
Get:17 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libssl1.0.0 armhf 1.0.2g-1ubuntu4.15 [711 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf openssl armhf 1.0.2g-1ubuntu4.15 [485 kB]
Get:19 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf ca-certificates all 20170717~16.04.2 [167 kB]
Get:20 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libroken18-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [34.1 kB]
Get:21 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libasn1-8-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [138 kB]
Get:22 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libkrb5support0 armhf 1.13.2+dfsg-5ubuntu2.1 [27.5 kB]
Get:23 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libk5crypto3 armhf 1.13.2+dfsg-5ubuntu2.1 [77.6 kB]
Get:24 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libkeyutils1 armhf 1.5.9-8ubuntu1 [9192 B]
Get:25 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libkrb5-3 armhf 1.13.2+dfsg-5ubuntu2.1 [230 kB]
Get:26 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgssapi-krb5-2 armhf 1.13.2+dfsg-5ubuntu2.1 [98.8 kB]
Get:27 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libhcrypto4-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [75.7 kB]
Get:28 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libheimbase1-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [24.0 kB]
Get:29 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libwind0-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [47.0 kB]
Get:30 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libhx509-5-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [88.4 kB]
Get:31 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libkrb5-26-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [164 kB]
Get:32 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libheimntlm0-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [13.4 kB]
Get:33 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libgssapi3-heimdal armhf 1.7~git20150920+dfsg-4ubuntu1.16.04.1 [78.5 kB]
Get:34 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libsasl2-modules-db armhf 2.1.26.dfsg1-14ubuntu0.2 [13.0 kB]
Get:35 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libsasl2-2 armhf 2.1.26.dfsg1-14ubuntu0.2 [42.0 kB]
Get:36 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libldap-2.4-2 armhf 2.4.42+dfsg-2ubuntu3.7 [137 kB]
Get:37 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf librtmp1 armhf 2.4+20151223.gitfa8646d-1ubuntu0.1 [49.4 kB]
Get:38 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libcurl3-gnutls armhf 7.47.0-1ubuntu2.14 [159 kB]
Get:39 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libicu55 armhf 55.1-7ubuntu0.4 [7404 kB]
Get:40 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf curl armhf 7.47.0-1ubuntu2.14 [135 kB]
Get:41 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf liberror-perl all 0.17-1.2 [19.6 kB]
Get:42 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf git-man all 1:2.7.4-0ubuntu1.7 [736 kB]
Get:43 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf git armhf 1:2.7.4-0ubuntu1.7 [2182 kB]
Get:44 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe armhf libonig2 armhf 5.9.6-1ubuntu0.1 [72.5 kB]
Get:45 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe armhf jq armhf 1.5+dfsg-1ubuntu0.1 [144 kB]
Get:46 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf libcurl3 armhf 7.47.0-1ubuntu2.14 [162 kB]
Get:47 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf libunwind8 armhf 1.1-4.1 [43.6 kB]
Get:48 http://ports.ubuntu.com/ubuntu-ports xenial/universe armhf netcat-traditional armhf 1.10-41 [59.2 kB]
Get:49 http://ports.ubuntu.com/ubuntu-ports xenial/universe armhf netcat all 1.10-41 [3438 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 21.0 MB in 2s (8109 kB/s)
Selecting previously unselected package libgdbm3:armhf.
(Reading database ... 4771 files and directories currently installed.)
Preparing to unpack .../libgdbm3_1.8.3-13.1_armhf.deb ...
Unpacking libgdbm3:armhf (1.8.3-13.1) ...
Selecting previously unselected package perl-modules-5.22.
Preparing to unpack .../perl-modules-5.22_5.22.1-9ubuntu0.6_all.deb ...
Unpacking perl-modules-5.22 (5.22.1-9ubuntu0.6) ...
Selecting previously unselected package libperl5.22:armhf.
Preparing to unpack .../libperl5.22_5.22.1-9ubuntu0.6_armhf.deb ...
Unpacking libperl5.22:armhf (5.22.1-9ubuntu0.6) ...
Selecting previously unselected package perl.
Preparing to unpack .../perl_5.22.1-9ubuntu0.6_armhf.deb ...
Unpacking perl (5.22.1-9ubuntu0.6) ...
Selecting previously unselected package libgmp10:armhf.
Preparing to unpack .../libgmp10_2%3a6.1.0+dfsg-2_armhf.deb ...
Unpacking libgmp10:armhf (2:6.1.0+dfsg-2) ...
Selecting previously unselected package libnettle6:armhf.
Preparing to unpack .../libnettle6_3.2-1ubuntu0.16.04.1_armhf.deb ...
Unpacking libnettle6:armhf (3.2-1ubuntu0.16.04.1) ...
Selecting previously unselected package libhogweed4:armhf.
Preparing to unpack .../libhogweed4_3.2-1ubuntu0.16.04.1_armhf.deb ...
Unpacking libhogweed4:armhf (3.2-1ubuntu0.16.04.1) ...
Selecting previously unselected package libidn11:armhf.
Preparing to unpack .../libidn11_1.32-3ubuntu1.2_armhf.deb ...
Unpacking libidn11:armhf (1.32-3ubuntu1.2) ...
Selecting previously unselected package libffi6:armhf.
Preparing to unpack .../libffi6_3.2.1-4_armhf.deb ...
Unpacking libffi6:armhf (3.2.1-4) ...
Selecting previously unselected package libp11-kit0:armhf.
Preparing to unpack .../libp11-kit0_0.23.2-5~ubuntu16.04.1_armhf.deb ...
Unpacking libp11-kit0:armhf (0.23.2-5~ubuntu16.04.1) ...
Selecting previously unselected package libtasn1-6:armhf.
Preparing to unpack .../libtasn1-6_4.7-3ubuntu0.16.04.3_armhf.deb ...
Unpacking libtasn1-6:armhf (4.7-3ubuntu0.16.04.3) ...
Selecting previously unselected package libgnutls30:armhf.
Preparing to unpack .../libgnutls30_3.4.10-4ubuntu1.7_armhf.deb ...
Unpacking libgnutls30:armhf (3.4.10-4ubuntu1.7) ...
Selecting previously unselected package libgnutls-openssl27:armhf.
Preparing to unpack .../libgnutls-openssl27_3.4.10-4ubuntu1.7_armhf.deb ...
Unpacking libgnutls-openssl27:armhf (3.4.10-4ubuntu1.7) ...
Selecting previously unselected package iputils-ping.
Preparing to unpack .../iputils-ping_3%3a20121221-5ubuntu2_armhf.deb ...
Unpacking iputils-ping (3:20121221-5ubuntu2) ...
Selecting previously unselected package libexpat1:armhf.
Preparing to unpack .../libexpat1_2.1.0-7ubuntu0.16.04.5_armhf.deb ...
Unpacking libexpat1:armhf (2.1.0-7ubuntu0.16.04.5) ...
Selecting previously unselected package libsqlite3-0:armhf.
Preparing to unpack .../libsqlite3-0_3.11.0-1ubuntu1.3_armhf.deb ...
Unpacking libsqlite3-0:armhf (3.11.0-1ubuntu1.3) ...
Selecting previously unselected package libssl1.0.0:armhf.
Preparing to unpack .../libssl1.0.0_1.0.2g-1ubuntu4.15_armhf.deb ...
Unpacking libssl1.0.0:armhf (1.0.2g-1ubuntu4.15) ...
Selecting previously unselected package openssl.
Preparing to unpack .../openssl_1.0.2g-1ubuntu4.15_armhf.deb ...
Unpacking openssl (1.0.2g-1ubuntu4.15) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../ca-certificates_20170717~16.04.2_all.deb ...
Unpacking ca-certificates (20170717~16.04.2) ...
Selecting previously unselected package libroken18-heimdal:armhf.
Preparing to unpack .../libroken18-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libroken18-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libasn1-8-heimdal:armhf.
Preparing to unpack .../libasn1-8-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libasn1-8-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libkrb5support0:armhf.
Preparing to unpack .../libkrb5support0_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libkrb5support0:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libk5crypto3:armhf.
Preparing to unpack .../libk5crypto3_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libk5crypto3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libkeyutils1:armhf.
Preparing to unpack .../libkeyutils1_1.5.9-8ubuntu1_armhf.deb ...
Unpacking libkeyutils1:armhf (1.5.9-8ubuntu1) ...
Selecting previously unselected package libkrb5-3:armhf.
Preparing to unpack .../libkrb5-3_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libkrb5-3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libgssapi-krb5-2:armhf.
Preparing to unpack .../libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_armhf.deb ...
Unpacking libgssapi-krb5-2:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Selecting previously unselected package libhcrypto4-heimdal:armhf.
Preparing to unpack .../libhcrypto4-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libhcrypto4-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libheimbase1-heimdal:armhf.
Preparing to unpack .../libheimbase1-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libheimbase1-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libwind0-heimdal:armhf.
Preparing to unpack .../libwind0-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libwind0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libhx509-5-heimdal:armhf.
Preparing to unpack .../libhx509-5-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libhx509-5-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libkrb5-26-heimdal:armhf.
Preparing to unpack .../libkrb5-26-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libkrb5-26-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libheimntlm0-heimdal:armhf.
Preparing to unpack .../libheimntlm0-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libheimntlm0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libgssapi3-heimdal:armhf.
Preparing to unpack .../libgssapi3-heimdal_1.7~git20150920+dfsg-4ubuntu1.16.04.1_armhf.deb ...
Unpacking libgssapi3-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Selecting previously unselected package libsasl2-modules-db:armhf.
Preparing to unpack .../libsasl2-modules-db_2.1.26.dfsg1-14ubuntu0.2_armhf.deb ...
Unpacking libsasl2-modules-db:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Selecting previously unselected package libsasl2-2:armhf.
Preparing to unpack .../libsasl2-2_2.1.26.dfsg1-14ubuntu0.2_armhf.deb ...
Unpacking libsasl2-2:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Selecting previously unselected package libldap-2.4-2:armhf.
Preparing to unpack .../libldap-2.4-2_2.4.42+dfsg-2ubuntu3.7_armhf.deb ...
Unpacking libldap-2.4-2:armhf (2.4.42+dfsg-2ubuntu3.7) ...
Selecting previously unselected package librtmp1:armhf.
Preparing to unpack .../librtmp1_2.4+20151223.gitfa8646d-1ubuntu0.1_armhf.deb ...
Unpacking librtmp1:armhf (2.4+20151223.gitfa8646d-1ubuntu0.1) ...
Selecting previously unselected package libcurl3-gnutls:armhf.
Preparing to unpack .../libcurl3-gnutls_7.47.0-1ubuntu2.14_armhf.deb ...
Unpacking libcurl3-gnutls:armhf (7.47.0-1ubuntu2.14) ...
Selecting previously unselected package libicu55:armhf.
Preparing to unpack .../libicu55_55.1-7ubuntu0.4_armhf.deb ...
Unpacking libicu55:armhf (55.1-7ubuntu0.4) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.47.0-1ubuntu2.14_armhf.deb ...
Unpacking curl (7.47.0-1ubuntu2.14) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../liberror-perl_0.17-1.2_all.deb ...
Unpacking liberror-perl (0.17-1.2) ...
Selecting previously unselected package git-man.
Preparing to unpack .../git-man_1%3a2.7.4-0ubuntu1.7_all.deb ...
Unpacking git-man (1:2.7.4-0ubuntu1.7) ...
Selecting previously unselected package git.
Preparing to unpack .../git_1%3a2.7.4-0ubuntu1.7_armhf.deb ...
Unpacking git (1:2.7.4-0ubuntu1.7) ...
Selecting previously unselected package libonig2:armhf.
Preparing to unpack .../libonig2_5.9.6-1ubuntu0.1_armhf.deb ...
Unpacking libonig2:armhf (5.9.6-1ubuntu0.1) ...
Selecting previously unselected package jq.
Preparing to unpack .../jq_1.5+dfsg-1ubuntu0.1_armhf.deb ...
Unpacking jq (1.5+dfsg-1ubuntu0.1) ...
Selecting previously unselected package libcurl3:armhf.
Preparing to unpack .../libcurl3_7.47.0-1ubuntu2.14_armhf.deb ...
Unpacking libcurl3:armhf (7.47.0-1ubuntu2.14) ...
Selecting previously unselected package libunwind8.
Preparing to unpack .../libunwind8_1.1-4.1_armhf.deb ...
Unpacking libunwind8 (1.1-4.1) ...
Selecting previously unselected package netcat-traditional.
Preparing to unpack .../netcat-traditional_1.10-41_armhf.deb ...
Unpacking netcat-traditional (1.10-41) ...
Selecting previously unselected package netcat.
Preparing to unpack .../netcat_1.10-41_all.deb ...
Unpacking netcat (1.10-41) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libgdbm3:armhf (1.8.3-13.1) ...
Setting up perl-modules-5.22 (5.22.1-9ubuntu0.6) ...
Setting up libperl5.22:armhf (5.22.1-9ubuntu0.6) ...
Setting up perl (5.22.1-9ubuntu0.6) ...
update-alternatives: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode
Setting up libgmp10:armhf (2:6.1.0+dfsg-2) ...
Setting up libnettle6:armhf (3.2-1ubuntu0.16.04.1) ...
Setting up libhogweed4:armhf (3.2-1ubuntu0.16.04.1) ...
Setting up libidn11:armhf (1.32-3ubuntu1.2) ...
Setting up libffi6:armhf (3.2.1-4) ...
Setting up libp11-kit0:armhf (0.23.2-5~ubuntu16.04.1) ...
Setting up libtasn1-6:armhf (4.7-3ubuntu0.16.04.3) ...
Setting up libgnutls30:armhf (3.4.10-4ubuntu1.7) ...
Setting up libgnutls-openssl27:armhf (3.4.10-4ubuntu1.7) ...
Setting up iputils-ping (3:20121221-5ubuntu2) ...
Setcap is not installed, falling back to setuid
Setting up libexpat1:armhf (2.1.0-7ubuntu0.16.04.5) ...
Setting up libsqlite3-0:armhf (3.11.0-1ubuntu1.3) ...
Setting up libssl1.0.0:armhf (1.0.2g-1ubuntu4.15) ...
Setting up openssl (1.0.2g-1ubuntu4.15) ...
Setting up ca-certificates (20170717~16.04.2) ...
Setting up libroken18-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libasn1-8-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libkrb5support0:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libk5crypto3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libkeyutils1:armhf (1.5.9-8ubuntu1) ...
Setting up libkrb5-3:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libgssapi-krb5-2:armhf (1.13.2+dfsg-5ubuntu2.1) ...
Setting up libhcrypto4-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libheimbase1-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libwind0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libhx509-5-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libkrb5-26-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libheimntlm0-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libgssapi3-heimdal:armhf (1.7~git20150920+dfsg-4ubuntu1.16.04.1) ...
Setting up libsasl2-modules-db:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Setting up libsasl2-2:armhf (2.1.26.dfsg1-14ubuntu0.2) ...
Setting up libldap-2.4-2:armhf (2.4.42+dfsg-2ubuntu3.7) ...
Setting up librtmp1:armhf (2.4+20151223.gitfa8646d-1ubuntu0.1) ...
Setting up libcurl3-gnutls:armhf (7.47.0-1ubuntu2.14) ...
Setting up libicu55:armhf (55.1-7ubuntu0.4) ...
Setting up curl (7.47.0-1ubuntu2.14) ...
Setting up liberror-perl (0.17-1.2) ...
Setting up git-man (1:2.7.4-0ubuntu1.7) ...
Setting up git (1:2.7.4-0ubuntu1.7) ...
Setting up libonig2:armhf (5.9.6-1ubuntu0.1) ...
Setting up jq (1.5+dfsg-1ubuntu0.1) ...
Setting up libcurl3:armhf (7.47.0-1ubuntu2.14) ...
Setting up libunwind8 (1.1-4.1) ...
Setting up netcat-traditional (1.10-41) ...
update-alternatives: using /bin/nc.traditional to provide /bin/nc (nc) in auto mode
Setting up netcat (1.10-41) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for ca-certificates (20170717~16.04.2) ...
Updating certificates in /etc/ssl/certs...
148 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Removing intermediate container e1ae490f1220
---> 436f1a0a555b
Step 5/8 : WORKDIR /azp
---> Running in a6cd328afd5f
Removing intermediate container a6cd328afd5f
---> 8cd9408c3d2a
Step 6/8 : COPY ./start.sh .
---> a0106165900f
Step 7/8 : RUN chmod +x start.sh
---> Running in 7de0217ce723
Removing intermediate container 7de0217ce723
---> c741cd4e9480
Step 8/8 : CMD ["./start.sh"]
---> Running in 4b335dc618da
Removing intermediate container 4b335dc618da
---> eac4c6b0709e
Successfully built eac4c6b0709e
Successfully tagged dockeragent:latest
We can then push to dockerhub (or the registry of our choice):
pi@raspberrypi:~/dockeragent $ docker tag dockeragent:latest idjohnson/pivstsagent:latest
pi@raspberrypi:~/dockeragent $ docker push idjohnson/pivstsagent:latest
The push refers to repository [docker.io/idjohnson/pivstsagent]
299c2b3ce4f0: Pushed
bb599863d08a: Pushed
889a29fac21d: Pushed
c58c5431d648: Pushed
3fc1655ba429: Pushed
8fdf48a435b0: Mounted from library/ubuntu
89961002fb88: Mounted from library/ubuntu
160f8b9f32ca: Mounted from library/ubuntu
c03e0758467b: Pushed
latest: digest: sha256:3bd8022c881ef97ad2a527a3d644709bd8185616f43e3116336ff1637b9cde6e size: 2192
In my case, you’ll see the arm based Azure DevOps agent here:
https://hub.docker.com/repository/registry-1.docker.io/idjohnson/pivstsagent/tags?page=1
(and you can use it if you want: docker pull idjohnson/pivstsagent:latest)
launching an agent in our k3s cluster
Create the yaml that will hold our deployment, configmap and secret yaml. You can use base64 to encode your Azure DevOps PAT (token) for the secret file (e.g. echo mytoken | base64).
vsts-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: isaac-vsts-agent-deployment
labels:
app: isaac-vsts-agent-deployment
spec:
replicas: 1
selector:
matchLabels:
app: isaac-vsts-agent-deployment
template:
metadata:
labels:
app: isaac-vsts-agent-deployment
spec:
containers:
- name: dockeragent
image: idjohnson/pivstsagent:latest
env:
- name: AZP_TOKEN
valueFrom:
secretKeyRef:
name: azdo-secret-config
key: AZP_TOKEN
- name: AZP_URL
valueFrom:
configMapKeyRef:
name: azdo-environment-config
key: AZP_URL
- name: AZP_AGENT_NAME
valueFrom:
configMapKeyRef:
name: azdo-environment-config
key: AZP_AGENT_NAME
- name: testing.variable
value: "hello world"
---
apiVersion: v1
kind: Secret
metadata:
name: azdo-secret-config
data:
AZP_TOKEN: bm9wZS1ub3QtbXktdG9rZW4tYnV0LWdvb2QtdHJ5IQo=
---
apiVersion: v1
kind: ConfigMap
metadata:
name: azdo-environment-config
data:
AZP_URL: https://dev.azure.com/princessking/
AZP_AGENT_NAME: pi4agent
Now let’s launch it:
$ kubectl apply -f vsts-deployment.yaml
deployment.apps/isaac-vsts-agent-deployment created
secret/azdo-secret-config created
configmap/azdo-environment-config created
If we head over to our agent pools, we can see that it launched.
We can create a quick build pipeline to verify that indeed it’s running…
Making it more useful
This is great, but having some build tools will make this image far more useful in pipelines.
Let’s update the Dockerfile to include golang and java;
$ cat Dockerfile
FROM ubuntu:16.04
# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl3 \
libicu55 \
libunwind8 \
netcat
# Common build tools
RUN apt-get update \
&& apt-get install -y openjdk-8-jdk
RUN curl -O https://dl.google.com/go/go1.12.17.linux-armv6l.tar.gz \
&& tar -C /usr/local -xzf go1.12.17.linux-armv6l.tar.gz \
&& rm go1.12.17.linux-armv6l.tar.gz
RUN echo "PATH=$PATH:/usr/local/go/bin" >> ~/.profile
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh"]
In fact, if we want to get fancy, we can create a large docker container with Node, Go and Java!
$ cat Dockerfile
FROM ubuntu:16.04
# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl3 \
libicu55 \
libunwind8 \
netcat
# Common build tools
RUN apt-get update \
&& apt-get install -y openjdk-8-jdk
RUN curl -O https://dl.google.com/go/go1.12.17.linux-armv6l.tar.gz \
&& tar -C /usr/local -xzf go1.12.17.linux-armv6l.tar.gz \
&& rm go1.12.17.linux-armv6l.tar.gz
RUN echo "PATH=$PATH:/usr/local/go/bin" >> ~/.profile
# replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
# update the repository sources list
# and install dependencies
RUN apt-get update \
&& apt-get install -y curl \
&& apt-get -y autoclean
# nvm environment variables
ENV NVM_DIR /usr/local/nvm
ENV NODE_VERSION 4.4.7
# install nvm
# https://github.com/creationix/nvm#install-script
# instructions from https://gist.github.com/remarkablemark/aacf14c29b3f01d6900d13137b21db3a
RUN curl --silent -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.2/install.sh | bash
# install node and npm
RUN source $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default
# add node and npm to path so the commands are available
ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules
ENV PATH $NVM_DIR/versions/node/v$NODE_VERSION/bin:$PATH
# confirm installation
RUN node -v
RUN npm -v
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh"]
After we do a docker build, we can see some of our images as well as tag and push to docker.io
pi@raspberrypi:~/dockeragent $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
dockeragent latest 244a5838d922 5 seconds ago 749MB
<none> <none> 22c719477057 3 minutes ago 711MB
<none> <none> c7b91824198a 8 hours ago 711MB
idjohnson/pivstsagent latest 57ecdd599a28 9 hours ago 209MB
idjohnson/pivstsagent latest2 57ecdd599a28 9 hours ago 209MB
idjohnson/pivstsagent tagname eac4c6b0709e 20 hours ago 209MB
ubuntu 16.04 771a2e2e1f23 4 weeks ago 98.2MB
pi@raspberrypi:~/dockeragent $ docker tag dockeragent:latest idjohnson/pivstsagent:nodegojava
pi@raspberrypi:~/dockeragent $ docker push idjohnson/pivstsagent:nodegojava
The push refers to repository [docker.io/idjohnson/pivstsagent]
e4771494a2fc: Pushed
45ab68c49c1c: Pushed
091b58c7f80e: Pushed
08dcde28dc90: Pushed
15e5922ff065: Pushed
3e0ae270bcbd: Pushed
43c4d5af1be0: Pushed
2026ebeace84: Pushed
7af300e93a53: Pushing 181.7MB/259.8MB
13a8ad2a479f: Pushing 201.6MB/241.4MB
c58c5431d648: Layer already exists
3fc1655ba429: Layer already exists
8fdf48a435b0: Layer already exists
89961002fb88: Layer already exists
160f8b9f32ca: Layer already exists
As you can see, it’s a hefty image. Adding docker itself to it pushes it over the 1gb level.
pi@raspberrypi:~/dockeragent $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
dockeragent latest 00e6a46d1747 30 minutes ago 1.01GB
pi@raspberrypi:~/dockeragent $ docker tag dockeragent:latest idjohnson/pivstsagent:ndgjvdocker
pi@raspberrypi:~/dockeragent $ docker push idjohnson/pivstsagent:ndgjvdocker
The push refers to repository [docker.io/idjohnson/pivstsagent]
091fccd8f7ba: Pushed
180edef0eb30: Pushed
d5db2fd9c90b: Pushed
4b7562f44ee8: Pushed
08dcde28dc90: Layer already exists
15e5922ff065: Layer already exists
3e0ae270bcbd: Layer already exists
43c4d5af1be0: Layer already exists
2026ebeace84: Layer already exists
7af300e93a53: Layer already exists
13a8ad2a479f: Layer already exists
c58c5431d648: Layer already exists
3fc1655ba429: Layer already exists
8fdf48a435b0: Layer already exists
89961002fb88: Layer already exists
160f8b9f32ca: Layer already exists
c03e0758467b: Layer already exists
ndgjvdocker: digest: sha256:5c72b0c6b5a673c6b75148fd04f4bdea02c47189193a730b44f35d5dfe22ffa5 size: 3876
One thing we can do, so that we can actually leverage multiple agents from the deployment, is to dynamically set the agent name - this allows us to have a replicaset scaled up to meet our needs:
apiVersion: apps/v1
kind: Deployment
metadata:
name: isaac-vsts-agent-deployment
labels:
app: isaac-vsts-agent-deployment
spec:
replicas: 2
selector:
matchLabels:
app: isaac-vsts-agent-deployment
template:
metadata:
labels:
app: isaac-vsts-agent-deployment
spec:
containers:
- name: dockeragent
image: idjohnson/pivstsagent:ndgjvdocker
env:
- name: AZP_TOKEN
valueFrom:
secretKeyRef:
name: azdo-secret-config
key: AZP_TOKEN
- name: AZP_URL
valueFrom:
configMapKeyRef:
name: azdo-environment-config
key: AZP_URL
- name: AZP_AGENT_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
---
apiVersion: v1
kind: Secret
metadata:
name: azdo-secret-config
data:
AZP_TOKEN: bm9wZS1ub3QtcmVhbGx5LW15LXRva2VuLWJ1dC10aGFua3MtZm9yLXJlYWRpbmctdGhlLWJsb2cK=
You'll notice I dropped the configmap as the name is now pulled from 'metadata.name'
- name: AZP_AGENT_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
The result, when applied, is agents are now added with their pod names:
Scaling events or updating the image will cause the old agents to go offline in AzDO, which is expected:
We can also confirm the name came from metadata this time by looking at the pod details:
$ kubectl describe pod isaac-vsts-agent-deployment-645bcb968c-qgxbh
Name: isaac-vsts-agent-deployment-645bcb968c-qgxbh
Namespace: default
Priority: 0
Node: raspberrypi/192.168.1.207
Start Time: Sun, 16 Feb 2020 07:19:09 -0600
Labels: app=isaac-vsts-agent-deployment
pod-template-hash=645bcb968c
Annotations: <none>
Status: Running
IP: 10.42.0.18
Controlled By: ReplicaSet/isaac-vsts-agent-deployment-645bcb968c
Containers:
dockeragent:
Container ID: containerd://87c9478fe8b7f46104cda7017512c6a38b340489c4c9ba9c7d91c63c8edb457c
Image: idjohnson/pivstsagent:ndgjvdocker
Image ID: docker.io/idjohnson/pivstsagent@sha256:5c72b0c6b5a673c6b75148fd04f4bdea02c47189193a730b44f35d5dfe22ffa5
Port: <none>
Host Port: <none>
State: Running
Started: Sun, 16 Feb 2020 07:19:42 -0600
Ready: True
Restart Count: 0
Environment:
AZP_TOKEN: <set to the key 'AZP_TOKEN' in secret 'azdo-secret-config'> Optional: false
AZP_URL: <set to the key 'AZP_URL' of config map 'azdo-environment-config'> Optional: false
AZP_AGENT_NAME: isaac-vsts-agent-deployment-645bcb968c-qgxbh (v1:metadata.name)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xm6zh (ro)
...While the docker binary will be there, it wont actually be able to work (because we are missing a few key things like iptables and lxc). While digging into this, another blogger pointed out that we really aren't looking to run docker in docker, merely automate CI/CD from within a containerized agent. So he suggested exposing the docker sock of the host, which makes a lot more sense.
First, get the groupid of the dockersock;
pi@raspberrypi:~/dockeragent $ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Feb 14 22:25 /var/run/docker.sock
pi@raspberrypi:~/dockeragent $ cat /etc/group | grep docker
docker:x:995:pi
We can then apply it in the yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: isaac-vsts-agent-deployment
labels:
app: isaac-vsts-agent-deployment
spec:
replicas: 2
selector:
matchLabels:
app: isaac-vsts-agent-deployment
template:
metadata:
labels:
app: isaac-vsts-agent-deployment
spec:
securityContext:
fsGroup: 995 # Group ID of docker group on k2s node.
containers:
- name: dockeragent
image: idjohnson/pivstsagent:ndgjvdocker
env:
- name: AZP_TOKEN
valueFrom:
secretKeyRef:
name: azdo-secret-config
key: AZP_TOKEN
- name: AZP_URL
valueFrom:
configMapKeyRef:
name: azdo-environment-config
key: AZP_URL
- name: AZP_AGENT_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- name: dockersock
mountPath: "/var/run/docker.sock"
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
---
apiVersion: v1
kind: Secret
metadata:
name: azdo-secret-config
data:
AZP_TOKEN: bm9wZS1ub3QtcmVhbGx5LW15LXRva2VuLWJ1dC10aGFua3MtZm9yLXJlYWRpbmctdGhlLWJsb2cK=
---
apiVersion: v1
kind: ConfigMap
metadata:
name: azdo-environment-config
data:
AZP_URL: https://dev.azure.com/princessking/
AZP_AGENT_NAME: pi4agent
And after applying the yaml:
$ kubectl apply -f vsts-deployment.yaml
deployment.apps/isaac-vsts-agent-deployment configured
secret/azdo-secret-config unchanged
configmap/azdo-environment-config unchanged
We can login and see that docker is working:
$ kubectl exec -it isaac-vsts-agent-deployment-576bd97bf9-9xj2c -- /bin/bash
groups: cannot find name for group ID 995
root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
idjohnson/pivstsagent ndgjvdocker 00e6a46d1747 2 hours ago 1.01GB
On pushing
As it stands, we can take and pull, but when we try and push, we will get blocked:
root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker push idjohnson/ubuntu:test
The push refers to repository [docker.io/idjohnson/ubuntu]
8fdf48a435b0: Preparing
89961002fb88: Preparing
160f8b9f32ca: Preparing
c03e0758467b: Preparing
denied: requested access to the resource is denied
This is because the credentials are stored locally and not via the socket.
If we login to docker.io (or the registry of our choice), we should be able to push just fine:
root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker login -u idjohnson
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
root@isaac-vsts-agent-deployment-576bd97bf9-9xj2c:/azp# docker push idjohnson/ubuntu:test
The push refers to repository [docker.io/idjohnson/ubuntu]
8fdf48a435b0: Mounted from idjohnson/pivstsagent
89961002fb88: Mounted from idjohnson/pivstsagent
160f8b9f32ca: Mounted from idjohnson/pivstsagent
c03e0758467b: Mounted from idjohnson/pivstsagent
test: digest: sha256:cd400513ac48600238b03cb2506fe433467c167faf4cd5694a6dd8b445e8b8e9 size: 1150
Putting it all together - CI / Pipelines
Let’s test this out. I’ll create a new git repo with an index.html, Dockerfile and initialized azure-pipelines.yaml that uses our Default node pool. I did make a Service Connection (called dockerhub) with my docker.io credentials in advance that this pipeline will be able to leverage.
$ cat azure-pipelines.yml
# Docker
# Build a Docker image
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- master
resources:
- repo: self
variables:
tag: '$(Build.BuildId)'
stages:
- stage: Build
displayName: Build image
jobs:
- job: Build
pool: Default
displayName: Build
steps:
- task: Docker@2
inputs:
containerRegistry: 'dockerhub'
repository: 'idjohnson/helloworld'
command: 'buildAndPush'
Dockerfile: '$(Build.SourcesDirectory)/Dockerfile'
$ cat index.html
<HTML>
<HEAD>
<TITLE>Hello World</TITLE>
</HEAD>
<BODY>
I'm on a Pi!
</BODY>
</HTML>
$ cat Dockerfile
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html
Running the pipeline, we see it build and push the image:
And we can see the resultant image on dockerhub: https://hub.docker.com/repository/docker/idjohnson/helloworld
We can even see the tags the build is using and verify that indeed the architecture is arm:
And if you are thinking, great, now how do we use, let’s finish this up and deploy to our k3s cluster to see it in action:
$ cat my-hello-world.yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: isaac-hello-world
spec:
selector:
matchLabels:
app: isaac-hello-world
replicas: 1
template:
metadata:
labels:
app: isaac-hello-world
spec:
containers:
- name: isaac-hello-world
image: idjohnson/helloworld:414
ports:
- containerPort: 80
$ kubectl apply -f my-hello-world.yaml
deployment.apps/isaac-hello-world created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
isaac-vsts-agent-deployment-576bd97bf9-9xj2c 1/1 Running 0 50m
isaac-vsts-agent-deployment-576bd97bf9-b84x4 1/1 Running 0 50m
isaac-hello-world-579d5b9796-h9pxx 1/1 Running 0 27s
Testing is as easy as a quick port forward (from our laptop, not the pi, of course)
$ kubectl port-forward isaac-hello-world-579d5b9796-h9pxx 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080
Summary
The Raspberry Pi4 is a very functional little box. I spent $100 on Amazon (https://www.amazon.com/gp/product/B07YRSYR3M) to get a full kit - everything you see above was run on it. And if you have most of the bits, Amazon has Pi4’s with 4Gb as low as $62 right now.
K3s seems to have improved from our last go running incredibly well on the Pi4. We created several containers and verified we could push them from the pi4 to docker.io. Then we created an Azure DevOps build agent on the Pi, loaded with tools and lastly verified we could create an alpine based nginx container to host a helloworld app (which we verified by running on the same pi).