Published: Apr 30, 2024 by Isaac Johnson
I came across Zarf from this Help Net Security review. It is based on an intriguing concept - a suite, driven with YAML manifests, that can bundle a variety of package types into a single archive (zst file) that can then be delivered to a disconnect kubernetes cluster.
Today we will set it up and try the demo DOS games (including a Blizzard classic). Weβll then try packaging our own content and delivering it to a fresh kubernetes cluster.
Installation
Zarf can be installed with homebrew, a Github release or just a bash script.
For this setup, weβll use brew
on WSL.
First I add the tap
$ brew tap defenseunicorns/tap
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
Updated 9 taps (codefresh-io/cli, mogenius/punq, derailed/popeye, derailed/k9s, kdash-rs/kdash, azure/functions, f1bonacc1/tap, homebrew/core and homebrew/cask).
==> New Formulae
... snip ...
==> Tapping defenseunicorns/tap
Cloning into '/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/defenseunicorns/homebrew-tap'...
remote: Enumerating objects: 661, done.
remote: Counting objects: 100% (200/200), done.
remote: Compressing objects: 100% (106/106), done.
remote: Total 661 (delta 129), reused 144 (delta 94), pack-reused 461
Receiving objects: 100% (661/661), 99.30 KiB | 1.20 MiB/s, done.
Resolving deltas: 100% (413/413), done.
Tapped 99 formulae (111 files, 336.6KB).
Then install
$ brew install zarf
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Fetching dependencies for defenseunicorns/tap/zarf: linux-headers@5.15, xz, zlib, zstd and binutils
==> Fetching linux-headers@5.15
==> Downloading https://ghcr.io/v2/homebrew/core/linux-headers/5.15/manifests/5.15.156
################################################################################################################# 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/linux-headers/5.15/blobs/sha256:8c53726f26e30f967b80e67d045f82ae44e29cc
################################################################################################################# 100.0%
==> Fetching xz
==> Downloading https://ghcr.io/v2/homebrew/core/xz/manifests/5.4.6
################################################################################################################# 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/xz/blobs/sha256:0736983b952c5273bb5a345008bac7311c2f4b60758d69cc05495d5
################################################################################################################# 100.0%
==> Fetching zlib
==> Downloading https://ghcr.io/v2/homebrew/core/zlib/manifests/1.3.1
... snip ...
==> Pouring expat--2.6.2.x86_64_linux.bottle.tar.gz
πΊ /home/linuxbrew/.linuxbrew/Cellar/expat/2.6.2: 22 files, 895.4KB
==> Running `brew cleanup expat`...
Removing: /home/linuxbrew/.linuxbrew/Cellar/expat/2.5.0... (21 files, 858KB)
==> Upgrading fontconfig
2.14.2 -> 2.15.0
==> Installing dependencies for fontconfig: util-linux
==> Installing fontconfig dependency: util-linux
==> Downloading https://ghcr.io/v2/homebrew/core/util-linux/manifests/2.39.3-1
Already downloaded: /home/builder/.cache/Homebrew/downloads/811c2708e034b96b17e1a032fe2265d27e09e0deba568a4bbb1f39b6caef4510--util-linux-2.39.3-1.bottle_manifest.json
==> Pouring util-linux--2.39.3.x86_64_linux.bottle.1.tar.gz
Error: Cannot link util-linux
Another version is already linked: /home/linuxbrew/.linuxbrew/Cellar/util-linux/2.39.3
I can now see it installed
$ which zarf
/home/linuxbrew/.linuxbrew/bin/zarf
$ zarf version
v0.33.0
Letβs do a quick test install first of dos-games
which is an example they have available which deploys a βretro arcadeβ
Iβll login to my test cluster
$ kubectx mac81
Switched to context "mac81".
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
anna-macbookair Ready control-plane,master 127d v1.27.6+k3s1
isaac-macbookpro Ready <none> 127d v1.27.6+k3s1
builder-macbookpro2 Ready <none> 127d v1.27.6+k3s1
I fire zarf init, which has such lovely ASCII art I feel compelled to share it as a screenshot:
Iβll confirm the prompts which pull the zarf init package (about 560Mb)
NOTE Note: This will require an internet connection.
? Do you want to pull this init package? Yes
β Pulling "oci://ghcr.io/defenseunicorns/packages/init:v0.33.0" (559.30 MBs)
β Validating full package checksums
β Loading package from "/home/builder/.zarf-cache/zarf-init-amd64-v0.33.0.tar.zst"
β Loading package from "/home/builder/.zarf-cache/zarf-init-amd64-v0.33.0.tar.zst"
π¦ PACKAGE DEFINITION
kind: ZarfInitConfig
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
metadata: information about this package
name: init
description: Used to establish a new Zarf cluster
version: v0.33.0
architecture: amd64
aggregateChecksum: 18317dcf28170103fca0c727e3635c5e07ee3bb454c67a00c7eb0ba99c9b51fa
The first time through I found it wanted to install Gitea for gitops and got stuck trying to write to the /opt
directory
Deploys Gitea to provide git repositories for Kubernetes configurations. Required for GitOps
deployments if no other git server is available.
? Deploy the git-server component? Yes
π¦ K3S COMPONENT
NOTE Saving log file to /tmp/zarf-2024-04-18-16-16-55-248458651.log
β Completed "Check if the current system has a, RFC1123 compliant hostname"
β Completed "If running a RHEL variant, disable 'firewalld' per k3s docs"
β Completed "Check that the host architecture matches the package architecture"
β Saving /opt/zarf/zarf-clean-k3s.sh
ERROR: unable to deploy component "k3s": unable to process the component files: unable to copy file /tmp/zarf-3934827885/components/k3s/files/0/zarf-clean-k3s.sh to /opt/zarf/zarf-clean-k3s.sh: mkdir /opt/zarf: permission denied
I made that path for it to use and gave my builder user perms
builder@DESKTOP-QADGF36:~/Workspaces/Weather-App$ sudo mkdir /opt/zarf
[sudo] password for builder:
builder@DESKTOP-QADGF36:~/Workspaces/Weather-App$ sudo chown builder:builder /opt/zarf
Then it complained about trying to direct access k3s (which isnβt local)
ERROR: unable to deploy component "k3s": unable to process the component files: unable to copy file /tmp/zarf-3927528168/components/k3s/files/1/k3s.service to /etc/systemd/system/k3s.service: open /etc/systemd/system/k3s.service: permission denied
I realized this was caused by saying yes to the first prompt on βk3s componentβ.
I was successful when opting out of that component:
I did get one stuck pod which I nuked
builder@DESKTOP-QADGF36:~$ kubectl get pods -n zarf
NAME READY STATUS RESTARTS AGE
zarf-docker-registry-568bff85fd-s74sf 1/1 Running 0 8m2s
agent-hook-5d7d5d55fb-m7bjf 1/1 Running 0 7m37s
agent-hook-5d7d5d55fb-xwr25 0/1 ImagePullBackOff 0 7m37s
builder@DESKTOP-QADGF36:~$ kubectl delete pod agent-hook-5d7d5d55fb-xwr25 -n zarf
pod "agent-hook-5d7d5d55fb-xwr25" deleted
builder@DESKTOP-QADGF36:~$ kubectl get pods -n zarf
NAME READY STATUS RESTARTS AGE
zarf-docker-registry-568bff85fd-s74sf 1/1 Running 0 78m
it then wrapped up with errors
β Starting helm chart generation zarf-agent
β Deployment is not ready: zarf/agent-hook. 1 out of 2 expected pods are ready (15m0s))
WARNING Retrying (1/3) in 5s: unable to complete the helm chart install/upgrade: context deadline
exceeded
β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error: context deadline exceeded
WARNING Retrying (2/3) in 10s: unable to complete the helm chart install/upgrade: client rate
limiter Wait returned an error: context deadline exceeded
β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error: context deadline exceeded (45m17s)
WARNING Retrying (3/3) in 20s: unable to complete the helm chart install/upgrade: client rate
limiter Wait returned an error: context deadline exceeded
β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ¦ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β § warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ή warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Έ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β Ό warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β ΄ warning: Upgrade "zarf-d2db14ef40305397791454e883b26fc94ad9615d" failed: client rate limiter Wait returned an error β purge requested for zarf-d2db14ef40305397791454e883b26fc94ad9615d (45m37s)
...
I worried a bit about whether the test cluster was just too overused.
I decided it would be best to try on a clean K3s. I spun a fresh k3s install of v1.27.6-k3s1 and tried again.
This time it had no issues.
Next, I tried deploying the arcade package
$ zarf package deploy oci://ghcr.io/defenseunicorns/packages/dos-games:1.0.0-$(uname -m) --key=https://zarf.dev/cosign.pub
NOTE Saving log file to /tmp/zarf-2024-04-20-16-44-23-4054700597.log
β Pulling "oci://ghcr.io/defenseunicorns/packages/dos-games:1.0.0-x86_64" (25.12 MBs)
β Validating pulled layer checksums
Verified OK
β Package signature validated!
π¦ PACKAGE DEFINITION
kind: ZarfPackageConfig
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
metadata: information about this package
name: dos-games
description: Simple example to load classic DOS games into K8s in the airgap
version: 1.0.0
architecture: amd64
aggregateChecksum: 8dc04134f44314bdba3e30fb24d8dc779a1e6315aea683c4046b293e30830ddd
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
build: info about the machine, zarf version, and user that created this package
terminal: fv-az254-28
user: runner
architecture: amd64
timestamp: Mon, 30 Oct 2023 20:03:58 +0000
version: UnknownVersion
migrations:
- scripts-to-actions
- pluralize-set-variable
lastNonBreakingVersion: v0.27.0
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
components: components selected for this operation
- name: baseline
required: true
manifests:
- name: multi-games
namespace: dos-games
files:
- manifests/deployment.yaml
- manifests/service.yaml
images:
- defenseunicorns/zarf-game:multi-tile-dark
actions:
onDeploy:
after:
- wait:
cluster:
kind: deployment
name: game
namespace: dos-games
condition: available
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Software Bill of Materials an inventory of all software contained in this package
NOTE This package has 1 artifact to be reviewed. This is available in a temporary
'zarf-sbom' folder in this directory and will be removed upon deployment.
- View SBOMs now by navigating to the 'zarf-sbom' folder or copying this link into a browser:
/home/builder/Workspaces/jekyll-blog/zarf-sbom/sbom-viewer-docker.io_defenseunicorns_zarf-game_multi-tile-dark.html
- View SBOMs after deployment with this command:
$ zarf package inspect oci://ghcr.io/defenseunicorns/packages/dos-games:1.0.0-x86_64
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
? Deploy this Zarf package? Yes
β Waiting for cluster connection (30s timeout)
β Gathering additional cluster information (if available)
π¦ BASELINE COMPONENT
β Loading the Zarf State from the Kubernetes cluster
β Pushed 1 images to the zarf registry
β Starting helm chart generation multi-games
β Processing helm chart raw-dos-games-baseline-multi-games:0.1.1713649463 from Zarf-generated helm chart
β Wait for "./zarf tools wait-for deployment game available -n dos-ga..." succeeded
β Zarf deployment complete
Connect Command | Description
zarf connect doom | Play doom!!!
zarf connect games | Play some old dos games π¦
Iβll try some DOS games.
I found myself spending 20 minutes just playing Warcraft II - such fond memories.
$ zarf connect games
NOTE Saving log file to /tmp/zarf-2024-04-22-06-44-34-868093331.log
β Ή Tunnel established at http://127.0.0.1:35399, opening your default web browser (ctrl-c to end) (2m19s)
Just too much funβ¦
Packaging
While DOS Games are fun, how might we use Zarf with real projects?
Recently we rolled out a Weather App in Python.
It had this manifest YAML
# apiVersion: v1
# kind: Secret
# metadata:
# name: pyweather-api-key
# type: Opaque
# data:
# API_KEY: YXNkZmFzZGY= # Base64-encoded value of "asdfasdf"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pyweatherapp-deployment
spec:
replicas: 1
selector:
matchLabels:
app: pyweatherapp
template:
metadata:
labels:
app: pyweatherapp
spec:
containers:
- name: pyweatherapp
image: idjohnson/pyweatherapp:1.0.0
ports:
- containerPort: 5000
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: pyweather-api-key
key: API_KEY
---
apiVersion: v1
kind: Service
metadata:
name: pyweatherapp-service
spec:
selector:
app: pyweatherapp
ports:
- protocol: TCP
port: 80
targetPort: 5000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: pyweatherapp-ingress
annotations:
cert-manager.io/cluster-issuer: azuredns-tpkpw
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: nginx
spec:
tls:
- hosts:
- pyweather.tpk.pw
secretName: pyweather-tls
rules:
- host: pyweather.tpk.pw
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: pyweatherapp-service
port:
number: 80
I can now create a Zarf package that will bundle both that manifest and itβs container
$ cat zarf.yaml
kind: ZarfPackageConfig
metadata:
name: manifests
version: 0.0.1
components:
- name: weather-local
required: true
manifests:
- name: pyweatherapp-deployment
namespace: weather
files:
# local manifests are specified relative to the `zarf.yaml` that uses them:
- manifest.yaml
actions:
onDeploy:
# the following checks were computed by viewing the success state of the package deployment
# and creating `wait` actions that match
after:
- wait:
cluster:
kind: deployment
name: pyweatherapp-deployment
namespace: weather
condition: "{.status.readyReplicas}=1"
# image discovery is supported in all manifests and charts using:
# zarf prepare find-images
images:
- idjohnson/pyweatherapp:1.0.0
I knew the image would be idjohnson/pyweatherapp:1.0.0
because I was familiar with the manifest.
However, if one is bundling Helm charts or other package types and is unsure, the command zarf prepare find-images
will check and display the appropriate images to include in your zarf package YAML:
builder@DESKTOP-QADGF36:~/Workspaces/Weather-App$ zarf prepare find-images
NOTE Saving log file to /tmp/zarf-2024-04-21-19-14-34-957849143.log
NOTE Using build directory .
β Looking for images in component "weather-local" across 3 resources
β Looking up cosign artifacts for discovered images (0/1)
components:
- name: weather-local
images:
- idjohnson/pyweatherapp:1.0.0
The next step is to create the package with zarf package create
builder@DESKTOP-QADGF36:~/Workspaces/Weather-App$ zarf package create .
NOTE Saving log file to /tmp/zarf-2024-04-21-19-19-54-2091137318.log
NOTE Using build directory .
π¦ PACKAGE DEFINITION
kind: ZarfPackageConfig
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
metadata: information about this package
name: manifests
version: 0.0.1
architecture: amd64
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
components: components selected for this operation
- name: weather-local
required: true
manifests:
- name: pyweatherapp-deployment
namespace: weather
files:
- manifest.yaml
images:
- idjohnson/pyweatherapp:1.0.0
actions:
onDeploy:
after:
- wait:
cluster:
kind: deployment
name: pyweatherapp-deployment
namespace: weather
condition: "{.status.readyReplicas}=1"
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
? Create this Zarf package? Yes
π¦ WEATHER-LOCAL COMPONENT
β Loading 1 K8s manifests
π¦ PACKAGE IMAGES
β Loading metadata for 1 images.
β Pulling 1 images (67.31 MBs)
β Creating SBOMs for 1 images and 0 components with files.
β Package saved to "zarf-package-manifests-amd64-0.0.1.tar.zst"
This creates a zst
package that looks to be the right size for including the container image
builder@DESKTOP-QADGF36:~/Workspaces/Weather-App$ ls -ltrah | tail -n1
-rw-r--r-- 1 builder builder 64M Apr 21 19:20 zarf-package-manifests-amd64-0.0.1.tar.zst
Next, we can use the package we just created in an install to a new cluster.
Here we can see a fresh test K8s cluster that has Zarf and just the DOS-games package
builder@DESKTOP-QADGF36:~/Workspaces/Weather-App$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-77ccd57875-gbf68 1/1 Running 0 28h
kube-system local-path-provisioner-957fdf8bc-6pwnx 1/1 Running 0 28h
kube-system helm-install-traefik-crd-c6p6r 0/1 Completed 0 28h
kube-system helm-install-traefik-xv627 0/1 Completed 1 28h
kube-system svclb-traefik-e626d69b-6rg8q 2/2 Running 0 28h
kube-system traefik-64f55bb67d-nl2f9 1/1 Running 0 28h
kube-system metrics-server-648b5df564-tm28s 1/1 Running 0 28h
kube-system svclb-traefik-e626d69b-jz6xz 2/2 Running 0 28h
kube-system svclb-traefik-e626d69b-67bsz 2/2 Running 0 26h
zarf zarf-docker-registry-7f7c48b8fd-srgvm 1/1 Running 0 26h
zarf agent-hook-5d7d5d55fb-n2j6j 1/1 Running 0 26h
zarf agent-hook-5d7d5d55fb-7tbnk 1/1 Running 0 26h
zarf zarf-loki-stack-promtail-hc924 1/1 Running 0 26h
zarf zarf-loki-stack-promtail-twxxt 1/1 Running 0 26h
zarf zarf-loki-stack-promtail-p9k7t 1/1 Running 0 26h
zarf zarf-loki-stack-0 1/1 Running 0 26h
zarf zarf-loki-stack-grafana-c69cbfb5f-8cd46 2/2 Running 0 26h
dos-games game-8dfc6fcfc-srplz 1/1 Running 0 26h
Now deploy with the following:
builder@DESKTOP-QADGF36:~/Workspaces/Weather-App$ zarf package deploy ./zarf-package-manifests-amd64-0.0.1.tar.zst
NOTE Saving log file to /tmp/zarf-2024-04-21-19-23-21-2752048272.log
β Loading package from "./zarf-package-manifests-amd64-0.0.1.tar.zst"
β Loading package from "./zarf-package-manifests-amd64-0.0.1.tar.zst"
π¦ PACKAGE DEFINITION
kind: ZarfPackageConfig
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
metadata: information about this package
name: manifests
version: 0.0.1
architecture: amd64
aggregateChecksum: ff68f026472845c12f7666a6ef75f4fee1eaaa40f59e0343dc61c6c61473f85b
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
build: info about the machine, zarf version, and user that created this package
terminal: DESKTOP-QADGF36
user: builder
architecture: amd64
timestamp: Sun, 21 Apr 2024 19:20:04 -0500
version: v0.33.0
migrations:
- scripts-to-actions
- pluralize-set-variable
lastNonBreakingVersion: v0.27.0
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
components: components selected for this operation
- name: weather-local
required: true
manifests:
- name: pyweatherapp-deployment
namespace: weather
files:
- manifest.yaml
images:
- idjohnson/pyweatherapp:1.0.0
actions:
onDeploy:
after:
- wait:
cluster:
kind: deployment
name: pyweatherapp-deployment
namespace: weather
condition: "{.status.readyReplicas}=1"
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Software Bill of Materials an inventory of all software contained in this package
NOTE This package has 1 artifact to be reviewed. This is available in a temporary
'zarf-sbom' folder in this directory and will be removed upon deployment.
- View SBOMs now by navigating to the 'zarf-sbom' folder or copying this link into a browser:
/home/builder/Workspaces/Weather-App/zarf-sbom/sbom-viewer-docker.io_idjohnson_pyweatherapp_1.0.0.html
- View SBOMs after deployment with this command:
$ zarf package inspect ./zarf-package-manifests-amd64-0.0.1.tar.zst
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
? Deploy this Zarf package? Yes
β Waiting for cluster connection (30s timeout)
β Gathering additional cluster information (if available)
π¦ WEATHER-LOCAL COMPONENT
β Loading the Zarf State from the Kubernetes cluster
β Pushed 1 images to the zarf registry
β Starting helm chart generation pyweatherapp-deployment
β Processing helm chart raw-manifests-weather-local-pyweatherapp-deployment:0.1.1713745401 from Zarf-generated helm chart
β Wait for "./zarf tools wait-for deployment pyweatherapp-deployment ..." succeeded
β Zarf deployment complete
You can watch the process below (as well as how I neglected to consider i needed to manually create an API secret I didnβt include in the manifest YAML)
And a quick test shows itβs working
AKS
I did try to get this running in Azure Kubernetes Service (AKS). However, I ran into several roadblocks and had to stop.
I debated leaving this in here, but in case others are trying, I figured I would share.
Iβll create an RG
builder@DESKTOP-QADGF36:~$ az group create --name idjtestrg --location centralus
{
"id": "/subscriptions/8defc61d-657a-453d-a6ff-cb9f91289a61/resourceGroups/idjtestrg",
"location": "centralus",
"managedBy": null,
"name": "idjtestrg",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}
And create (or update) an SP
$ az ad sp create-for-rbac -n idjaksupg01sp --skip-assignment --output json > my_sp.json
WARNING: Option '--skip-assignment' has been deprecated and will be removed in a future release.
WARNING: Found an existing application instance: (id) 34fede26-2ae7-4ccd-9c29-3bc220b9784a. We will patch it.
WARNING: The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
Next, I can use the credentials to create a fresh AKS clusters
$ export SP_PASS=`cat my_sp.json | jq -r .password`
$ export SP_ID=`cat my_sp.json | jq -r .appId`
$ az aks create --resource-group idjtestrg --name idjtestaks1 --location centralus --node-count 3 --enable-cluster-autoscaler --min-count 2 --max-count 4 --generate-ssh-keys --network-plugin azure --network-policy azure --service-principal $SP_ID --client-secret $SP_PASS
docker_bridge_cidr is not a known attribute of class <class 'azure.mgmt.containerservice.v2023_11_01.models._models_py3.ContainerServiceNetworkProfile'> and will be ignored
{
"aadProfile": null,
"addonProfiles": null,
"agentPoolProfiles": [
{
"availabilityZones": null,
"capacityReservationGroupId": null,
"count": 3,
... snip ...
"workloadAutoScalerProfile": {
"keda": null,
"verticalPodAutoscaler": null
}
}
Now letβs get some creds
$ az aks get-credentials -n idjtestaks1 -g idjtestrg --admin
A different object named idjtestaks1 already exists in your kubeconfig file.
Overwrite? (y/n): y
A different object named clusterAdmin_idjtestrg_idjtestaks1 already exists in your kubeconfig file.
Overwrite? (y/n): y
Merged "idjtestaks1-admin" as current context in /home/builder/.kube/config
builder@DESKTOP-QADGF36:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-16151709-vmss000000 Ready agent 40m v1.28.5
aks-nodepool1-16151709-vmss000001 Ready agent 41m v1.28.5
I tried twice with some time between. Both times ended in failure
and then again
About Zarf
From Zarfβs docs they say it βis a free and open-source tool that enables declarative creation & distribution of software into air-gapped/constrained/standalone environmentsβ.
In the FAQ, they point out it was created by Defense Unicorns (hence my article graphic) which is a defense tech provider for government secure technology stacks. They are based in Colorado Springs, CO and have between 100 and 250 employees, according to Crunchbase.
There is no need to write a full company bio on Defense Unicorns, but I encourage you to listen to their podcast which talks about how they came into being.
They smartly made Zarf under Apache 2.0 licensing which keeps it fully Open-Source.
Summary
In this writeup we setup Zarf on a local K3s as well as AKS (though not successfully). We tried deploying a sample DOS Games pack (and got sucked into Warcraft 2 for a bit, or at least I certainly did). Then we pivoted to packaging up the manifest from our recent Weather Apps post.
I fully intend to explore Zarf more as it really fills a unique niche. In fact, one can explore all the current Defense Unicorns open-source projects on Github. The one that caught my eye is Pepr which lets one create mutating webhooks using typescript. Iβm thinking I could accomplish in K8s with Pepr what I would with CloudCustodian in a Cloud Environment (e.g. auto labelling/tagging).