Getting Started: Skaffold, a k8s build framework

Skaffold is a recent project from Google recently covered in an article by CBR. It is meant to be a simple pipeline tool for building container images. It dates back to January 3 2018 by Matt Rickard who used to work at Google on their containers team. Skaffold was later announced on google’s container blog March 2018. Google has since open-sourced it (and now has over 5200 commits by over 150 developers and just GA’ed it a week ago (Nov 7 2019).

It declares itself a “command line tool that facilitates continuous development for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters”. In short, a fast and light k8s-based CI/CD tool for things like golang.

Getting Started (with Getting Started)

Get skaffold from https://skaffold.dev/

$ brew update && brew install skaffold
Updated 2 taps (homebrew/core and homebrew/bundle).
==> Updated Formulae
aws-cdk         exploitdb       jfrog-cli-go    now-cli         yadm
bullet          goreleaser      lazygit         suil
dvc             hcloud          mdbook          ungit
==> Downloading https://homebrew.bintray.com/bottles/skaffold-1.0.0.mojave.bottl
==> Downloading from https://akamai.bintray.com/d9/d9591089d76a7cb833cf3f09a87ef
######################################################################## 100.0%
==> Pouring skaffold-1.0.0.mojave.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Summary
🍺  /usr/local/Cellar/skaffold/1.0.0: 8 files, 40.5MB

Let’s start with their Hello World sample app:

git clone https://github.com/GoogleContainerTools/skaffold

We see there are just a few files skaffold needs to build and test a go lang app:

$ cd examples/getting-started/
$ ls
Dockerfile	README.adoc	k8s-pod.yaml	main.go		skaffold.yaml

It’s actually quite tight. First, the Dockerfile itself is really the build agent. It handles the copy and build of the golang app:

$ cat k8s-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: getting-started
spec:
  containers:
  - name: getting-started
    image: gcr.io/k8s-skaffold/skaffold-example

$ cat Dockerfile 
FROM golang:1.12.9-alpine3.10 as builder
COPY main.go .
RUN go build -o /app main.go

FROM alpine:3.10
CMD ["./app"]
COPY --from=builder /app .

The skaffold.yaml tells skaffold which pods should be launched:

$ cat skaffold.yaml 
apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
  - image: gcr.io/k8s-skaffold/skaffold-example
deploy:
  kubectl:
    manifests:
      - k8s-*

Such as:

$ cat k8s-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: getting-started
spec:
  containers:
  - name: getting-started
    image: gcr.io/k8s-skaffold/skaffold-example

My first build failed since it seems it needs my mac’s local docker to auth to gcloud..

$ skaffold dev
WARN[0000] port 50051 for gRPC server already in use: using 50052 instead 
WARN[0000] port 50052 for gRPC HTTP server already in use: using 50053 instead 
Listing files to watch...
 - gcr.io/k8s-skaffold/skaffold-example
Generating tags...
 - gcr.io/k8s-skaffold/skaffold-example -> gcr.io/k8s-skaffold/skaffold-example:v1.0.0-21-g4dd6b59e
Checking cache...
 - gcr.io/k8s-skaffold/skaffold-example: Not found. Building
Building [gcr.io/k8s-skaffold/skaffold-example]...
Sending build context to Docker daemon  3.072kB
Step 1/6 : FROM golang:1.12.9-alpine3.10 as builder
 ---> e0d646523991
Step 2/6 : COPY main.go .
 ---> Using cache
 ---> 04f78252d617
Step 3/6 : RUN go build -o /app main.go
 ---> Using cache
 ---> e58e2f45ea3b
Step 4/6 : FROM alpine:3.10
 ---> 965ea09ff2eb
Step 5/6 : CMD ["./app"]
 ---> Using cache
 ---> c8d90b8a4f75
Step 6/6 : COPY --from=builder /app .
 ---> Using cache
 ---> b85e9ef75d32
Successfully built b85e9ef75d32
Successfully tagged gcr.io/k8s-skaffold/skaffold-example:v1.0.0-21-g4dd6b59e
The push refers to repository [gcr.io/k8s-skaffold/skaffold-example]
7aecf8524e6c: Preparing
77cae8ab23bf: Preparing
FATA[0001] exiting dev mode because first build failed: build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-example]: build artifact: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

I had to pause and install gcloud cli: https://cloud.google.com/sdk/docs/downloads-interactive

Once done, i could add gcloud as a docker auth provider:

$ gcloud auth configure-docker
The following settings will be added to your Docker config file 
located at [/Users/Isaac/.docker/config.json]:
 {
  "credHelpers": {
    "gcr.io": "gcloud", 
    "us.gcr.io": "gcloud", 
    "eu.gcr.io": "gcloud", 
    "asia.gcr.io": "gcloud", 
    "staging-k8s.gcr.io": "gcloud", 
    "marketplace.gcr.io": "gcloud"
  }
}

That just brought me to another error…

Successfully built b85e9ef75d32
Successfully tagged gcr.io/k8s-skaffold/skaffold-example:v1.0.0-21-g4dd6b59e
The push refers to repository [gcr.io/k8s-skaffold/skaffold-example]
7aecf8524e6c: Preparing
77cae8ab23bf: Preparing
FATA[0003] exiting dev mode because first build failed: build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-example]: build artifact: denied: Token exchange failed for project 'k8s-skaffold'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

It wasn’t obvious to me at first… but the “k8s-skaffold” in the yaml files was supposed to be your project.

So if we circle back to the output from my gcloud login:

...
Pick cloud project to use: 
 [1] api-project-799042244963
 [2] careful-compass-241122
 [3] myexpressapp
 [4] pebbletaskapp
 [5] Create a new project
Please enter numeric choice or text value (must exactly match list 
item):  1

Your current project has been set to: [api-project-799042244963].
...

We see i authed into my old api-project (api-project-799042244963). So i changed the files to refer to my project:

$ cat skaffold.yaml 
apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
  - image: gcr.io/api-project-799042244963/skaffold-example
deploy:
  kubectl:
    manifests:
      - k8s-*

$ cat k8s-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: getting-started
spec:
  containers:
  - name: getting-started
    image: gcr.io/api-project-799042244963/skaffold-example

Then it worked:

$ skaffold dev
WARN[0000] port 50051 for gRPC server already in use: using 50052 instead 
WARN[0000] port 50052 for gRPC HTTP server already in use: using 50053 instead 
Listing files to watch...
 - gcr.io/api-project-799042244963/skaffold-example
Generating tags...
 - gcr.io/api-project-799042244963/skaffold-example -> gcr.io/api-project-799042244963/skaffold-example:v1.0.0-21-g4dd6b59e-dirty
Checking cache...
 - gcr.io/api-project-799042244963/skaffold-example: Not found. Building
Building [gcr.io/api-project-799042244963/skaffold-example]...
Sending build context to Docker daemon  3.072kB
Step 1/6 : FROM golang:1.12.9-alpine3.10 as builder
 ---> e0d646523991
Step 2/6 : COPY main.go .
 ---> Using cache
 ---> 04f78252d617
Step 3/6 : RUN go build -o /app main.go
 ---> Using cache
 ---> e58e2f45ea3b
Step 4/6 : FROM alpine:3.10
 ---> 965ea09ff2eb
Step 5/6 : CMD ["./app"]
 ---> Using cache
 ---> c8d90b8a4f75
Step 6/6 : COPY --from=builder /app .
 ---> Using cache
 ---> b85e9ef75d32
Successfully built b85e9ef75d32
Successfully tagged gcr.io/api-project-799042244963/skaffold-example:v1.0.0-21-g4dd6b59e-dirty
The push refers to repository [gcr.io/api-project-799042244963/skaffold-example]
7aecf8524e6c: Preparing
77cae8ab23bf: Preparing
77cae8ab23bf: Layer already exists
7aecf8524e6c: Pushed
v1.0.0-21-g4dd6b59e-dirty: digest: sha256:055534d214782d521595db84a2ef80bbb18f58adbf20388b3df6248611d0cbd6 size: 739
Tags used in deployment:
 - gcr.io/api-project-799042244963/skaffold-example -> gcr.io/api-project-799042244963/skaffold-example:v1.0.0-21-g4dd6b59e-dirty@sha256:055534d214782d521595db84a2ef80bbb18f58adbf20388b3df6248611d0cbd6
Starting deploy...
 - pod/getting-started created
Watching for changes...

#And ctrl-c to delete:
^CCleaning up...
 - pod "getting-started" deleted

However, you will see the container is left in your cloud storage (so you will want to delete it so as not to incur charges):

Note: leaving it a day showed just 1 cent so it wont be a large cost:

my charges for the Getting Started container for 1 day

Using Azure

One nice thing is it assumes kubectl commands and will use whatever registry you set. So I changed to a current AKS and ACR and had only to log into ACR:

JOHNSI10-M1:getting-started johnsi10$ cat k8s-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: getting-started
spec:
  containers:
  - name: getting-started
    image: usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example

JOHNSI10-M1:getting-started johnsi10$ cat skaffold.yaml 
apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
  - image: usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example
deploy:
  kubectl:
    manifests:
      - k8s-*

JOHNSI10-M1:getting-started johnsi10$ az acr login --name usxxxxxxxxcr
Login Succeeded

We can see (when i was running it), that the container is there in the default namespace:

JOHNSI10-M1:terraform johnsi10$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
default       getting-started                        1/1     Running   0          19s
kube-system   coredns-7fc597cc45-crvll               1/1     Running   0          36m
kube-system   coredns-7fc597cc45-hr4hc               1/1     Running   0          42m

running:

JOHNSI10-M1:getting-started johnsi10$ skaffold dev
WARN[0000] port 50051 for gRPC server already in use: using 50052 instead 
WARN[0000] port 50052 for gRPC HTTP server already in use: using 50053 instead 
Listing files to watch...
 - usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example
Generating tags...
 - usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example -> usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example:v1.0.0-21-g4dd6b59e-dirty
Checking cache...
 - usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example: Found. Pushing
The push refers to repository [usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example]
7aecf8524e6c: Preparing
77cae8ab23bf: Preparing
7aecf8524e6c: Pushed
77cae8ab23bf: Pushed
v1.0.0-21-g4dd6b59e-dirty: digest: sha256:055534d214782d521595db84a2ef80bbb18f58adbf20388b3df6248611d0cbd6 size: 739
Tags used in deployment:
 - usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example -> usxxxxxxxxcr.azurecr.io/skaffold-project/skaffold-example:v1.0.0-21-g4dd6b59e-dirty@sha256:055534d214782d521595db84a2ef80bbb18f58adbf20388b3df6248611d0cbd6
Starting deploy...
 - pod/getting-started created
Watching for changes...
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
...

the image as stored in the ACR above

Summary

Skaffold holds a lot of promise for quick kubernetes development based system. As it uses native kubectl and docker push/pull it is easy to migrate to any container registry or kubernetes cluster of one’s choosing.

I would like to see how well this might work with k3s and pi3 based clusters. It will be worth following up with a more deep dive as it shows a lot of promise.