Waypoint: Part Deux

Published: Mar 22, 2022 by Isaac Johnson

The last time we looked at Hashi Waypoint it had just been released. While interesting, it was somewhat strange to see a new CI/CD tool come to the game, especially since Hashi tends to put out systems that integrate with CI/CD tooling.

It has been a while since release and I figured now was a good time to revisit Waypoint. Has it matured? In comparing tools like Jenkins, Azure Devops and Argo Workflows, where does it fit (if at all)?

Today we’ll cover setting up waypoint locally as well as on Kubernetes (both K3s and AKS). We’ll setup ingress, projects and experiment with a NodeJS Kubernetes based project. Lastly, we’ll tie it together by trying to build remotely and integrating with Github Actions for a full continuous integration and deployment setup.

In our examples below, we’ll tie to a local private Harbor container registry so we can explore the use of registry secrets and private registry authentication credentials.

Setup

The last time we tried Waypoint it was back in October of 2020 and the recommend approach was to use apt to install. Today they have a helm chart.

I’ll be installing this into a cluster that is wrapping up an update to the latest k3s (1.23.x):

$ kubectl get nodes
NAME                          STATUS   ROLES                  AGE    VERSION
isaac-macbookair              Ready    control-plane,master   422d   v1.23.3+k3s1
anna-macbookair               Ready    <none>                 309d   v1.23.3+k3s1
builder-hp-elitebook-850-g2   Ready    <none>                 112d   v1.21.5+k3s1
builder-macbookpro2           Ready    <none>                 49d    v1.21.5+k3s1
builder-hp-elitebook-850-g1   Ready    <none>                 117d   v1.23.3+k3s1
hp-hp-elitebook-850-g2        Ready    <none>                 137d   v1.23.3+k3s1
isaac-macbookpro              Ready    <none>                 421d   v1.23.3+k3s1

Helm

First, we need to add the helm repository

helm repo add hashicorp https://helm.releases.hashicorp.com

Then we can install from the chart

$ helm install waypoint hashicorp/waypoint
NAME: waypoint
LAST DEPLOYED: Mon Feb 21 17:51:15 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Waypoint!

Waypoint will take a few minutes to bootstrap. Once it is bootstrapped, you
can log in to the local CLI by running the following command. If this command
fails, please wait a few more minutes and try again.

  $ waypoint login -from-kubernetes

Once logged in, you can use the "waypoint ui" command to open the web interface.
Alternately, you can visit the address of the load balancer service configured.

If anything fails to configure properly you can uninstall this release:

  $ helm uninstall waypoint

By default, it wants to setup the service as a loadbalancer which rather vomits on my cluster. I moved it to ClusterIP and it solved the Init issues.

$ helm install waypoint --set ui.service.type=ClusterIP hashicorp/waypoint

Once the pods come up, we can fetch the login token

$ kubectl get secret waypoint-server-token -o jsonpath="{.data.token}" &
& echo
SGV5LCBob3cgaXMgaXQgZ29pbmcuICBTdXJlIHdpbGwgYmUgbmljZSB3aGVuIHdpbnRlciBpcyBkb25lLiBMb29raW5nIGZvcndhcmQgdG8gZ2V0dGluZyBvdXQgYW5kIGRvaW5nIHNvbWUgY2FtcGluZy4K

$ kubectl port-forward svc/waypoint-ui 8080:443
Forwarding from 127.0.0.1:8080 -> 9702
Forwarding from [::1]:8080 -> 9702
Handling connection for 8080

/content/images/2022/02/waypoint-01.png

which loaded this rather blank page

/content/images/2022/02/waypoint-02.png

clicking detail didn’t work too hot

/content/images/2022/02/waypoint-03.png

(you can get this error if you use the wrong token)

I’ll setup Ingress while I’m at it

$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-waypoint.json
{
    "ChangeInfo": {
        "Id": "/change/C08858852NHKMJ93HSTJI",
        "Status": "PENDING",
        "SubmittedAt": "2022-02-21T23:55:15.688Z",
        "Comment": "CREATE waypoint fb.s A record "
    }
}

$ cat ingress-waypoint.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    ingress.kubernetes.io/proxy-body-size: 2048m
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required - waypoint
    nginx.ingress.kubernetes.io/auth-secret: argocd-basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/proxy-body-size: 2048m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.org/client-max-body-size: 2048m
  creationTimestamp: "2022-02-07T12:14:58Z"
  generation: 1
  labels:
    app: waypoint
    release: waypointrelease
  name: waypoint-with-auth
  namespace: default
  resourceVersion: "200201296"
  uid: f4196f14-9ace-4bcf-9a2f-60a8dac25bb3
spec:
  ingressClassName: nginx
  rules:$ cat ingress-waypoint.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    ingress.kubernetes.io/proxy-body-size: 2048m
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required - waypoint
    nginx.ingress.kubernetes.io/auth-secret: argocd-basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/proxy-body-size: 2048m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.org/client-max-body-size: 2048m
  creationTimestamp: "2022-02-07T12:14:58Z"
  generation: 1
  labels:
    app: waypoint
    release: waypointrelease
  name: waypoint-with-auth
  namespace: default
  resourceVersion: "200201296"
  uid: f4196f14-9ace-4bcf-9a2f-60a8dac25bb3
spec:
  ingressClassName: nginx
  rules:
  - host: waypoint.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: waypointrelease
            port:
              number: 8080
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - waypoint.freshbrewed.science

$ kubectl apply -f ingress-waypoint.yml
ingress.networking.k8s.io/waypoint-with-auth created

Local Install

If we wanted to install outside of Kubernetes, we could use instructions such as these:

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install waypoint

We can use these steps to setup the local waypoint binary.

We can also use it to run a local dockerized server:

waypoint install -platform=docker -accept-tos -- -vvv -disable-ui

AKS

Let’s use Azure Kubernetes Service next to show a rather clean full K8s deployment.

Create the Resource Group

$ az group create -n idjaks45 --location centralus
{
  "id": "/subscriptions/8defc61d-657a-453d-a6ff-cb9f91289a61/resourceGroups/idjaks45",
  "location": "centralus",
  "managedBy": null,
  "name": "idjaks45",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

Create (or use) a Service Principal. Unlike the past, it now seems that create-for-rbac really demands a proper AD compliant name. You can use the default from AAD on your tenant; e.g.

/content/images/2022/03/waypoint-01.png

$ az ad sp create-for-rbac -n idjaks45sp.princessking.onmicrosoft.com --skip-assignment --output json > my_sp.json
Changing "idjaks45sp.princessking.onmicrosoft.com" to a valid URI of "http://idjaks45sp.princessking.onmicrosoft.com", which is the required format used for service principal names

$ export SP_PASS=`cat my_sp.json | jq -r .password`
$ export SP_ID=`cat my_sp.json | jq -r .appId`

Then create the AKS cluster:

$ az aks create --resource-group idjaks45 --name idjaks45 --location centralus --node-count 2 --enable-cluster-autoscaler --min-count 2 --max-count 3 --generate-ssh-keys --network-plugin azure --network-policy azure --service-principal $SP_ID --client-secret $SP_PASS
 - Running ..

 $ az aks list -o table
Name      Location    ResourceGroup    KubernetesVersion    ProvisioningState    Fqdn
--------  ----------  ---------------  -------------------  -------------------  ---------------------------------------------------------
idjaks45  centralus   idjaks45         1.21.9               Succeeded            idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io

Now get credentials and verify nodes are running

$ az aks get-credentials -n idjaks45 -g idjaks45 --admin
Merged "idjaks45-admin" as current context in /home/builder/.kube/config
builder@DESKTOP-72D2D9T:~/Workspaces/jekyll-blog$ kubectl get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-70322854-vmss000000   Ready    agent   71s   v1.21.9
aks-nodepool1-70322854-vmss000001   Ready    agent   83s   v1.21.9

We can now install Waypoint via Helm

$ !1372
helm install waypoint hashicorp/waypoint
NAME: waypoint
LAST DEPLOYED: Mon Feb 21 19:47:09 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Waypoint!

Waypoint will take a few minutes to bootstrap. Once it is bootstrapped, you
can log in to the local CLI by running the following command. If this command
fails, please wait a few more minutes and try again.

  $ waypoint login -from-kubernetes

Once logged in, you can use the "waypoint ui" command to open the web interface.
Alternately, you can visit the address of the load balancer service configured.

If anything fails to configure properly you can uninstall this release:

  $ helm uninstall waypoint

We’ll check the pods and service to see if it’s up

$ kubectl get pods
NAME                               READY   STATUS              RESTARTS   AGE
waypoint-bootstrap-w4l6j           1/1     Running             0          45s
waypoint-runner-59dbddf4f4-d2jxn   0/1     Init:0/1            0          45s
waypoint-server-0                  0/1     ContainerCreating   0          45s

$ kubectl get pods
NAME                               READY   STATUS     RESTARTS   AGE
waypoint-bootstrap-w4l6j           1/1     Running    0          106s
waypoint-runner-59dbddf4f4-d2jxn   0/1     Init:0/1   0          106s
waypoint-server-0                  1/1     Running    0          106s

$ kubectl get svc
NAME              TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)
 AGE
kubernetes        ClusterIP      10.0.0.1     <none>          443/TCP
 8m30s
waypoint-server   ClusterIP      None         <none>          9702/TCP,9701/TCP
 4m43s
waypoint-ui       LoadBalancer   10.0.87.68   13.89.111.244   80:32259/TCP,443:31145/TCP,9701:30061/TCP,9702:30778/TCP   4m43s

Next, we need a login token to get started

$ !1421
kubectl get secret waypoint-server-token -o jsonpath="{.data.token}" && echo
M0s0d1FVZEgxZGZIYXlwM3g1UFFLSGtoYUw2S3BhVDhFRFRSdnM5Y1BTWm1lUndwOTE1NldTRU16dWd3QVNBb2lwSkdlTVkyY0Q4bnp0TFNEQmRYRlc0dTN4Y2FHbjNVcVZHeDVSUFQ1N3JtNjFvSEVpTFNBYmk5NXYxbjdLYlZBRjJ6TUNYY0FLdUw5cXhD

With the Token we should be able to login, however I see some errors.

/content/images/2022/02/waypoint-04.png

Moving back to the demo code, Let’s try to waypoint up now…

/content/images/2022/02/waypoint-05.png

Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
 │ [exporter] Adding layer 'process-types'
 │ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
 │ [exporter] Adding label 'io.buildpacks.build.metadata'
 │ [exporter] Adding label 'io.buildpacks.project.metadata'
 │ [exporter] Setting default process type 'web'
 │ [exporter] Saving example-nodejs...
 │ [exporter] *** Images (ea4b1f8a4f8f):
 │ [exporter]       example-nodejs
 │ [exporter] Adding cache layer 'heroku/nodejs-engine:nodejs'
 │ [exporter] Adding cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image

Generated new Docker image: example-nodejs:latest

» Deploying example-nodejs...
✓ Setting up network...
✓ Starting container
✓ App deployed as container: example-nodejs-01FWFJS3NVZMGF8MTAX6KCMRZM
✓ Docker image "example-nodejs:latest" up to date!

✓ Finished building report for Docker platform
✓ Finished building report for Docker network resource
✓ Finished building report for Docker container resource
✓ All 2 resources are reporting READY

» Releasing example-nodejs...
No release phase specified, skipping...

The deploy was successful!

The release did not provide a URL and the URL service is disabled on the
server, so no further URL information can be automatically provided. If
this is unexpected, please ensure the Waypoint server has both the URL service
enabled and advertise addresses set.

We check to see if the image was created (locally):

$ docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS                     NAMES
773e8c61480f   example-nodejs:latest   "/waypoint-entrypoin…"   29 seconds ago   Up 27 seconds   0.0.0.0:54723->3000/tcp   example-nodejs-01FWFJS3NVZMGF8MTAX6KCMRZM

This rather annoyed me. I really don’t want to build from my local box.

The point, in my opinion, of a CI system is to move the compilation from the local workstation to a remote service. I don’t want to depend on a local user’s ability to pull and create images.

My first step is to try and use the waypoint server itself. Perhaps it has Docker-in-Docker abilities? That could work.

We’ll exec into the pod, clone the repo and try…

$ kubectl exec -it waypoint-server-0 -- /bin/sh
/ $ git clone https://github.com/hashicorp/waypoint-examples.git
fatal: could not create work tree dir 'waypoint-examples': Permission denied
/ $ cd /tmp
/tmp $ git clone https://github.com/hashicorp/waypoint-examples.git
Cloning into 'waypoint-examples'...
remote: Enumerating objects: 1771, done.
remote: Counting objects: 100% (601/601), done.
remote: Compressing objects: 100% (348/348), done.
remote: Total 1771 (delta 353), reused 444 (delta 233), pack-reused 1170
Receiving objects: 100% (1771/1771), 1.33 MiB | 5.90 MiB/s, done.
Resolving deltas: 100% (791/791), done.
/tmp $ cd waypoint-examples/docker/
/tmp/waypoint-examples/docker $ cd nodejs/
/tmp/waypoint-examples/docker/nodejs $ waypoint init
 + Configuration file appears valid
 + Local mode initialized successfully
 + Project "example-nodejs" and all apps are registered with the server.

Project initialized!

You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
/tmp/waypoint-examples/docker/nodejs $ waypoint up
  Performing operation locally


» Building example-nodejs...
WARNING: Docker daemon appears unavailable. The 'pack' builder requires access to a Docker daemon. Pack does not support
WARNING: Docker daemon appears unavailable. The 'pack' builder requires access to a Docker daemon. Pack does not support
dockerless builds. We will still attempt to run the build but it will likely fail. If you are running this build
locally, please install Docker. If you are running this build remotely (in a Waypoint runner), the runner must be
configured to have access to the Docker daemon.

! Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
  docker daemon running?

That didn’t work, no DinD.

Doing some more reading, perhaps I need to create a “context” for waypoint to use. Here I’ll use the external service IP for the waypoint-ui as exposed by AKS (13.89.111.244)

builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context create -server-addr=13.89.111.244:9701 -server-auth-token=`kubectl get secret waypoint-server-token -o jsonpath="{.data.token}"` -server-tls -set-default my-server
Context "my-server" created.
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context verify
❌ Connecting with context "my-server"...
! Error connecting with context "my-server": context deadline exceeded
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context create -server-addr=13.89.111.244:9701 -server-auth-token=`kubectl get secret waypoint-server-token -o jsonpath="{.data.token}"` -set-default my-server
Context "my-server" created.
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context verify
❌ Connecting with context "my-server"...
! Error connecting with context "my-server": context deadline exceeded

Let me try to uninstall and try again

$ helm list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART          APP VERSION
waypoint        default         1               2022-02-21 19:47:09.3355676 -0600 CST   deployed        waypoint-0.1.4 0.7.1

$ helm delete waypoint
release "waypoint" uninstalled

Trying to install the first time found some abandoned services (likely from the failed LB install)

$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
❌ Creating Kubernetes resources...
! Error creating service services "waypoint" already exists
! Error installing server into kubernetes: services "waypoint" already exists

$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
❌ Creating Kubernetes resources...
! Error creating service services "waypoint" already exists
! Error installing server into kubernetes: services "waypoint" already exists

$ kubectl get svc
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                         AGE
kubernetes   ClusterIP      10.0.0.1       <none>           443/TCP                         45m
waypoint     LoadBalancer   10.0.135.226   52.154.171.243   9701:32177/TCP,9702:31556/TCP   40s

$ helm uninstall waypoint
Error: uninstall: Release not loaded: waypoint: release: not found

Here I just manually deleted the service

$ kubectl delete svc waypoint
service "waypoint" deleted

I’ll try again by installing with the waypoint binary instead of the helm chart

$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!

The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1645496978".
Use the "waypoint context" CLI to manage CLI contexts.

The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".

To launch and authenticate into the Web UI, run:
waypoint ui -authenticate

Advertise Address: 13.89.111.244:9701
Web UI Address: https://13.89.111.244:9702

This worked much better. Here we can see, by default, the waypoint binary installs the service in Kubernetes and also set’s a context for us.

$ waypoint context list
    |        NAME        |  PLATFORM  |   SERVER ADDRESS
----+--------------------+------------+---------------------
  * | install-1645496978 | kubernetes | 13.89.111.244:9701
    | my-server          | n/a        | 13.89.111.244:9701

We can verify the context

$ waypoint context verify
✓ Context "install-1645496978" connected successfully.

Now, let’s go back to our demo code and try the init and build.

builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$  waypoint init
✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "example-nodejs" and all apps are registered with the server.

Project initialized!

You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.

Test build

builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint build

» Building example-nodejs...
  Performing operation locally
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
⠇ Building image
 │ 2022/02/21 20:37:05.319337 DEBUG:  Pulling image index.docker.io/heroku/buildpac
 │ ks:20

However, as we can see, it is building locally again…

Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: gcr.io/buildpacks/builder:v1
✓ Creating pack client
✓ Building image
 │ [exporter] Adding layer 'config'
 │ [exporter] Adding layer 'process-types'
 │ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
 │ [exporter] Adding label 'io.buildpacks.build.metadata'
 │ [exporter] Adding label 'io.buildpacks.project.metadata'
 │ [exporter] Setting default process type 'web'
 │ [exporter] Saving helloworld-go...
 │ [exporter] *** Images (b4e6560f1555):
 │ [exporter]       helloworld-go
 │ [exporter] Adding cache layer 'google.go.runtime:go'
✓ Injecting entrypoint binary to image

Generated new Docker image: helloworld-go:latest
✓ Tagging Docker image: helloworld-go:latest => idjohnson/helloworld-go:latest
✓ Pushing Docker image...
 │ a4d53325badd: Pushed
 │ 0b6b00fa3492: Pushed
 │ e32996f7b4b0: Pushed
 │ d48e235b496a: Pushed
 │ 1d7b24ca7af7: Pushed
 │ 84ff92691f90: Pushed
 │ 9d15ff164ac7: Pushed
 │ 158840bd8028: Pushed
 │ latest: digest: sha256:9a6dd98c1837a5181fcb699440493a8e0c78782f64a147d460c631bad
 │ 9668bc2 size: 2623
✓ Docker image pushed: idjohnson/helloworld-go:latest

» Deploying helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Creating deployment...
✓ Expected "http" port "8080" for app "helloworld-go-v1"
✓ Deployment successfully rolled out!

✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes deployment resource
✓ Deployment "helloworld-go-v1" is reporting ready!

» Releasing helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Creating service...
✓ Service is ready!

✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource

The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."

   Release URL: http://13.86.33.130:8080
Deployment URL: https://formerly-social-bobcat--v1.waypoint.run
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-to-k8s$

It does work to deploy. If we test the release URL, we can see the NodeJS app running.

/content/images/2022/02/waypoint-06.png

We can also login to the UI to see the deployment

/content/images/2022/02/waypoint-07.png

Then dig into details on the deployment

/content/images/2022/02/waypoint-08.png

We can look at release logs

/content/images/2022/02/waypoint-09.png

and the resources created

/content/images/2022/02/waypoint-10.png

We can exec into the pod to run commands if we wanted to

/content/images/2022/02/waypoint-11.png

While it was built locally, the build logs were pushed to waypoint

/content/images/2022/02/waypoint-12.png

Testing changes

Let’s push a change

$ git diff
diff --git a/main.go b/main.go
index c811ada..e8e266c 100644
--- a/main.go
+++ b/main.go
@@ -10,7 +10,7 @@ type Server struct{}
 func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        w.Header().Set("Content-Type", "application/json")
-       w.Write([]byte(`{"message": "Hello World!"}`))
+       w.Write([]byte(`{"message": "Hello World Again!"}`))
 }

 func main() {
diff --git a/waypoint.hcl b/waypoint.hcl
index 883f370..21b113c 100644
--- a/waypoint.hcl
+++ b/waypoint.hcl
@@ -7,7 +7,7 @@ app "helloworld-go" {
         }
         registry {
           use "docker" {
-            image = "ronakbanka/helloworld-go"
+            image = "idjohnson/helloworld-go"
             tag   = "latest"
             local = false
         }

We will waypoint up to build and release again

Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: gcr.io/buildpacks/builder:v1
✓ Creating pack client
✓ Building image
 │ [exporter] Reusing layer 'config'
 │ [exporter] Reusing layer 'process-types'
 │ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
 │ [exporter] Adding label 'io.buildpacks.build.metadata'
 │ [exporter] Adding label 'io.buildpacks.project.metadata'
 │ [exporter] Setting default process type 'web'
 │ [exporter] Saving helloworld-go...
 │ [exporter] *** Images (0429d4c10f67):
 │ [exporter]       helloworld-go
 │ [exporter] Adding cache layer 'google.go.runtime:go'
✓ Injecting entrypoint binary to image

Generated new Docker image: helloworld-go:latest
✓ Tagging Docker image: helloworld-go:latest => idjohnson/helloworld-go:latest
✓ Pushing Docker image...
 │ a4d53325badd: Layer already exists
 │ f1f11d09993c: Pushed
 │ ea8974eb62ff: Pushed
 │ d48e235b496a: Layer already exists
 │ 1d7b24ca7af7: Layer already exists
 │ 84ff92691f90: Layer already exists
 │ 9d15ff164ac7: Layer already exists
 │ 158840bd8028: Layer already exists
 │ latest: digest: sha256:8ac78583bb5fbf78de423795f88f1367f3c1cc258cd797dec254c9bc2
 │ f34af83 size: 2623
✓ Docker image pushed: idjohnson/helloworld-go:latest

» Deploying helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Creating deployment...
✓ Expected "http" port "8080" for app "helloworld-go-v2"
✓ Deployment successfully rolled out!

✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes deployment resource
✓ Deployment "helloworld-go-v2" is reporting ready!

» Releasing helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Updating existing service...
✓ Service is ready!

✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource

The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."

   Release URL: http://13.86.33.130:8080
Deployment URL: https://formerly-social-bobcat--v2.waypoint.run
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-to-k8s$

and upon deploy, we see it deployed (to the same URL)

/content/images/2022/02/waypoint-13.png

We can check the build logs

/content/images/2022/02/waypoint-14.png

/content/images/2022/02/waypoint-15.png

Say that we found an issue, we could move back to release 1 easily

$ waypoint release -repeat 1

» Releasing helloworld-go...
  Performing operation locally

  Performing operation locally
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource

Release URL: http://13.86.33.130:8080

However, I should not, that did not change the service http://13.86.33.130:8080/

I can see that both versions are actually running:

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
helloworld-go-v1-66cb794895-25tmf   1/1     Running   0          68m
helloworld-go-v2-5c7f767d67-s854p   1/1     Running   0          7m13s
waypoint-runner-6695b69bc7-wvv7p    1/1     Running   0          89m
waypoint-server-0                   1/1     Running   0          91m

That said, the svc endpoint points to v2


$ kubectl get svc helloworld-go -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-02-22T02:52:23Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  name: helloworld-go
  namespace: default
  resourceVersion: "27004"
  uid: b46c0c5e-2732-4e66-a196-34eedd51333b
spec:
  clusterIP: 10.0.172.204
  clusterIPs:
  - 10.0.172.204
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 31024
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    name: helloworld-go-v2
    waypoint.hashicorp.com/id: 01FWFS0BTJ8X971T2JMCXM6KB9
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 13.86.33.130
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-to-k8s$

But we can go to it from the built-in url https://formerly-social-bobcat--v1.waypoint.run/

/content/images/2022/02/waypoint-16.png

Try again locally

We’ll follow this guide again to install the binary:

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install waypoint

Remove the helm install (if installed via helm)

$ helm delete waypoint
release "waypoint" uninstalled

Now install using the waypoint binary:

$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!

The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1646140543".
Use the "waypoint context" CLI to manage CLI contexts.

The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".

To launch and authenticate into the Web UI, run:
waypoint ui -authenticate

Advertise Address: 192.168.1.12:9701
Web UI Address: https://192.168.1.12:9702

We see that Waypoint is indeed up at that address

/content/images/2022/02/waypoint-17.png

We need a token to auth. you can use waypoint ui -authenticate or just use the following to fetch it:

$ kubectl  get secret `kubectl get secret | grep waypoint | sed 's/ .*//' | tr -d '\n'` -o json
| jq -r .data.token
ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltMVZUakI0VTBOUVdtWmtNblJvTkc5UE9GQkRXRlZ3ZFdWZlVVbGZaWEUyZFdoZmVFczRlRGRYZG04aWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbmRoZVhCdmFXNTBMWEoxYm01bGNpMTBiMnRsYmkxNmNITTJaeUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG01aGJXVWlPaUozWVhsd2IybHVkQzF5ZFc1dVpYSWlMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObGNuWnBZMlV0WVdOamIzVnVkQzUxYVdRaU9pSXhaREJqTmpjNVlTMWpOakE0TFRRM1lqSXRPVFJrTkMxa1pqRTNPV05tT1dObE5EUWlMQ0p6ZFdJaU9pSnplWE4wWlcwNmMyVnlkbWxqWldGalkyOTFiblE2WkdWbVlYVnNkRHAzWVhsd2IybHVkQzF5ZFc1dVpYSWlmUS5HRVcycHBTLU5CbjE1ZWMxMXZFTTRJU3J5SWVJWmE2N1NuRzFzUklsclEtSldBTkZsYjkzekdNY2tKOE9Eckt3NGxKWUhhLXViZDg0ZkR4LXhvX3RnMlBtT0lXVG9JMFFuVENEZ2tfNi1DNFlXOFdGb0ZSbEtxNXJOVGFmUTdQVmM5enNxM1dLeGhPems2dUR4a3ZVVGxfQWVJU1FuMWhxQVliR3plbUJzTTlXYnJfcVpuSjJvYU5FSTg1MXpoMmtJbzZvQXBjaVl0d1BWZFUyanQyTFdtZURBemRBci1pWWhHblh1azRabDIzTVE0LWVPZ1N5cFJfa3FUOEZZOVhBak9vcDlvTFRRRGNGdGhFZ2NKek1STEdaY01CeHp6ZllrWE42aXBZdEZhUkNDR1FaaGk5S1NGMFJ0T1RRaDE3ZG5XSUJLa3RpNTBEQ2ZkMVJ2R2lPN0E=

/content/images/2022/02/waypoint-18.png

Because we used the CLI to install, we see the context already saved:

$ waypoint context list
    |        NAME        |  PLATFORM  |  SERVER ADDRESS
----+--------------------+------------+--------------------
  * | install-1646140543 | kubernetes | 192.168.1.12:9701

Let’s start with what we know works. We’ll use the NodeJS example from Hashi’s examples:

$ git clone https://github.com/hashicorp/waypoint-examples.git
$ cd waypoint-examples/kubernetes/nodejs

We now just need to waypoint init to set up the project

$ waypoint init
✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "example-nodejs" and all apps are registered with the server.

Project initialized!

You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
builder@DESKTOP-QADGF36:~/Workspaces/waypoint-examples/kubernetes/nodejs$

Using waypoint up it will build locally and then push. Since I’m not using Docker Desktop, I’m interested to see if it can push to a Container Registry of my choosing.

$ waypoint up
  Performing operation locally

» Building example-nodejs...
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64".
We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
 │ [exporter] Adding layer 'process-types'
 │ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
 │ [exporter] Adding label 'io.buildpacks.build.metadata'
 │ [exporter] Adding label 'io.buildpacks.project.metadata'
 │ [exporter] Setting default process type 'web'
 │ [exporter] Saving example-nodejs...
 │ [exporter] *** Images (e6a2eac76957):
 │ [exporter]       example-nodejs
 │ [exporter] Adding cache layer 'heroku/nodejs-engine:nodejs'
 │ [exporter] Adding cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image

Generated new Docker image: example-nodejs:latest
✓ Tagging Docker image: example-nodejs:latest => example-nodejs:1
✓ Docker image pushed: example-nodejs:1

» Deploying example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "example-nodejs-v1"
⠸ Waiting on deployment to become available: requested=1 running=1 ready=0
Detected pods having an issue starting - Pod unable to access Docker image: rpc error: code = Unknown desc = failed to pull and unpack image
"docker.io/library/example-nodejs:1": failed to resolve reference "docker.io/library/example-nodejs:1": pull access denied, repository does not
exist or may require authorization: server message: insufficient_scope: authorization failed

That is fair, I can’t push to the global example-nodejs (docker.io/library/example-nodejs).

Using local private container registry

We need to create a Registry credential:

$ cat /home/builder/harborAuth.json
{
        "username": "isaac",
        "password": "MYPASSWORD",
        "email": "isaac.johnson@gmail.com"
}

Now we just change the waypoint.hcl to use the path (and set the full path to the image in the CR)

$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index e2d979e..6920de2 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -10,9 +10,9 @@ app "example-nodejs" {
     use "pack" {}
     registry {
       use "docker" {
-        image = "example-nodejs"
+        image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
         tag   = "1"
-        local = true
+        encoded_auth = filebase64("/home/builder/harborAuth.json")
       }
     }
   }

This time we got further, but failed to pull the built image

$ waypoint up
  Performing operation locally

» Building example-nodejs...
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64".
We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
 │ [exporter] Reusing layer 'process-types'
 │ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
 │ [exporter] Adding label 'io.buildpacks.build.metadata'
 │ [exporter] Adding label 'io.buildpacks.project.metadata'
 │ [exporter] Setting default process type 'web'
 │ [exporter] Saving example-nodejs...
 │ [exporter] *** Images (7c9b873f381b):
 │ [exporter]       example-nodejs
 │ [exporter] Reusing cache layer 'heroku/nodejs-engine:nodejs'
 │ [exporter] Reusing cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image

Generated new Docker image: example-nodejs:latest
✓ Tagging Docker image: example-nodejs:latest => harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
✓ Pushing Docker image...
 │ ab2b7fb6b8c0: Pushed
 │ a2e78a35a946: Pushed
 │ 6d3d39532668: Pushed
 │ 438aa9470562: Pushed
 │ 0724d7609fc4: Pushed
 │ fd10542322e5: Pushed
 │ ed348f24d8dd: Pushed
 │ 36ffdceb4c77: Pushed
 │ 1: digest: sha256:7b9409ab5a29bea3fa3599afc9a9a697b63f530cefbf73593147d537f6f414
 │ 77 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1

» Deploying example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "example-nodejs-v2"
⠧ Waiting on deployment to become available: requested=1 running=1 ready=0
Detected pods having an issue starting - Pod unable to access Docker image: rpc error: code = Unknown desc = failed to pull and unpack image
"harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1": failed to resolve reference
"harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1": pulling from host harbor.freshbrewed.science failed with status code
[manifests 1]: 401 Unauthorized

Adding image pull secret to waypoint HCL for private registries

We need to make note of the imagepull secret we have to pull the private container in Kubernetes. Add an “image_secret” to the deploy Kubernetes block as such:

$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index e2d979e..43c6c3b 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -10,15 +10,16 @@ app "example-nodejs" {
     use "pack" {}
     registry {
       use "docker" {
-        image = "example-nodejs"
+        image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
         tag   = "1"
-        local = true
+        encoded_auth = filebase64("/home/builder/harborAuth.json")
       }
     }
   }

   deploy {
     use "kubernetes" {
+      image_secret = "myharborreg"
       probe_path = "/"
     }
   }

Validation

Now this works:

$ waypoint up
  Performing operation locally

» Building example-nodejs...
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64".
We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
 │ [exporter] Reusing layer 'process-types'[exporter] Adding label 'io.buildpacks.lifecycle.metadata'[exporter] Adding label 'io.buildpacks.build.metadata'[exporter] Adding label 'io.buildpacks.project.metadata'[exporter] Setting default process type 'web'[exporter] Saving example-nodejs...
 │ [exporter] *** Images (fabfa63a64fb):
 │ [exporter]       example-nodejs
 │ [exporter] Reusing cache layer 'heroku/nodejs-engine:nodejs'[exporter] Reusing cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image

Generated new Docker image: example-nodejs:latest
✓ Tagging Docker image: example-nodejs:latest => harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
✓ Pushing Docker image...
 │ ab2b7fb6b8c0: Layer already exists
 │ c04687436ccf: Pushed
 │ 6d3d39532668: Layer already exists
 │ 438aa9470562: Layer already exists
 │ 0724d7609fc4: Layer already exists
 │ fd10542322e5: Layer already exists
 │ ed348f24d8dd: Layer already exists
 │ 36ffdceb4c77: Layer already exists
 │ 1: digest: sha256:58a9b3a5eec2a8c4bf15a2879638838211c41ce24cdb2401f5d1bfe8a3ee0a
 │ 77 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1

» Deploying example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "example-nodejs-v3"
✓ Deployment successfully rolled out!

✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes deployment resource
✓ Deployment "example-nodejs-v3" is reporting ready!

» Releasing example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating service...
✓ Service is ready!

» Pruning old deployments...
  Deployment: 01FX2V06WWVJT218A09ZGRS1K3 (v1)

✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource

The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."

   Release URL: http://192.168.1.214:3000
Deployment URL: https://blatantly-smiling-goldfish--v3.waypoint.run

/content/images/2022/02/waypoint-19.png

The other URL works just as well (with valid TLS)

/content/images/2022/02/waypoint-20.png

I now see the app in projects (I did end up needing to relogin with waypoint ui -authenticate)

/content/images/2022/02/waypoint-21.png

And from there I can get build details

/content/images/2022/02/waypoint-22.png

Let’s make a small change to the webpage we could see

$ git diff views/pages/index.ejs
diff --git a/kubernetes/nodejs/views/pages/index.ejs b/kubernetes/nodejs/views/pages/index.ejs
index f6c45be..65f56ff 100644
--- a/kubernetes/nodejs/views/pages/index.ejs
+++ b/kubernetes/nodejs/views/pages/index.ejs
@@ -17,7 +17,8 @@
         </div>
         <h1>This Node.js app was deployed with Waypoint.</h1>
         <p>
-          Try making a change to this text locally and run <code>waypoint up</code> again to see it.
+          Try making a change to this text locally and run <code>waypoint up</code> again to see it.
+         Hey, I made a change. wow.
         </p>
         <p>
           Read the <a href="https://waypointproject.io/docs">documentation</a> for more about Waypoint.

Also, I would really like to do this on a remote runner.

I need to see CI to believe it.

In order to really use a hosted node, we must define the GIT repo from which to pull. Otherwise, how would Waypoint know where to get sources (once it’s invoked) to docker build?

To do this, I first forked their examples repo: https://github.com/idjohnson/waypoint-examples.

/content/images/2022/02/waypoint-23.png

Then I cloned and changed the waypoint.hcl file. The key here is to set the “runner” block. By default, “runner” is disabled. That is to say, by default, Waypoint assumes you want to build locally.

We can easily fix both these things by defining the runner block and the data_source:

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git diff
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index e2d979e..21437d7 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -1,5 +1,14 @@
 project = "example-nodejs"

+runner {
+  enabled = true
+
+  data_source "git" {
+     url = "https://github.com/idjohnson/waypoint-examples.git"
+     path = "kubernetes/nodejs"
+  }
+}
+
 app "example-nodejs" {
   labels = {
     "service" = "example-nodejs",
@@ -10,15 +19,16 @@ app "example-nodejs" {
     use "pack" {}
     registry {
       use "docker" {
-        image = "example-nodejs"
+        image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
         tag   = "1"
-        local = true
+        encoded_auth = filebase64("/home/builder/harborAuth.json")
       }
     }
   }

   deploy {
     use "kubernetes" {
+      image_secret = "myharborreg"
       probe_path = "/"
     }
   }

Handling Auth Secrets for Private Container Registries in Waypoint utilizing Remote build agents

okay, that was a mouthful…

Now, clearly, I am not going to check in my password unencrypted to push to my CR.

So we need to create a runner secret we could use.

$ cat /home/builder/harborAuth.json | base64 -w 0
bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=
$ waypoint config set -runner HARBORCRED=bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=

And we can verify it took by asking Waypoint for our configs:

$ waypoint config get -runner
   SCOPE  |    NAME    |                                                          VALUE
| WORKSPACE | LABELS
----------+------------+------------------------------------------------------------------------------------------------------------------------
  project | HARBORCRED | bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo= |           |

We can use that cred by referring to it in the waypoint.hcl file. If you use AzDO, the variable syntax should be familiar:

$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index 21437d7..2ad0cae 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -21,7 +21,7 @@ app "example-nodejs" {
       use "docker" {
         image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
         tag   = "1"
-        encoded_auth = filebase64("/home/builder/harborAuth.json")
+        encoded_auth = ${HARBORCRED}
       }
     }
   }

I ended up needing to change the project name (since Waypoint had a different configuration and was not hearing a new datasource):

$ cat waypoint.hcl
project = "example-nodejs2"

runner {
  enabled = true

  data_source "git" {
     url = "https://github.com/idjohnson/waypoint-examples.git"
     path = "kubernetes/nodejs"
  }
}

app "example-nodejs2" {

  labels = {
    "service" = "example-nodejs",
    "env"     = "dev"
  }

  build {
    use "pack" {}
    registry {
      use "docker" {
        image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
        tag   = "1"
        encoded_auth = "${HARBORCRED}"
      }
    }
  }

  deploy {
    use "kubernetes" {
      image_secret = "myharborreg"
      probe_path = "/"
    }
  }

  release {
    use "kubernetes" {
      // Sets up a load balancer to access released application
      load_balancer = true
      port          = 3000
    }
  }
}

Now we can say waypoint up, and be explicit about not running locally with the local false flag.

However, I got errors from the remote server

$ waypoint up -local=false
! error reading from server: EOF

In fact, any attempt to build fell down

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=false

» Building example-nodejs2...
! error reading from server: EOF
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build
! failed to create client: context deadline exceeded
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build
! failed to create client: context deadline exceeded

Checking the pods, I can see it crashing the waypoint server:

$ kubectl get pods | tail -n3
waypoint-runner-84984bdcd5-mcgkj                         1/1     Running             3 (5m5s ago)     62m
example-nodejs-v1-6d89c89657-24n5t                       0/1     ImagePullBackOff    0                45m
waypoint-server-0                                        1/1     Running             4 (75s ago)      63m

Debugging….

I invoked the build command then checked the server pod before it crashed:

$ waypoint build -local=false

» Building example-nodejs2...
! error reading from server: EOF
$ kubectl logs waypoint-server-0
2022-03-01T14:18:56.063Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-03-01T14:18:56.063Z [DEBUG] waypoint: home configuration directory: path=/data/.config/waypoint
2022-03-01T14:18:56.063Z [INFO]  waypoint.server: opening DB: path=/data/data.db
2022-03-01T14:18:56.064Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-03-01T14:18:56.064Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-03-01T14:18:56.343Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-03-01T14:18:56.343Z [DEBUG] waypoint.server.singleprocess.url_service: using saved URL guest token
2022-03-01T14:18:56.597Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-03-01T14:18:56.597Z [INFO]  waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-03-01T14:18:56.597Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-03-01T14:18:56.597Z [INFO]  waypoint.server.singleprocess.poll_queuer.project: starting
2022-03-01T14:18:56.597Z [INFO]  waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-03-01T14:18:56.597Z [INFO]  waypoint.server.singleprocess.prune: starting
2022-03-01T14:18:56.597Z [INFO]  waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-01T14:18:56.631Z [INFO]  waypoint.server.grpc: listener is wrapped with TLS
2022-03-01T14:18:56.631Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-03-01T14:18:56.631Z [INFO]  waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-01T14:18:56.674Z [INFO]  waypoint.server.http: listener is wrapped with TLS

» Server configuration:

             DB Path: /data/data.db
        gRPC Address: [::]:9701
        HTTP Address: [::]:9702
       Auth Required: yes
  Browser UI Enabled: yes
         URL Service: api.waypoint.run:443 (account: guest)


» Server logs:

2022-03-01T14:18:56.675Z [INFO]  waypoint.server: starting built-in server: addr=[::]:9701
2022-03-01T14:18:56.675Z [INFO]  waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
2022-03-01T14:18:56.675Z [INFO]  waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022/03/01 14:19:01 http: TLS handshake error from 10.42.2.0:25941: remote error: tls: unknown certificate
2022-03-01T14:19:20.204Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:19:20.204Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:19:20.431Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:19:20.431Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:19:21.819Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-01T14:19:21.820Z [ERROR] waypoint.server.grpc: unknown runner connected: runner_id=01FX2XAW8C9ENQ3PB4ARM5JBGQ id=01FX2XAW8C9ENQ3PB4ARM5JBGQ
2022-03-01T14:19:21.820Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream response: error="rpc error: code = NotFound desc = runner ID not found: 01FX2XAW8C9ENQ3PB4ARM5JBGQ" duration=1.101831ms
2022-03-01T14:20:16.640Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:20:16.640Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=32.142µs
2022-03-01T14:20:16.669Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerConfig request
2022-03-01T14:20:17.170Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-01T14:20:51.154Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:20:51.154Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=42.552µs
2022-03-01T14:20:51.157Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject request
2022-03-01T14:20:51.158Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject response: error=<nil> duration=158.219µs
2022-03-01T14:20:51.386Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/QueueJob request
2022-03-01T14:20:51.386Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2
2022-03-01T14:20:51.386Z [INFO]  waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FX2XNWSTD0G31VXWS3MHGTXZ
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]

goroutine 151 [running]:
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0xc0000d6b40, 0x0)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0xc0000d6b40)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0xc0006c8040)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0x8000101)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc0004af680}, {0x2fad5e0, 0xc0006c8040})
        /tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b
github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000b03c20}, {0x2fad5e0, 0xc0006c8040}, 0x40, 0xc000962318)
        /tmp/wp-src/internal/server/auth.go:80 +0x204
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000b03c20}, {0x2fad5e0, 0xc0006c8040})
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000b03ad0}, {0x2fad5e0, 0xc0006c8040}, 0xc0013f2060, 0xc0006c8340)
        /tmp/wp-src/internal/server/grpc_version.go:35 +0x8e
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000b03ad0}, {0x2fad5e0, 0xc0006c8040})
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc000b032f0}, {0x2fad5e0, 0xc0006c8040}, 0xc0013f2060, 0xc0006c8280)
        /tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3
google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc000b032f0}, {0x2fad5e0, 0xc0006c8040}, 0xc000c00bb8, 0x2ce8d00)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc00093ed20}, {0x37d25a8, 0xc000b032f0}, 0xc00087a1e0, 0xc00075eac0)
        /tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000a65340, {0x382aaa8, 0xc00051f980}, 0xc000af2ea0, 0xc000e78120, 0xa8684a0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f
google.golang.org/grpc.(*Server).handleStream(0xc000a65340, {0x382aaa8, 0xc00051f980}, 0xc000af2ea0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294

Let’s switch the runner profile (first attempt shows it was in a bad state). We do this by changing the OCI URL to a different runner image (OCI URL):

$ waypoint runner profile list
! failed to create client: context deadline exceeded

$ waypoint runner profile list
Runner profiles
     NAME    | PLUGIN TYPE |            OCI URL            | TARGET RUNNER ID | DEFAULT
-------------+-------------+-------------------------------+------------------+----------
  kubernetes | kubernetes  | hashicorp/waypoint-odr:latest | *                | yes

$ waypoint runner profile set -name=kubernetes -oci-url=ghcr.io/hashicorp/waypoint/alpha-odr:latest -plugin-type=kubernetes
✓ Runner profile updated

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint runner profile list
Runner profiles
     NAME    | PLUGIN TYPE |                   OCI URL                   | TARGET RUNNER ID | DEFAULT
-------------+-------------+---------------------------------------------+------------------+----------
  kubernetes | kubernetes  | ghcr.io/hashicorp/waypoint/alpha-odr:latest | *                |

And again, when trying to build, we see the same thing

$ waypoint build -local=false

» Building example-nodejs2...
! error reading from server: EOF

with a crash

$ kubectl logs waypoint-server-0
2022-03-01T14:22:27.840Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-03-01T14:22:27.841Z [DEBUG] waypoint: home configuration directory: path=/data/.config/waypoint
2022-03-01T14:22:27.841Z [INFO]  waypoint.server: opening DB: path=/data/data.db
2022-03-01T14:22:27.841Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-03-01T14:22:27.841Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-03-01T14:22:27.966Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-03-01T14:22:27.966Z [DEBUG] waypoint.server.singleprocess.url_service: using saved URL guest token
2022-03-01T14:22:28.207Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-03-01T14:22:28.208Z [INFO]  waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-03-01T14:22:28.208Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-03-01T14:22:28.208Z [INFO]  waypoint.server.singleprocess.poll_queuer.project: starting
2022-03-01T14:22:28.208Z [INFO]  waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-03-01T14:22:28.208Z [INFO]  waypoint.server.singleprocess.prune: starting
2022-03-01T14:22:28.208Z [INFO]  waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-01T14:22:28.243Z [INFO]  waypoint.server.grpc: listener is wrapped with TLS
2022-03-01T14:22:28.243Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-03-01T14:22:28.243Z [INFO]  waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created

» Server configuration:

2022-03-01T14:22:28.285Z [INFO]  waypoint.server.http: listener is wrapped with TLS
2022-03-01T14:22:28.285Z [INFO]  waypoint.server: starting built-in server: addr=[::]:9701
2022-03-01T14:22:28.285Z [INFO]  waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
             DB Path: /data/data.db
        gRPC Address: [::]:9701
        HTTP Address: [::]:9702
       Auth Required: yes
  Browser UI Enabled: yes
         URL Service: api.waypoint.run:443 (account: guest)


» Server logs:

2022-03-01T14:22:28.285Z [INFO]  waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022-03-01T14:22:32.138Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:22:32.138Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=52.514µs
2022-03-01T14:22:32.141Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs request
2022-03-01T14:22:32.142Z [INFO]  waypoint.server.singleprocess: looking up ondemand runner config by id: id=01fx2t1dxtfvzn0e1ykvnryzjk
2022-03-01T14:22:32.142Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs response: error=<nil> duration=809.491µs
2022-03-01T14:22:36.750Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:22:36.750Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=95.275µs
2022-03-01T14:22:36.754Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetOnDemandRunnerConfig request
2022-03-01T14:22:36.754Z [INFO]  waypoint.server.singleprocess: looking up ondemand runner config by name: name=kubernetes
2022-03-01T14:22:36.754Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetOnDemandRunnerConfig response: error=<nil> duration=239.325µs
2022-03-01T14:22:36.755Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/UpsertOnDemandRunnerConfig request
2022-03-01T14:22:36.846Z [INFO]  waypoint.server.singleprocess: looking up ondemand runner config by id: id=01FX2T1DXTFVZN0E1YKVNRYZJK
2022-03-01T14:22:37.657Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/UpsertOnDemandRunnerConfig response: error=<nil> duration=901.154437ms
2022-03-01T14:22:38.695Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-01T14:22:38.696Z [ERROR] waypoint.server.grpc: unknown runner connected: runner_id=01FX2XMTWVXB6W3YN34Y9AZ2D8 id=01FX2XMTWVXB6W3YN34Y9AZ2D8
2022-03-01T14:22:38.696Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream response: error="rpc error: code = NotFound desc = runner ID not found: 01FX2XMTWVXB6W3YN34Y9AZ2D8" duration=877.234µs
2022-03-01T14:22:38.794Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:22:38.794Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:22:38.975Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:22:38.975Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:22:40.594Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:22:40.594Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=103.629µs
2022-03-01T14:22:40.597Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs request
2022-03-01T14:22:40.597Z [INFO]  waypoint.server.singleprocess: looking up ondemand runner config by id: id=01fx2t1dxtfvzn0e1ykvnryzjk
2022-03-01T14:22:40.597Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs response: error=<nil> duration=132.514µs
2022-03-01T14:23:12.891Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:23:12.891Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=48.728µs
2022-03-01T14:23:12.895Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject request
2022-03-01T14:23:12.895Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject response: error=<nil> duration=134.615µs
2022-03-01T14:23:13.138Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/QueueJob request
2022-03-01T14:23:13.139Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2
2022-03-01T14:23:13.139Z [INFO]  waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FX2XT77K36ERRBNECY2ETYRX
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]

goroutine 95 [running]:
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0xc0009efb80, 0x0)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0xc0009efb80)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0xc0005dd800)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0x8000107)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc000368810}, {0x2fad5e0, 0xc0005dd800})
        /tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b
github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000368720}, {0x2fad5e0, 0xc0005dd800}, 0x40, 0xc0008c0858)
        /tmp/wp-src/internal/server/auth.go:80 +0x204
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000368720}, {0x2fad5e0, 0xc0005dd800})
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000368690}, {0x2fad5e0, 0xc0005dd800}, 0xc000311640, 0xc0005dd980)
        /tmp/wp-src/internal/server/grpc_version.go:35 +0x8e
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000368690}, {0x2fad5e0, 0xc0005dd800})
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc0003682d0}, {0x2fad5e0, 0xc0005dd800}, 0xc000311640, 0xc0005dd940)
        /tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3
google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc0003682d0}, {0x2fad5e0, 0xc0005dd800}, 0xc000b5cbb8, 0x2ce8d00)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc0008015e0}, {0x37d25a8, 0xc0003682d0}, 0xc0008cef60, 0xc00120b000)
        /tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000899340, {0x382aaa8, 0xc0005f2d80}, 0xc000875200, 0xc000afeab0, 0xa8684a0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f
google.golang.org/grpc.(*Server).handleStream(0xc000899340, {0x382aaa8, 0xc0005f2d80}, 0xc000875200, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294

It’s now been several weeks. Checking the Alpha ODR listing of iamges: https://github.com/hashicorp/waypoint/pkgs/container/waypoint%2Falpha-odr I see there have been many since my last attempt.

I respect using “latest” on images. But I really need to know that when it pulls the new alpha-ocr, it gets the actual latest. So I’ll use the tagged SHA instead of “latest” this time

$ waypoint runner profile set -name=kubernetes -oci-url=ghcr.io/hashicorp/waypoint/alpha-odr:f38e20450 -plugin-type=kubernetes
✓ Runner profile updated

$ waypoint runner profile list
Runner profiles
     NAME    | PLUGIN TYPE |                    OCI URL                     | TARGET RUNNER ID | DEFAULT
-------------+-------------+------------------------------------------------+------------------+----------
  kubernetes | kubernetes  | ghcr.io/hashicorp/waypoint/alpha-odr:f38e20450 | *                |

However, that crashes waypoint-server-0 right away as well

waypoint-server-0                                        0/1     Error               8 (87s ago)     18d
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ kubectl logs waypoint-server-0
2022-03-19T15:41:34.689Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-03-19T15:41:34.690Z [DEBUG] waypoint: home configuration directory: path=/data/.config/waypoint
2022-03-19T15:41:34.690Z [INFO]  waypoint.server: opening DB: path=/data/data.db
2022-03-19T15:41:34.690Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-03-19T15:41:34.690Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-03-19T15:41:34.791Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-03-19T15:41:34.791Z [DEBUG] waypoint.server.singleprocess.url_service: using saved URL guest token
2022-03-19T15:41:34.997Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-03-19T15:41:34.997Z [INFO]  waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-03-19T15:41:34.997Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-03-19T15:41:34.997Z [INFO]  waypoint.server.singleprocess.poll_queuer.project: starting
2022-03-19T15:41:34.998Z [INFO]  waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-03-19T15:41:34.998Z [INFO]  waypoint.server.singleprocess.prune: starting
2022-03-19T15:41:34.998Z [INFO]  waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-19T15:41:35.032Z [INFO]  waypoint.server.grpc: listener is wrapped with TLS
2022-03-19T15:41:35.032Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-03-19T15:41:35.032Z [INFO]  waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created

» Server configuration:

             DB Path: /data/data.db
        gRPC Address: [::]:9701
        HTTP Address: [::]:9702
       Auth Required: yes
  Browser UI Enabled: yes
         URL Service: api.waypoint.run:443 (account: guest)


» Server logs:

2022-03-19T15:41:35.065Z [INFO]  waypoint.server.http: listener is wrapped with TLS
2022-03-19T15:41:35.066Z [INFO]  waypoint.server: starting built-in server: addr=[::]:9701
2022-03-19T15:41:35.066Z [INFO]  waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
2022-03-19T15:41:35.066Z [INFO]  waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022-03-19T15:41:54.649Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-19T15:41:54.649Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-19T15:41:58.346Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-19T15:41:58.383Z [ERROR] waypoint.server.grpc: unknown runner connected: runner_id=01FX2Y19MFEP6KQC35P7BZWWE8 id=01FX2Y19MFEP6KQC35P7BZWWE8
2022-03-19T15:41:58.383Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream response: error="rpc error: code = NotFound desc = runner ID not found: 01FX2Y19MFEP6KQC35P7BZWWE8" duration=36.81884ms
2022-03-19T15:41:58.749Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-19T15:41:58.749Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-19T15:42:09.352Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-19T15:42:09.352Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=29.643µs
2022-03-19T15:42:09.480Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerConfig request
2022-03-19T15:42:09.991Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-19T15:42:42.109Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-19T15:42:42.109Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=58.049µs
2022-03-19T15:42:42.114Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject request
2022-03-19T15:42:42.114Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject response: error=<nil> duration=240.114µs
2022-03-19T15:42:42.367Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/QueueJob request
2022-03-19T15:42:42.367Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2
2022-03-19T15:42:42.367Z [INFO]  waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FYHDGPNZK2S3EYC408CN57Y0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]

goroutine 134 [running]:
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0xc0006ff040, 0x0)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0xc0006ff040)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0xc000b75400)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0x8000102)
        /tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc000e84f00}, {0x2fad5e0, 0xc000b75400})
        /tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b
github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000e84d50}, {0x2fad5e0, 0xc000b75400}, 0x40, 0xc000e581f8)
        /tmp/wp-src/internal/server/auth.go:80 +0x204
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000e84d50}, {0x2fad5e0, 0xc000b75400})
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000e84cc0}, {0x2fad5e0, 0xc000b75400}, 0xc001129a20, 0xc000b75540)
        /tmp/wp-src/internal/server/grpc_version.go:35 +0x8e
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000e84cc0}, {0x2fad5e0, 0xc000b75400})
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc000e84660}, {0x2fad5e0, 0xc000b75400}, 0xc001129a20, 0xc000b754c0)
        /tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3
google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc000e84660}, {0x2fad5e0, 0xc000b75400}, 0xc000f0ebb8, 0x2ce8d00)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc0008cb360}, {0x37d25a8, 0xc000e84660}, 0xc000268600, 0xc0013db640)
        /tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000aeb340, {0x382aaa8, 0xc000c6ac00}, 0xc0008fe7e0, 0xc000c0d8f0, 0xa8684a0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f
google.golang.org/grpc.(*Server).handleStream(0xc000aeb340, {0x382aaa8, 0xc000c6ac00}, 0xc0008fe7e0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294

Uninstall

We showed earlier how to uninstall with helm. Here, since we installed with the waypoint binary, we need to use the binary to uninstall as well.

$ waypoint server uninstall -platform kubernetes
! Uninstalling Waypoint server requires approval.
  Rerun the command with -auto-approve to continue with the uninstall.


$ waypoint server uninstall -platform kubernetes -auto-approve
Uninstalling Waypoint server on platform "kubernetes" with context "install-1646140543"
✓ Snapshot "waypoint-server-snapshot-1647704697" generated

✓ Runner deployment deleted
✓ Statefulset and pods deleted
✓ Persistent Volume Claim deleted
✓ Service deleted
✓ Role binding deleted
✓ Cluster role deleted
✓ Cluster role binding deleted

Waypoint server successfully uninstalled for kubernetes platform

I checked to see if perhaps waypoint needed an update

$ sudo apt update
$ waypoint version
CLI: v0.7.2 (6e06e1a2d)
$ sudo apt install waypoint
Reading package lists... Done
Building dependency tree
Reading state information... Done
waypoint is already the newest version (0.7.2).
The following packages were automatically installed and are no longer required:
  liblttng-ust-ctl4 liblttng-ust0
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 134 not upgraded.

And install it again

$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!

The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1647704975".
Use the "waypoint context" CLI to manage CLI contexts.

The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".

To launch and authenticate into the Web UI, run:
waypoint ui -authenticate

Advertise Address: 192.168.1.12:9701
Web UI Address: https://192.168.1.12:9702

/content/images/2022/03/waypoint-02.png

We see our context has moved to the new IP

$ waypoint context list
    |        NAME        |  PLATFORM  |  SERVER ADDRESS
----+--------------------+------------+--------------------
  * | install-1647704975 | kubernetes | 192.168.1.12:9701

Also, we can see Waypoint runners are set to the default latest image

$ waypoint runner profile list
Runner profiles
     NAME    | PLUGIN TYPE |            OCI URL            | TARGET RUNNER ID | DEFAULT
-------------+-------------+-------------------------------+------------------+----------
  kubernetes | kubernetes  | hashicorp/waypoint-odr:latest | *                | yes

We need to set the HarborCred before trying again

$ cat /home/builder/harborAuth.json | base64 -w 0
bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=
$ waypoint config set -runner HARBORCRED=bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=

We waypoint init to setup our local project

$ waypoint init
✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "example-nodejs2" and all apps are registered with the server.

Project initialized!

You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.

I want to check the project, so I’ll the auth token

$ waypoint user token
rHhYzVXQBcJzrGzb5wK8WSUWSEY7FXs3Gm4XuEEuXSmP6jQDi1kwLuyyQQap9GxwcZmTQGUnnbxKFMQuwRpLVMtZUuMu9bCm9vSAtu9YyLPsLbysTW4YU1t7DKA2g2Qjfa9qaaiPALtczTDjQ2KQCj4Jt4HFFEjzx

And login via the UI

/content/images/2022/03/waypoint-03.png

And we see our project defined

/content/images/2022/03/waypoint-04.png

Let’s try and fire it up.

/content/images/2022/03/waypoint-05.png

and we see the crash

2022-03-19T16:26:06.956Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2                                                                          ││ 2022-03-19T16:26:06.956Z [INFO]  waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FYHG067C1AB2MR2304HP0WJP                                        ││ panic: runtime error: invalid memory address or nil pointer dereference                                                                                                       ││ [signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]                                                                                                      ││                                                                                                                                                                               ││ goroutine 1648 [running]:                                                                                                                                                     ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0xc000a50000, 0x0)                     ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d                                                                                                       ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0xc000a50000)                               ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f                                                                                                        ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0xc000c14380)                                ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b                                                                                                       ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0x8000103)                                           ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c                                                                                                        ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc0000f8c00}, {0x2fad5e0, 0xc000c14380})                                      ││     /tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b                                                                                                                  ││ github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc0000f8b10}, {0x2fad5e0, 0xc000c14380}, 0x40, 0xc000d46978)                            ││     /tmp/wp-src/internal/server/auth.go:80 +0x204                                                                                                                             ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc0000f8b10}, {0x2fad5e0, 0xc000c14380})                                                                       ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9                                                                                                           ││ github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc0000f8a80}, {0x2fad5e0, 0xc000c14380}, 0xc0001bf2e0, 0xc000c14580)                 ││     /tmp/wp-src/internal/server/grpc_version.go:35 +0x8e                                                                                                                      ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc0000f8a80}, {0x2fad5e0, 0xc000c14380})                                                                       ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9                                                                                                           ││ github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc0000f8180}, {0x2fad5e0, 0xc000c14380}, 0xc0001bf2e0, 0xc000c14540)                     ││     /tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3                                                                                                                         ││ google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc0000f8180}, {0x2fad5e0, 0xc000c14380}, 0xc00125ebb8, 0x2ce8d00)                                      ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f                                                                                                           ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc00094a280}, {0x37d25a8, 0xc0000f8180}, 0xc000314b40, 0xc0010f5d20)                ││     /tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138                                                                                                                 ││ google.golang.org/grpc.(*Server).processUnaryRPC(0xc000b40000, {0x382aaa8, 0xc0008de600}, 0xc0007daa20, 0xc000e0c0f0, 0xa8684a0, 0x0)                                         ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f                                                                                                          ││ google.golang.org/grpc.(*Server).handleStream(0xc000b40000, {0x382aaa8, 0xc0008de600}, 0xc0007daa20, 0x0)                                                                     ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a                                                                                                          ││ google.golang.org/grpc.(*Server).serveStreams.func1.2()                                                                                                                       ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98                                                                                                            ││ created by google.golang.org/grpc.(*Server).serveStreams.func1                                                                                                                ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294                                                                                                           ││ Stream closed EOF for default/waypoint-server-0 (server)                                                                    

I tried several more things…

I tried:

  • forcing a new project
  • hardcoding creds
  • adding a git ref

the last run used (and the password has since been reset, but that is valid b64 string):

$ cat waypoint.hcl
project = "example-nodejs3"

runner {
  enabled = true

  data_source "git" {
     url = "https://github.com/idjohnson/waypoint-examples.git"
     ref = "main"
     path = "kubernetes/nodejs"
  }
}

app "example-nodejs3" {

  labels = {
    "service" = "example-nodejs",
    "env"     = "dev"
  }

  build {
    use "pack" {}
    registry {
      use "docker" {
        image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
        tag   = "1"
        encoded_auth = "ewoJInVzZXJuYW1lIjogImNocmRlbW8iLAoJInBhc3N3b3JkIjogIlRlbXBwYXNzMSIsCgkiZW1haWwiOiAiaXNhYWMuam9obnNvbkBnbWFpbC5jb20iCn0K"
      }
    }
  }

  deploy {
    use "kubernetes" {
      image_secret = "myharborreg"
      probe_path = "/"
    }
  }

  release {
    use "kubernetes" {
      // Sets up a load balancer to access released application
      load_balancer = true
      port          = 3000
    }
  }
}

But in each case, waypoint-server-0 crashed

2022-03-19T16:38:53.176Z [INFO]  waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FYHGQJFRV4VDD4XEEJS999ME                                        ││ panic: runtime error: invalid memory address or nil pointer dereference                                                                                                       ││ [signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]                                                                                                      ││                                                                                                                                                                               ││ goroutine 216 [running]:                                                                                                                                                      ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0xc000ca32c0, 0x0)                     ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d                                                                                                       ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0xc000ca32c0)                               ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f                                                                                                        ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0xc000023b40)                                ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b                                                                                                       ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0x8000103)                                           ││     /tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c                                                                                                        ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc000ba59e0}, {0x2fad5e0, 0xc000023b40})                                      ││     /tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b                                                                                                                  ││ github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000ba58c0}, {0x2fad5e0, 0xc000023b40}, 0x40, 0xc0006eaf48)                            ││     /tmp/wp-src/internal/server/auth.go:80 +0x204                                                                                                                             ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000ba58c0}, {0x2fad5e0, 0xc000023b40})                                                                       ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9                                                                                                           ││ github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000ba5830}, {0x2fad5e0, 0xc000023b40}, 0xc000661360, 0xc000023d40)                 ││     /tmp/wp-src/internal/server/grpc_version.go:35 +0x8e                                                                                                                      ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000ba5830}, {0x2fad5e0, 0xc000023b40})                                                                       ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9                                                                                                           ││ github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc000ba5440}, {0x2fad5e0, 0xc000023b40}, 0xc000661360, 0xc000023c40)                     ││     /tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3                                                                                                                         ││ google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc000ba5440}, {0x2fad5e0, 0xc000023b40}, 0xc000fc7bb8, 0x2ce8d00)                                      ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f                                                                                                           ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc000ca0780}, {0x37d25a8, 0xc000ba5440}, 0xc00094c8a0, 0xc00029dd20)                ││     /tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138                                                                                                                 ││ google.golang.org/grpc.(*Server).processUnaryRPC(0xc000cb61c0, {0x382aaa8, 0xc000baec00}, 0xc0009ae000, 0xc0008dca80, 0xa8684a0, 0x0)                                         ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f                                                                                                          ││ google.golang.org/grpc.(*Server).handleStream(0xc000cb61c0, {0x382aaa8, 0xc000baec00}, 0xc0009ae000, 0x0)                                                                     ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a                                                                                                          ││ google.golang.org/grpc.(*Server).serveStreams.func1.2()                                                                                                                       ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98                                                                                                            ││ created by google.golang.org/grpc.(*Server).serveStreams.func1                                                                                                                ││     /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294                                                                                                           ││ Stream closed EOF for default/waypoint-server-0 (server)                                                  

Crashing

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build

» Building example-nodejs3...
There are local changes that do not match the remote repository. By default,
Waypoint will perform this operation using a remote runner that will use the
remote repository’s git ref and not these local changes. For these changes
to be used for future operations, either commit and push, or run the operation
locally with the -local flag.
! error reading from server: EOF
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build
! failed to create client: context deadline exceeded

But if i change to local, this works just fine

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=true

» Building example-nodejs3...
  Performing operation locally
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
 │ [exporter] Adding layer 'process-types'[exporter] Adding label 'io.buildpacks.lifecycle.metadata'[exporter] Adding label 'io.buildpacks.build.metadata'[exporter] Adding label 'io.buildpacks.project.metadata'[exporter] Setting default process type 'web'[exporter] Saving example-nodejs3...
 │ [exporter] *** Images (56ddadcdd1b9):
 │ [exporter]       example-nodejs3
 │ [exporter] Adding cache layer 'heroku/nodejs-engine:nodejs'[exporter] Adding cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image

Generated new Docker image: example-nodejs3:latest
✓ Tagging Docker image: example-nodejs3:latest => harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
✓ Pushing Docker image...
 │ ab2b7fb6b8c0: Layer already exists
 │ 5de7c0ba1e7a: Pushed
 │ 4c226e279fd5: Pushed
 │ aada41e93513: Layer already exists
 │ c06c935628e1: Layer already exists
 │ 4db08ac7c633: Layer already exists
 │ 8891c4369930: Layer already exists
 │ 68a85fa9d77e: Layer already exists
 │ 1: digest: sha256:3bdc39376bd4e26c76785cca34555b844e1ec8f5b6b3c1a0c7e6d99d03fc61
 │ 14 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$

And we see IF we build locally, we are fine

/content/images/2022/03/waypoint-06.png

But as you can see, if I try again with remote, it crashes

 │ 68a85fa9d77e: Layer already exists
 │ 1: digest: sha256:3bdc39376bd4e26c76785cca34555b844e1ec8f5b6b3c1a0c7e6d99d03fc61
 │ 14 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=false

» Building example-nodejs3...
There are local changes that do not match the remote repository. By default,
Waypoint will perform this operation using a remote runner that will use the
remote repository’s git ref and not these local changes. For these changes
to be used for future operations, either commit and push, or run the operation
locally with the -local flag.
! error reading from server: EOF
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=false
! failed to create client: context deadline exceeded
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$

Adding local runner

Let me try adding a local runner on a linux box

Install waypoint

builder@builder-HP-EliteBook-745-G5:~$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
OK
builder@builder-HP-EliteBook-745-G5:~$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Get:2 https://apt.releases.hashicorp.com focal InRelease [14.6 kB]
Hit:3 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:6 https://baltocdn.com/helm/stable/debian all InRelease
Get:7 https://apt.releases.hashicorp.com focal/main amd64 Packages [50.4 kB]
Fetched 65.0 kB in 1s (48.9 kB/s)
Reading package lists... Done
builder@builder-HP-EliteBook-745-G5:~$ sudo apt-get update && sudo apt-get install waypoint
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 https://apt.releases.hashicorp.com focal InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 https://baltocdn.com/helm/stable/debian all InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfprint-2-tod1 libllvm10
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  waypoint
0 upgraded, 1 newly installed, 0 to remove and 73 not upgraded.
Need to get 112 MB of archives.
After this operation, 173 MB of additional disk space will be used.
Get:1 https://apt.releases.hashicorp.com focal/main amd64 waypoint amd64 0.7.2 [112 MB]
Fetched 112 MB in 5s (23.3 MB/s)
Selecting previously unselected package waypoint.
(Reading database ... 232177 files and directories currently installed.)
Preparing to unpack .../waypoint_0.7.2_amd64.deb ...
Unpacking waypoint (0.7.2) ...
Setting up waypoint (0.7.2) ...

If you try and add an agent with self signed certs (as is the default), you’ll see errors in the waypoint logs

│ 2022/03/19 16:55:09 http: TLS handshake error from 10.42.2.0:7886: remote error: tls: bad certificate                                                                         ││ 2022/03/19 16:55:10 http: TLS handshake error from 10.42.2.0:30927: remote error: tls: bad certificate                                                                        ││ 2022/03/19 16:55:12 http: TLS handshake error from 10.42.2.0:6183: remote error: tls: bad certificate                                                                         ││ 2022/03/19 16:55:17 http: TLS handshake error from 10.42.2.0:27513: remote error: tls: bad certificate  

So make sure to add TLK SKIP VERIFY

$ WAYPOINT_SERVER_TLS_SKIP_VERIFY=1 WAYPOINT_SERVER_ADDR=192.168.1.12:9701 WAYPOINT_SERVER_TLS=true WAYPOINT_SERVER_TOKEN=rHhYzVXQBcJzrGzb5wK8WSUW
SEY7FXs3Gm4XuEEuXSmP6jQDi1kwLuyyQQap9GxwcZmTQGUnnbxKFMQuwRpLVMtZUuMu9bCm9vSAtu9YyLPsLbysTW4YU1t7DKA2g2Qjfa9qaaiPALtczTDjQ2KQCj4Jt4HFFEjzx waypoint runner agent
2022-03-19T11:58:20.947-0500 [INFO]  waypoint.runner.agent: initializing the runner
2022-03-19T11:58:20.947-0500 [INFO]  waypoint.runner.agent: starting runner: id=01FYHHV6WK76ZK1SH109VECJ6M
2022-03-19T11:58:20.949-0500 [INFO]  waypoint.runner.agent.runner.config_recv: new configuration received

» Runner configuration:
  Server address: 192.168.1.12:9701

» Runner logs:
Operating as a static Runner
2022-03-19T11:58:21.450-0500 [INFO]  waypoint.runner.agent.runner: runner registered with server and ready
2022-03-19T11:58:21.450-0500 [INFO]  waypoint.runner.agent.runner: waiting for job assignment

However, despite seeing the waypoint runner in my shell, asking waypoint for active runners does not show any

$ waypoint runner profile list
Runner profiles
     NAME    | PLUGIN TYPE |            OCI URL            | TARGET RUNNER ID | DEFAULT
-------------+-------------+-------------------------------+------------------+----------
  kubernetes | kubernetes  | hashicorp/waypoint-odr:latest | *                | yes

But that too crashes things (and we crash the agent on the local linux host)

/content/images/2022/03/waypoint-07.png

builder@builder-HP-EliteBook-745-G5:~$ WAYPOINT_SERVER_TLS_SKIP_VERIFY=1 WAYPOINT_SERVER_ADDR=192.168.1.12:9701 WAYPOINT_SERVER_TLS=true WAYPOINT_SERVER_TOKEN=rHhYzVXQBcJzrGzb5wK8WSUW
SEY7FXs3Gm4XuEEuXSmP6jQDi1kwLuyyQQap9GxwcZmTQGUnnbxKFMQuwRpLVMtZUuMu9bCm9vSAtu9YyLPsLbysTW4YU1t7DKA2g2Qjfa9qaaiPALtczTDjQ2KQCj4Jt4HFFEjzx waypoint runner agent
2022-03-19T11:58:20.947-0500 [INFO]  waypoint.runner.agent: initializing the runner
2022-03-19T11:58:20.947-0500 [INFO]  waypoint.runner.agent: starting runner: id=01FYHHV6WK76ZK1SH109VECJ6M
2022-03-19T11:58:20.949-0500 [INFO]  waypoint.runner.agent.runner.config_recv: new configuration received

» Runner configuration:
  Server address: 192.168.1.12:9701

» Runner logs:
Operating as a static Runner
2022-03-19T11:58:21.450-0500 [INFO]  waypoint.runner.agent.runner: runner registered with server and ready
2022-03-19T11:58:21.450-0500 [INFO]  waypoint.runner.agent.runner: waiting for job assignment
2022-03-19T12:03:20.453-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-03-19T12:03:20.453-0500 [WARN]  waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:20.453-0500 [ERROR] waypoint.runner.agent.runner.config_recv: error receiving configuration, exiting: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-03-19T12:03:22.454-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.1.12:9701: connect: connection refused""
2022-03-19T12:03:22.454-0500 [WARN]  waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:43.230-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.1.12:9701: i/o timeout""
2022-03-19T12:03:43.230-0500 [WARN]  waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:45.231-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.1.12:9701: i/o timeout""
2022-03-19T12:03:45.231-0500 [WARN]  waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:47.232-0500 [INFO]  waypoint.runner.agent.runner: waiting for job assignment
2022-03-19T12:03:47.234-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = NotFound desc = runner ID not found: 01FYHHV6WK76ZK1SH109VECJ6M"
2022-03-19T12:03:47.234-0500 [ERROR] waypoint.runner.agent: runner unexpectedly deregistered, exiting
2022-03-19T12:03:47.234-0500 [INFO]  waypoint.runner.agent: quit request received, gracefully stopping runner
2022-03-19T12:03:47.234-0500 [WARN]  waypoint.runner.agent.runner.watch_config: exiting due to context ended

builder@builder-HP-EliteBook-745-G5:~$

Cleanup

We’ll remove waypoint locally again

$ waypoint server uninstall -platform kubernetes -auto-approve
Uninstalling Waypoint server on platform "kubernetes" with context "install-1647704975"
✓ Snapshot "waypoint-server-snapshot-1647709511" generated

✓ Runner deployment deleted
✓ Statefulset and pods deleted
✓ Persistent Volume Claim deleted
✓ Service deleted
✓ Role binding deleted
✓ Cluster role deleted
✓ Cluster role binding deleted

Waypoint server successfully uninstalled for kubernetes platform

And, of course, change passwords (as i exposed via plain text)

/content/images/2022/03/waypoint-08.png

Via Github Actions

First, we need to setup an external k8s base64 encoded env var

You can get the value if authed locally

$ cat ~/.kube/config | base64 -w 0

/content/images/2022/03/waypoint-09.png

We can then define a Github Actions workflow to build and deploy our hcl using a waypoint binary we pull down on the fly.

I built the Github Actions yaml based largely on this Gitlab example

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ cat .github/workflows/waypoint.yml
name: GitHub To Waypoint

on: [pull_request, push]

env:
  WAYPOINT_VERSION: '0.7.2'
  WAYPOINT_SERVER_ADDR: ''
  WAYPOINT_SERVER_TOKEN: ''
  WAYPOINT_SERVER_TLS: '1'
  WAYPOINT_SERVER_TLS_SKIP_VERIFY: '1'

jobs:
  HostedActions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 The job was automatically triggered by a $ event."
      - run: echo "🐧 This job is now running on a $ server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is $ and your repository is $."
      - name: Check out repository code
        uses: actions/checkout@v2
      - run: echo "💡 The $ repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: install waypoint
        run: |
          wget -q -O /tmp/waypoint.zip https://releases.hashicorp.com/waypoint/$/waypoint_$_linux_amd64.zip
          unzip -d /usr/local/bin /tmp/waypoint.zip
          rm -rf /tmp/waypoint*
      - name: set kubernetes config
        run: |
          mkdir -p ~/.kube
          set -x
          echo $ | base64 --decode > ~/.kube/config

      - name: install waypoint
        run: |
          # go to src dir
          cd kubernetes/nodejs
          waypoint init
          waypoint build
          waypoint deploy
          waypoint release

In this model we do not have a waypoint server to interrogate or check. We are just using an on-the-fly waypoint to build and release.

We can see some output (though it appears hung at the check service step)

/content/images/2022/03/waypoint-10.png

We see it deployed, but again, it seems confused on how to properly expose a service on my k3s 1.23 cluster

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl get pods | grep example
example-nodejs-v3-ffdd6fd68-8bqlv                        1/1     Running             0               18d
svclb-example-nodejs-vzrcb                               1/1     Running             0               18d
svclb-example-nodejs-8tv7n                               1/1     Running             0               18d
svclb-example-nodejs-xsvgc                               1/1     Running             0               18d
svclb-example-nodejs-h5t9t                               1/1     Running             0               18d
example-nodejs-v2-78b4dd576d-tc9vr                       1/1     Running             0               18d
svclb-example-nodejs-c7th7                               1/1     Running             0               18d
svclb-example-nodejs-tznc4                               1/1     Running             0               18d
svclb-example-nodejs-sds7v                               1/1     Running             0               18d
example-nodejs2-v1-5d4dd7cc4-vmbv4                       1/1     Running             0               3h37m
example-nodejs-v1-6d89c89657-ss5fm                       0/1     ImagePullBackOff    0               4d8h
example-nodejs3-v1-df8dff4cd-mpjq8                       1/1     Running             0               4m27s
svclb-example-nodejs3-pmszp                              0/1     Pending             0               2m34s
svclb-example-nodejs3-mntnh                              0/1     Pending             0               2m34s
svclb-example-nodejs3-ffhsp                              0/1     Pending             0               2m34s
svclb-example-nodejs3-92l9r                              0/1     Pending             0               2m34s
svclb-example-nodejs3-2dw28                              0/1     Pending             0               2m34s
svclb-example-nodejs3-wdt86                              0/1     Pending             0               2m34s
svclb-example-nodejs3-tldxw                              0/1     Pending             0               2m34s

As we see, indeed the service is stuck in pending (since it cannot grab port 3000 on all my servers)

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl get svc | grep example
example-auth                               ClusterIP      10.43.106.217   <none>                                                                                          3000/TCP                              299d
example-nodejs                             LoadBalancer   10.43.80.174    192.168.1.12,192.168.1.159,192.168.1.205,192.168.1.214,192.168.1.33,192.168.1.57,192.168.1.77   3000:32482/TCP                        18d
example-nodejs3                            LoadBalancer   10.43.225.183   <pending>                                                                                       3000:32244/TCP                        3m41s

I can unblock by removing the last service

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl delete svc example-nodejs
service "example-nodejs" deleted

Then we can see it pull port 3000

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl get svc | grep example
example-auth                               ClusterIP      10.43.106.217   <none>                                                                                          3000/TCP                              299d
example-nodejs3                            LoadBalancer   10.43.225.183   192.168.1.12,192.168.1.159,192.168.1.205,192.168.1.214,192.168.1.33,192.168.1.57,192.168.1.77   3000:32244/TCP                        4m47s

And this then succeeds

/content/images/2022/03/waypoint-11.png

and we can indeed see the app is exposed on port 3000

/content/images/2022/03/waypoint-12.png

Let’s push a change and see if that works.

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ git diff kubernetes/nodejs/views/pages/index.ejs
diff --git a/kubernetes/nodejs/views/pages/index.ejs b/kubernetes/nodejs/views/pages/index.ejs
index f6c45be..f487206 100644
--- a/kubernetes/nodejs/views/pages/index.ejs
+++ b/kubernetes/nodejs/views/pages/index.ejs
@@ -18,6 +18,7 @@
         <h1>This Node.js app was deployed with Waypoint.</h1>
         <p>
           Try making a change to this text locally and run <code>waypoint up</code> again to see it.
+         <b>I changed it!</b>
         </p>
         <p>
           Read the <a href="https://waypointproject.io/docs">documentation</a> for more about Waypoint.

It made a change, but I see no evidence it actually changed

/content/images/2022/03/waypoint-13.png

I can confirm it did build and push to my private CR:

from the logs

/content/images/2022/03/waypoint-15.png

and on Harbor

/content/images/2022/03/waypoint-14.png

I port-forwarded to the new deploy and it too wasn’t updated

/content/images/2022/03/waypoint-16.png

I fear it’s doing to re-using tags. Looking at the events on the pod

Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  6m34s  default-scheduler  Successfully assigned default/example-nodejs3-v1-6867f6fc48-mswv4 to builder-macbookpro2
  Normal  Pulled     6m33s  kubelet            Container image "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1" already present on machine
  Normal  Created    6m32s  kubelet            Created container example-nodejs3

We can fix this by not reusing the same tag. This is configured in the HCL

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index 31227ba..fedba49 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -22,7 +22,7 @@ app "example-nodejs3" {
     registry {
       use "docker" {
         image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
-        tag   = "1"
+        tag   = gitrefpretty()
         encoded_auth = "ewoJInVzZXJuYW1lIjogImNocmRlbW8iLAoJInBhc3N3b3JkIjogIlRlbXBwYXNzMSIsCgkiZW1haWwiOiAiaXNhYWMuam9obnNvbkBnbWFpbC5jb20iCn0K"
       }
     }

Now we commit and push

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git add waypoint.hcl
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git commit -m "use ref tag"
[main 544220a] use ref tag
 1 file changed, 1 insertion(+), 1 deletion(-)
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 16 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 430 bytes | 430.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/idjohnson/waypoint-examples.git
   c6df4b1..544220a  main -> main

This time it worked

/content/images/2022/03/waypoint-17.png

$ kubectl describe pod example-nodejs3-v1-7c77f4697c-5qkm6 | grep image
  Normal  Pulling    2m57s  kubelet            Pulling image "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:544220adb33cf427d4bcf27519f31dc1486f59c7"
  Normal  Pulled     2m49s  kubelet            Successfully pulled image "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:544220adb33cf427d4bcf27519f31dc1486f59c7" in 7.415101299s

One last change; as you recall, to debug, we set the Harbor Reg creds inline.

Let’s fix those to use a GH secret as well.

First add a secret with the creds

/content/images/2022/03/waypoint-18.png

Then change the waypoint HCL to use a config that we will set in the Github Actions workflow to the secret

$ git diff
diff --git a/.github/workflows/waypoint.yml b/.github/workflows/waypoint.yml
index adfe239..0605bcf 100644
--- a/.github/workflows/waypoint.yml
+++ b/.github/workflows/waypoint.yml
@@ -36,6 +36,9 @@ jobs:
           # go to src dir
           cd kubernetes/nodejs
           waypoint init
+          set +x
+          waypoint config set -runner HARBORCRED=$
+          set -x
           waypoint build
           waypoint deploy
           waypoint release
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index fedba49..020033d 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -23,7 +23,7 @@ app "example-nodejs3" {
       use "docker" {
         image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
         tag   = gitrefpretty()
-        encoded_auth = "ewoJInVzZXJuYW1lIjogImNocmRlbW8iLAoJInBhc3N3b3JkIjogIlRlbXBwYXNzMSIsCgkiZW1haWwiOiAiaXNhYWMuam9obnNvbkBnbWFpbC5jb20iCn0K"
+        encoded_auth = ${HARBORCRED}
       }
     }
   }

Integrating GH Actions and a persistent Waypoint server

The final path I wanted to explore, and likely the one most would want to emulate, was to use a waypoint server on which teams could collaborate but build using a PR workflow using Github ephemeral agents.

I spun up a fresh waypoint server in my local cluster using the waypoint CLI

```bash
$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!

The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1645496978".
Use the "waypoint context" CLI to manage CLI contexts.

The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".

To launch and authenticate into the Web UI, run:
waypoint ui -authenticate

Advertise Address: 192.168.1.12:9701
Web UI Address: https://1192.168.1.12:9702

/content/images/2022/03/waypoint-21.png

I then punched a hole in my router so we could ingress from 9701 and 9702 to my Kubernetes service running locally

/content/images/2022/03/waypoint-19.png

Verification:

/content/images/2022/03/waypoint-22.png

I used waypoint user token to get a new token and added that as a secret in Github Actions secrets.

/content/images/2022/03/waypoint-20.png

Lastly, I updated my Github Actions workflow to build and deploy locally, but use the remote server for tracking.

name: GitHub To Waypoint

on: [pull_request, push] 

env:
  WAYPOINT_VERSION: '0.7.2'
  WAYPOINT_SERVER_ADDR: '73.242.50.46:9701'
  WAYPOINT_SERVER_TLS: '1'
  WAYPOINT_SERVER_TLS_SKIP_VERIFY: '1'

jobs:
  HostedActions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 The job was automatically triggered by a $ event."
      - run: echo "🐧 This job is now running on a $ server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is $ and your repository is $."
      - name: Check out repository code
        uses: actions/checkout@v2
      - run: echo "💡 The $ repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: install waypoint
        run: |
          wget -q -O /tmp/waypoint.zip https://releases.hashicorp.com/waypoint/$/waypoint_$_linux_amd64.zip
          unzip -d /usr/local/bin /tmp/waypoint.zip
          rm -rf /tmp/waypoint*
      - name: set kubernetes config
        run: |
          mkdir -p ~/.kube
          set -x
          echo $ | base64 --decode > ~/.kube/config

      - name: install waypoint
        run: |
          # go to src dir
          cd kubernetes/nodejs 
          # sed -i "s/encoded_auth =.*/encoded_auth = \"$\"/g" waypoint.hcl
          set +x
          waypoint config set HARBORCRED=$
          set -x
          waypoint init
          waypoint build -local=true
          waypoint deploy -local=true
          waypoint release
        env: # Or as an environment variable
          WAYPOINT_SERVER_TOKEN: $
          WP_VAR_harborcred: $

One nuance to using the local runner that depended on a set credential was that it would get validated and fail on parsing the waypoint.hcl

Run # go to src dir
-> Validating configuration file...
-> Configuration file appears valid
-> Validating server credentials...
-> Connection to Waypoint server was successful
-> Checking if project "example-nodejs3" is registered...
-> Project "example-nodejs3" and all apps are registered with the server.

Project initialized!

You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
+ waypoint build -local=true

» Building example-nodejs3...
  Performing operation locally
! /home/runner/work/waypoint-examples/waypoint-examples/kubernetes/nodejs/waypoint.hcl:26,27-37:
  Unknown variable; There is no variable named "HARBORCRED"., and 1 other
  diagnostic(s)

One issue was the syntax; having “=” in the real value threw it off. I could mitigate by wrapping the value in double quotes.

encoded_auth = "SGV5LCBob3cgaXMgaXQgZ29pbmcuICBTdXJlIHdpbGwgYmUgbmljZSB3aGVuIHdpbnRlciBpcyBkb25lLiBMb29raW5nIGZvcndhcmQgdG8gZ2V0dGluZyBvdXQgYW5kIGRvaW5nIHNvbWUgY2FtcGluZy4K=="

But clearly, we need to solve variables that can be at build time in an ephemeral Github runner and still be used in Waypoint as a Kubernetes runner.

To do this, I moved it to a proper waypoint “variable”.

project = "example-nodejs3"

runner {
  enabled = true

  data_source "git" {
     url = "https://github.com/idjohnson/waypoint-examples.git"
     ref = "main"
     path = "kubernetes/nodejs"
  }
}

app "example-nodejs3" {

  labels = {
    "service" = "example-nodejs",
    "env"     = "dev"
  }

  build {
    use "pack" {}
    registry {
      use "docker" {
        image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
        tag   = gitrefpretty()
        encoded_auth = var.harborcred
      }
    }
  }

  deploy {
    use "kubernetes" {
      image_secret = "myharborreg"
      probe_path = "/"
    }
  }

  release {
    use "kubernetes" {
      // Sets up a load balancer to access released application
      load_balancer = true
      port          = 3000
    }
  }
}

variable "harborcred" {
  type    = string
  default = null
}

Now there is a var that if I skip, it just defaults to blank.

As you saw above the Github Actions, we passed the var in using the WP_VAR environment variable syntax of waypoint that will turn “WP_VAR_harborcred” into the variable “harborcred”. (The other option would be to pass in a file or use the “-var” command line variable).

        env: # Or as an environment variable
          WAYPOINT_SERVER_TOKEN: $
          WP_VAR_harborcred: $

Now when we run we see it built in Github

/content/images/2022/03/waypoint-24.png

-> Kubernetes client connected to https://73.242.50.46:12345 with namespace default
-> Preparing deployment...
-> 
-> Expected "http" port "3000" for app "example-nodejs3-v1"
-> Updating deployment...
-> Waiting for deployment...
-> Waiting on deployment to become available: requested=1 running=2 ready=1
-> Deployment successfully rolled out!

  Performing operation locally
-> Gathering health report for Kubernetes deployment...
-> Checking status of the Kubernetes deployment resource...
-> Finished building report for Kubernetes deployment resource
Determining overall container health...
-> Finished building report for Kubernetes platform
 *: Deployment "example-nodejs3-v1" is reporting partially available!
 *: Waypoint detected that the current deployment is not ready, however your application
might be available or still starting up.
 *: pod "example-nodejs3-v1-6dfb9ffc96-2xwpm" is reporting "ALIVE"

» Releasing...
  Performing operation locally
-> Initializing Kubernetes client...
-> Kubernetes client connected to https://73.242.50.46:12345 with namespace default
-> Preparing service...
-> Updating existing service...
-> Waiting for service to become ready...
-> Service is ready!

  Performing operation locally
-> Gathering health report for Kubernetes release...
-> Checking status of Kubernetes service resource "example-nodejs3"...
-> Finished building report for Kubernetes service resource
-> Finished building report for Kubernetes platform

The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."

   Release URL: http://192.168.1.12:3000
Deployment URL: https://deadly-pure-ladybird--v1.waypoint.run
+ waypoint release

» Releasing example-nodejs3...
The deployment "1" is already the released deployment. Nothing to be done.

The app is deployed and running

/content/images/2022/03/waypoint-25.png

The Complete Flow

We see a change made:

diff --git a/kubernetes/nodejs/views/pages/index.ejs b/kubernetes/nodejs/views/pages/index.ejs
index f487206..3f7abbf 100644
--- a/kubernetes/nodejs/views/pages/index.ejs
+++ b/kubernetes/nodejs/views/pages/index.ejs
@@ -18,7 +18,7 @@
         <h1>This Node.js app was deployed with Waypoint.</h1>
         <p>
           Try making a change to this text locally and run <code>waypoint up</code> again to see it.
-         <b>I changed it!</b>
+         <b>I changed it again!</b>
         </p>
         <p>
           Read the <a href="https://waypointproject.io/docs">documentation</a> for more about Waypoint.

Triggers a build which results in a live site update

/content/images/2022/03/waypoint-26.png

And yet we can still see our build and deployment logs

/content/images/2022/03/waypoint-27.png

However, until we change deployments back to Waypoint, we will see errors in the deployment log (albeit the deployment happened)

/content/images/2022/03/waypoint-28.png

I noted that even if I set the project with -p, it would crash waypoint-server-0 if I tried to release remotely. Only “-local” worked

      - name: install waypoint
        run: |
          # go to src dir
          cd kubernetes/nodejs 
          # sed -i "s/encoded_auth =.*/encoded_auth = \"$\"/g" waypoint.hcl
          set +x
          waypoint config set HARBORCRED=$
          waypoint config set -runner HARBORCRED=$
          set -x
          waypoint init
          waypoint build -p example-nodejs3 -local=true
          waypoint deploy -p example-nodejs3
          waypoint release -p example-nodejs3

/content/images/2022/03/waypoint-29.png

I also overcame one other slight annoyance. By default, Waypoint leaves all the prior releases running:

$ kubectl get pods | grep example-node
example-nodejs-v3-ffdd6fd68-8bqlv                        1/1     Running             0                19d
example-nodejs-v2-78b4dd576d-tc9vr                       1/1     Running             0                19d
example-nodejs2-v1-5d4dd7cc4-vmbv4                       1/1     Running             0                31h
svclb-example-nodejs3-tldxw                              1/1     Running             0                28h
svclb-example-nodejs3-pmszp                              1/1     Running             0                28h
svclb-example-nodejs3-2dw28                              1/1     Running             0                28h
svclb-example-nodejs3-92l9r                              1/1     Running             0                28h
svclb-example-nodejs3-ffhsp                              1/1     Running             0                28h
svclb-example-nodejs3-wdt86                              1/1     Running             0                28h
svclb-example-nodejs3-mntnh                              1/1     Running             0                28h
example-nodejs3-v2-758ddc49dc-stlhv                      1/1     Running             1 (13m ago)      7h38m
example-nodejs3-v3-854dd4d55f-cr79v                      1/1     Running             1 (13m ago)      3h54m
example-nodejs-v1-6d89c89657-ss5fm                       0/1     ImagePullBackOff    0                5d12h

That is kind of a waste of resources for active development.

Howver, adding “-prune” to the release step will get rid of the former unused releases.

      - name: install waypoint
        run: |
          # go to src dir
          cd kubernetes/nodejs 
          # sed -i "s/encoded_auth =.*/encoded_auth = \"$\"/g" waypoint.hcl
          set +x
          waypoint config set HARBORCRED=$
          waypoint config set -runner HARBORCRED=$
          set -x
          waypoint init
          waypoint build -local=true
          waypoint deploy -local=true
          waypoint release -prune

And we can see it went from 3 versions running

$ kubectl get pods | grep ^example-nodejs3
example-nodejs3-v2-758ddc49dc-stlhv                      1/1     Running             1 (17m ago)      7h43m
example-nodejs3-v3-854dd4d55f-cr79v                      1/1     Running             1 (17m ago)      3h59m
example-nodejs3-v5-7b57f688b9-5t6zz                      0/1     Running             0                27s

to just the one

$ kubectl get pods | grep ^example-nodejs3
example-nodejs3-v5-7b57f688b9-5t6zz                      1/1     Running             0                2m21s

Auth tokens

If you are on the computer that installed waypoint, you can get fresh tokens easily

/content/images/2022/03/waypoint-35.png

However, if you lose the token, you are in a real pickle.

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint user token
! failed to create client: no server connection configuration found

builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ WAYPOINT_SERVER_TLS_SKIP_VERIFY=1 WAYPOINT_SERVER_ADDR=192.168.1.12:9701 WAYPOINT_SERVER_TLS=true waypoint user token
! Authorization token is not supplied

There appears to be no other options in the docs. The only thread I could find showed the user just reinstalling when it was lost.

So guard that token as if it’s lost and you’ve not added other users, you are in a stuck spot.

I should point out you can invite others which generates a pre-authed link with a token

/content/images/2022/03/waypoint-36.png

However, it should be worth noting that there aren’t really different types of users. Either you are in Waypoint or not. So I could not use that user, for instance, to reset the token for another. Nor can I create a “read-only” type of user either.

Summary

I really wanted to find a path to liking Waypoint. I watched a year old DevOps Toolkit Review of Waypoint to try and see if I missed something. In his review, Victor found errors and could not quite see a value.

While some of his points have counters; like the too many current releases running (use “-prune” to remove old ones), the fact is that he raises the real underlying question: why would we want to deploy to Kubernetes in a way different than we might promote? That is, if we use Helm for releasing, why would use anything else for deploying to lower environments?

In the last example we worked through, we explored using Github Actions to build our application but then Waypoint to track the releases. It was nice to see our build logs archived for past releases outside our CICD tool (Github).

/content/images/2022/03/waypoint-30.png

However, our build log was just the buildpack steps from Waypoint. This constitutes just a small portion of the overall workflow.

/content/images/2022/03/waypoint-31.png

If I wish to track externally, I might use something like Datadog or SumoLogic which could capture both logs

/content/images/2022/03/waypoint-32.png

pipeline metrics

/content/images/2022/03/waypoint-33.png (Datadog)

/content/images/2022/01/sumologic-110.png (Sumo Logic)

/content/images/2022/01/sumologic-112.png (Sumo Logic)

and tests.

/content/images/2022/03/waypoint-34.png

The last item there brings up the other point Victor raised in his review; where are the tests?

The waypoint tool clearly covers build, deploy and release, but completely absent is testing. This issue has been raised by others.

Perhaps one might argue that Waypoint isn’t for Kubernetes, it’s for other deployment models, If that was the case, I would lean towards using Nomad or using regular release pipelines including ones such as Azure DevOps Release pipelines. AzDO Release Pipelines can have complex logic with forked stages, approval gates and deployment groups. All things that would be hard if not impossible in Waypoint.

Waypoint lacks user administration, access levels and built-in notification systems. In all cases, they would want you to use a webhook or plugin. Presently, there are no notifcation plugins.

So again, I ask, what is the point of Waypoint? I just cannot see where it fits in the picture outside of adding a rudimentary CI tool to the Hashi suite. With it’s general lack of stability and full stack coverage (namely tests), it doesn’t seem it’s quite ready for widespread adoption at this time. I find it telling, as well, that the tool itself uses Github Actions and Circle CI to build and release (instead of Waypoint).

hashi waypoint

Have something to add? Feedback? Try our new forums

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes