Published: Mar 22, 2022 by Isaac Johnson
The last time we looked at Hashi Waypoint it had just been released. While interesting, it was somewhat strange to see a new CI/CD tool come to the game, especially since Hashi tends to put out systems that integrate with CI/CD tooling.
It has been a while since release and I figured now was a good time to revisit Waypoint. Has it matured? In comparing tools like Jenkins, Azure Devops and Argo Workflows, where does it fit (if at all)?
Today we’ll cover setting up waypoint locally as well as on Kubernetes (both K3s and AKS). We’ll setup ingress, projects and experiment with a NodeJS Kubernetes based project. Lastly, we’ll tie it together by trying to build remotely and integrating with Github Actions for a full continuous integration and deployment setup.
In our examples below, we’ll tie to a local private Harbor container registry so we can explore the use of registry secrets and private registry authentication credentials.
Setup
The last time we tried Waypoint it was back in October of 2020 and the recommend approach was to use apt to install. Today they have a helm chart.
I’ll be installing this into a cluster that is wrapping up an update to the latest k3s (1.23.x):
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
isaac-macbookair Ready control-plane,master 422d v1.23.3+k3s1
anna-macbookair Ready <none> 309d v1.23.3+k3s1
builder-hp-elitebook-850-g2 Ready <none> 112d v1.21.5+k3s1
builder-macbookpro2 Ready <none> 49d v1.21.5+k3s1
builder-hp-elitebook-850-g1 Ready <none> 117d v1.23.3+k3s1
hp-hp-elitebook-850-g2 Ready <none> 137d v1.23.3+k3s1
isaac-macbookpro Ready <none> 421d v1.23.3+k3s1
Helm
First, we need to add the helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
Then we can install from the chart
$ helm install waypoint hashicorp/waypoint
NAME: waypoint
LAST DEPLOYED: Mon Feb 21 17:51:15 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Waypoint!
Waypoint will take a few minutes to bootstrap. Once it is bootstrapped, you
can log in to the local CLI by running the following command. If this command
fails, please wait a few more minutes and try again.
$ waypoint login -from-kubernetes
Once logged in, you can use the "waypoint ui" command to open the web interface.
Alternately, you can visit the address of the load balancer service configured.
If anything fails to configure properly you can uninstall this release:
$ helm uninstall waypoint
By default, it wants to setup the service as a loadbalancer which rather vomits on my cluster. I moved it to ClusterIP and it solved the Init issues.
$ helm install waypoint --set ui.service.type=ClusterIP hashicorp/waypoint
Once the pods come up, we can fetch the login token
$ kubectl get secret waypoint-server-token -o jsonpath="{.data.token}" &
& echo
SGV5LCBob3cgaXMgaXQgZ29pbmcuICBTdXJlIHdpbGwgYmUgbmljZSB3aGVuIHdpbnRlciBpcyBkb25lLiBMb29raW5nIGZvcndhcmQgdG8gZ2V0dGluZyBvdXQgYW5kIGRvaW5nIHNvbWUgY2FtcGluZy4K
$ kubectl port-forward svc/waypoint-ui 8080:443
Forwarding from 127.0.0.1:8080 -> 9702
Forwarding from [::1]:8080 -> 9702
Handling connection for 8080
which loaded this rather blank page
clicking detail didn’t work too hot
(you can get this error if you use the wrong token)
I’ll setup Ingress while I’m at it
$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-waypoint.json
{
"ChangeInfo": {
"Id": "/change/C08858852NHKMJ93HSTJI",
"Status": "PENDING",
"SubmittedAt": "2022-02-21T23:55:15.688Z",
"Comment": "CREATE waypoint fb.s A record "
}
}
$ cat ingress-waypoint.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
ingress.kubernetes.io/proxy-body-size: 2048m
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/auth-realm: Authentication Required - waypoint
nginx.ingress.kubernetes.io/auth-secret: argocd-basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/proxy-body-size: 2048m
nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.org/client-max-body-size: 2048m
creationTimestamp: "2022-02-07T12:14:58Z"
generation: 1
labels:
app: waypoint
release: waypointrelease
name: waypoint-with-auth
namespace: default
resourceVersion: "200201296"
uid: f4196f14-9ace-4bcf-9a2f-60a8dac25bb3
spec:
ingressClassName: nginx
rules:$ cat ingress-waypoint.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
ingress.kubernetes.io/proxy-body-size: 2048m
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/auth-realm: Authentication Required - waypoint
nginx.ingress.kubernetes.io/auth-secret: argocd-basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/proxy-body-size: 2048m
nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.org/client-max-body-size: 2048m
creationTimestamp: "2022-02-07T12:14:58Z"
generation: 1
labels:
app: waypoint
release: waypointrelease
name: waypoint-with-auth
namespace: default
resourceVersion: "200201296"
uid: f4196f14-9ace-4bcf-9a2f-60a8dac25bb3
spec:
ingressClassName: nginx
rules:
- host: waypoint.freshbrewed.science
http:
paths:
- backend:
service:
name: waypointrelease
port:
number: 8080
path: /
pathType: Prefix
tls:
- hosts:
- waypoint.freshbrewed.science
$ kubectl apply -f ingress-waypoint.yml
ingress.networking.k8s.io/waypoint-with-auth created
Local Install
If we wanted to install outside of Kubernetes, we could use instructions such as these:
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install waypoint
We can use these steps to setup the local waypoint binary.
We can also use it to run a local dockerized server:
waypoint install -platform=docker -accept-tos -- -vvv -disable-ui
AKS
Let’s use Azure Kubernetes Service next to show a rather clean full K8s deployment.
Create the Resource Group
$ az group create -n idjaks45 --location centralus
{
"id": "/subscriptions/8defc61d-657a-453d-a6ff-cb9f91289a61/resourceGroups/idjaks45",
"location": "centralus",
"managedBy": null,
"name": "idjaks45",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}
Create (or use) a Service Principal. Unlike the past, it now seems that create-for-rbac really demands a proper AD compliant name. You can use the default from AAD on your tenant; e.g.
$ az ad sp create-for-rbac -n idjaks45sp.princessking.onmicrosoft.com --skip-assignment --output json > my_sp.json
Changing "idjaks45sp.princessking.onmicrosoft.com" to a valid URI of "http://idjaks45sp.princessking.onmicrosoft.com", which is the required format used for service principal names
$ export SP_PASS=`cat my_sp.json | jq -r .password`
$ export SP_ID=`cat my_sp.json | jq -r .appId`
Then create the AKS cluster:
$ az aks create --resource-group idjaks45 --name idjaks45 --location centralus --node-count 2 --enable-cluster-autoscaler --min-count 2 --max-count 3 --generate-ssh-keys --network-plugin azure --network-policy azure --service-principal $SP_ID --client-secret $SP_PASS
- Running ..
$ az aks list -o table
Name Location ResourceGroup KubernetesVersion ProvisioningState Fqdn
-------- ---------- --------------- ------------------- ------------------- ---------------------------------------------------------
idjaks45 centralus idjaks45 1.21.9 Succeeded idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io
Now get credentials and verify nodes are running
$ az aks get-credentials -n idjaks45 -g idjaks45 --admin
Merged "idjaks45-admin" as current context in /home/builder/.kube/config
builder@DESKTOP-72D2D9T:~/Workspaces/jekyll-blog$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-70322854-vmss000000 Ready agent 71s v1.21.9
aks-nodepool1-70322854-vmss000001 Ready agent 83s v1.21.9
We can now install Waypoint via Helm
$ !1372
helm install waypoint hashicorp/waypoint
NAME: waypoint
LAST DEPLOYED: Mon Feb 21 19:47:09 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Waypoint!
Waypoint will take a few minutes to bootstrap. Once it is bootstrapped, you
can log in to the local CLI by running the following command. If this command
fails, please wait a few more minutes and try again.
$ waypoint login -from-kubernetes
Once logged in, you can use the "waypoint ui" command to open the web interface.
Alternately, you can visit the address of the load balancer service configured.
If anything fails to configure properly you can uninstall this release:
$ helm uninstall waypoint
We’ll check the pods and service to see if it’s up
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
waypoint-bootstrap-w4l6j 1/1 Running 0 45s
waypoint-runner-59dbddf4f4-d2jxn 0/1 Init:0/1 0 45s
waypoint-server-0 0/1 ContainerCreating 0 45s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
waypoint-bootstrap-w4l6j 1/1 Running 0 106s
waypoint-runner-59dbddf4f4-d2jxn 0/1 Init:0/1 0 106s
waypoint-server-0 1/1 Running 0 106s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP
8m30s
waypoint-server ClusterIP None <none> 9702/TCP,9701/TCP
4m43s
waypoint-ui LoadBalancer 10.0.87.68 13.89.111.244 80:32259/TCP,443:31145/TCP,9701:30061/TCP,9702:30778/TCP 4m43s
Next, we need a login token to get started
$ !1421
kubectl get secret waypoint-server-token -o jsonpath="{.data.token}" && echo
M0s0d1FVZEgxZGZIYXlwM3g1UFFLSGtoYUw2S3BhVDhFRFRSdnM5Y1BTWm1lUndwOTE1NldTRU16dWd3QVNBb2lwSkdlTVkyY0Q4bnp0TFNEQmRYRlc0dTN4Y2FHbjNVcVZHeDVSUFQ1N3JtNjFvSEVpTFNBYmk5NXYxbjdLYlZBRjJ6TUNYY0FLdUw5cXhD
With the Token we should be able to login, however I see some errors.
Moving back to the demo code, Let’s try to waypoint up now…
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
│ [exporter] Adding layer 'process-types'
│ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
│ [exporter] Adding label 'io.buildpacks.build.metadata'
│ [exporter] Adding label 'io.buildpacks.project.metadata'
│ [exporter] Setting default process type 'web'
│ [exporter] Saving example-nodejs...
│ [exporter] *** Images (ea4b1f8a4f8f):
│ [exporter] example-nodejs
│ [exporter] Adding cache layer 'heroku/nodejs-engine:nodejs'
│ [exporter] Adding cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image
Generated new Docker image: example-nodejs:latest
» Deploying example-nodejs...
✓ Setting up network...
✓ Starting container
✓ App deployed as container: example-nodejs-01FWFJS3NVZMGF8MTAX6KCMRZM
✓ Docker image "example-nodejs:latest" up to date!
✓ Finished building report for Docker platform
✓ Finished building report for Docker network resource
✓ Finished building report for Docker container resource
✓ All 2 resources are reporting READY
» Releasing example-nodejs...
No release phase specified, skipping...
The deploy was successful!
The release did not provide a URL and the URL service is disabled on the
server, so no further URL information can be automatically provided. If
this is unexpected, please ensure the Waypoint server has both the URL service
enabled and advertise addresses set.
We check to see if the image was created (locally):
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
773e8c61480f example-nodejs:latest "/waypoint-entrypoin…" 29 seconds ago Up 27 seconds 0.0.0.0:54723->3000/tcp example-nodejs-01FWFJS3NVZMGF8MTAX6KCMRZM
This rather annoyed me. I really don’t want to build from my local box.
The point, in my opinion, of a CI system is to move the compilation from the local workstation to a remote service. I don’t want to depend on a local user’s ability to pull and create images.
My first step is to try and use the waypoint server itself. Perhaps it has Docker-in-Docker abilities? That could work.
We’ll exec into the pod, clone the repo and try…
$ kubectl exec -it waypoint-server-0 -- /bin/sh
/ $ git clone https://github.com/hashicorp/waypoint-examples.git
fatal: could not create work tree dir 'waypoint-examples': Permission denied
/ $ cd /tmp
/tmp $ git clone https://github.com/hashicorp/waypoint-examples.git
Cloning into 'waypoint-examples'...
remote: Enumerating objects: 1771, done.
remote: Counting objects: 100% (601/601), done.
remote: Compressing objects: 100% (348/348), done.
remote: Total 1771 (delta 353), reused 444 (delta 233), pack-reused 1170
Receiving objects: 100% (1771/1771), 1.33 MiB | 5.90 MiB/s, done.
Resolving deltas: 100% (791/791), done.
/tmp $ cd waypoint-examples/docker/
/tmp/waypoint-examples/docker $ cd nodejs/
/tmp/waypoint-examples/docker/nodejs $ waypoint init
+ Configuration file appears valid
+ Local mode initialized successfully
+ Project "example-nodejs" and all apps are registered with the server.
Project initialized!
You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
/tmp/waypoint-examples/docker/nodejs $ waypoint up
Performing operation locally
» Building example-nodejs...
WARNING: Docker daemon appears unavailable. The 'pack' builder requires access to a Docker daemon. Pack does not support
WARNING: Docker daemon appears unavailable. The 'pack' builder requires access to a Docker daemon. Pack does not support
dockerless builds. We will still attempt to run the build but it will likely fail. If you are running this build
locally, please install Docker. If you are running this build remotely (in a Waypoint runner), the runner must be
configured to have access to the Docker daemon.
! Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
That didn’t work, no DinD.
Doing some more reading, perhaps I need to create a “context” for waypoint to use. Here I’ll use the external service IP for the waypoint-ui as exposed by AKS (13.89.111.244)
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context create -server-addr=13.89.111.244:9701 -server-auth-token=`kubectl get secret waypoint-server-token -o jsonpath="{.data.token}"` -server-tls -set-default my-server
Context "my-server" created.
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context verify
❌ Connecting with context "my-server"...
! Error connecting with context "my-server": context deadline exceeded
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context create -server-addr=13.89.111.244:9701 -server-auth-token=`kubectl get secret waypoint-server-token -o jsonpath="{.data.token}"` -set-default my-server
Context "my-server" created.
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint context verify
❌ Connecting with context "my-server"...
! Error connecting with context "my-server": context deadline exceeded
Let me try to uninstall and try again
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
waypoint default 1 2022-02-21 19:47:09.3355676 -0600 CST deployed waypoint-0.1.4 0.7.1
$ helm delete waypoint
release "waypoint" uninstalled
Trying to install the first time found some abandoned services (likely from the failed LB install)
$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
❌ Creating Kubernetes resources...
! Error creating service services "waypoint" already exists
! Error installing server into kubernetes: services "waypoint" already exists
$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
❌ Creating Kubernetes resources...
! Error creating service services "waypoint" already exists
! Error installing server into kubernetes: services "waypoint" already exists
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 45m
waypoint LoadBalancer 10.0.135.226 52.154.171.243 9701:32177/TCP,9702:31556/TCP 40s
$ helm uninstall waypoint
Error: uninstall: Release not loaded: waypoint: release: not found
Here I just manually deleted the service
$ kubectl delete svc waypoint
service "waypoint" deleted
I’ll try again by installing with the waypoint binary instead of the helm chart
$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!
The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1645496978".
Use the "waypoint context" CLI to manage CLI contexts.
The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".
To launch and authenticate into the Web UI, run:
waypoint ui -authenticate
Advertise Address: 13.89.111.244:9701
Web UI Address: https://13.89.111.244:9702
This worked much better. Here we can see, by default, the waypoint binary installs the service in Kubernetes and also set’s a context for us.
$ waypoint context list
| NAME | PLATFORM | SERVER ADDRESS
----+--------------------+------------+---------------------
* | install-1645496978 | kubernetes | 13.89.111.244:9701
| my-server | n/a | 13.89.111.244:9701
We can verify the context
$ waypoint context verify
✓ Context "install-1645496978" connected successfully.
Now, let’s go back to our demo code and try the init and build.
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint init
✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "example-nodejs" and all apps are registered with the server.
Project initialized!
You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
Test build
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-examples/docker/nodejs$ waypoint build
» Building example-nodejs...
Performing operation locally
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
⠇ Building image
│ 2022/02/21 20:37:05.319337 DEBUG: Pulling image index.docker.io/heroku/buildpac
│ ks:20
However, as we can see, it is building locally again…
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: gcr.io/buildpacks/builder:v1
✓ Creating pack client
✓ Building image
│ [exporter] Adding layer 'config'
│ [exporter] Adding layer 'process-types'
│ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
│ [exporter] Adding label 'io.buildpacks.build.metadata'
│ [exporter] Adding label 'io.buildpacks.project.metadata'
│ [exporter] Setting default process type 'web'
│ [exporter] Saving helloworld-go...
│ [exporter] *** Images (b4e6560f1555):
│ [exporter] helloworld-go
│ [exporter] Adding cache layer 'google.go.runtime:go'
✓ Injecting entrypoint binary to image
Generated new Docker image: helloworld-go:latest
✓ Tagging Docker image: helloworld-go:latest => idjohnson/helloworld-go:latest
✓ Pushing Docker image...
│ a4d53325badd: Pushed
│ 0b6b00fa3492: Pushed
│ e32996f7b4b0: Pushed
│ d48e235b496a: Pushed
│ 1d7b24ca7af7: Pushed
│ 84ff92691f90: Pushed
│ 9d15ff164ac7: Pushed
│ 158840bd8028: Pushed
│ latest: digest: sha256:9a6dd98c1837a5181fcb699440493a8e0c78782f64a147d460c631bad
│ 9668bc2 size: 2623
✓ Docker image pushed: idjohnson/helloworld-go:latest
» Deploying helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Creating deployment...
✓ Expected "http" port "8080" for app "helloworld-go-v1"
✓ Deployment successfully rolled out!
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes deployment resource
✓ Deployment "helloworld-go-v1" is reporting ready!
» Releasing helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Creating service...
✓ Service is ready!
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource
The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."
Release URL: http://13.86.33.130:8080
Deployment URL: https://formerly-social-bobcat--v1.waypoint.run
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-to-k8s$
It does work to deploy. If we test the release URL, we can see the NodeJS app running.
We can also login to the UI to see the deployment
Then dig into details on the deployment
We can look at release logs
and the resources created
We can exec into the pod to run commands if we wanted to
While it was built locally, the build logs were pushed to waypoint
Testing changes
Let’s push a change
$ git diff
diff --git a/main.go b/main.go
index c811ada..e8e266c 100644
--- a/main.go
+++ b/main.go
@@ -10,7 +10,7 @@ type Server struct{}
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json")
- w.Write([]byte(`{"message": "Hello World!"}`))
+ w.Write([]byte(`{"message": "Hello World Again!"}`))
}
func main() {
diff --git a/waypoint.hcl b/waypoint.hcl
index 883f370..21b113c 100644
--- a/waypoint.hcl
+++ b/waypoint.hcl
@@ -7,7 +7,7 @@ app "helloworld-go" {
}
registry {
use "docker" {
- image = "ronakbanka/helloworld-go"
+ image = "idjohnson/helloworld-go"
tag = "latest"
local = false
}
We will waypoint up
to build and release again
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the
Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: gcr.io/buildpacks/builder:v1
✓ Creating pack client
✓ Building image
│ [exporter] Reusing layer 'config'
│ [exporter] Reusing layer 'process-types'
│ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
│ [exporter] Adding label 'io.buildpacks.build.metadata'
│ [exporter] Adding label 'io.buildpacks.project.metadata'
│ [exporter] Setting default process type 'web'
│ [exporter] Saving helloworld-go...
│ [exporter] *** Images (0429d4c10f67):
│ [exporter] helloworld-go
│ [exporter] Adding cache layer 'google.go.runtime:go'
✓ Injecting entrypoint binary to image
Generated new Docker image: helloworld-go:latest
✓ Tagging Docker image: helloworld-go:latest => idjohnson/helloworld-go:latest
✓ Pushing Docker image...
│ a4d53325badd: Layer already exists
│ f1f11d09993c: Pushed
│ ea8974eb62ff: Pushed
│ d48e235b496a: Layer already exists
│ 1d7b24ca7af7: Layer already exists
│ 84ff92691f90: Layer already exists
│ 9d15ff164ac7: Layer already exists
│ 158840bd8028: Layer already exists
│ latest: digest: sha256:8ac78583bb5fbf78de423795f88f1367f3c1cc258cd797dec254c9bc2
│ f34af83 size: 2623
✓ Docker image pushed: idjohnson/helloworld-go:latest
» Deploying helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Creating deployment...
✓ Expected "http" port "8080" for app "helloworld-go-v2"
✓ Deployment successfully rolled out!
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes deployment resource
✓ Deployment "helloworld-go-v2" is reporting ready!
» Releasing helloworld-go...
✓ Kubernetes client connected to https://idjaks45-idjaks45-8defc6-c18d1418.hcp.centralus.azmk8s.io:443 with namespace
default
✓ Updating existing service...
✓ Service is ready!
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource
The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."
Release URL: http://13.86.33.130:8080
Deployment URL: https://formerly-social-bobcat--v2.waypoint.run
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-to-k8s$
and upon deploy, we see it deployed (to the same URL)
We can check the build logs
Say that we found an issue, we could move back to release 1 easily
$ waypoint release -repeat 1
» Releasing helloworld-go...
Performing operation locally
Performing operation locally
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource
Release URL: http://13.86.33.130:8080
However, I should not, that did not change the service http://13.86.33.130:8080/
I can see that both versions are actually running:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
helloworld-go-v1-66cb794895-25tmf 1/1 Running 0 68m
helloworld-go-v2-5c7f767d67-s854p 1/1 Running 0 7m13s
waypoint-runner-6695b69bc7-wvv7p 1/1 Running 0 89m
waypoint-server-0 1/1 Running 0 91m
That said, the svc endpoint points to v2
$ kubectl get svc helloworld-go -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2022-02-22T02:52:23Z"
finalizers:
- service.kubernetes.io/load-balancer-cleanup
name: helloworld-go
namespace: default
resourceVersion: "27004"
uid: b46c0c5e-2732-4e66-a196-34eedd51333b
spec:
clusterIP: 10.0.172.204
clusterIPs:
- 10.0.172.204
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- nodePort: 31024
port: 8080
protocol: TCP
targetPort: http
selector:
name: helloworld-go-v2
waypoint.hashicorp.com/id: 01FWFS0BTJ8X971T2JMCXM6KB9
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 13.86.33.130
builder@DESKTOP-72D2D9T:~/Workspaces/waypoint-to-k8s$
But we can go to it from the built-in url https://formerly-social-bobcat--v1.waypoint.run/
Try again locally
We’ll follow this guide again to install the binary:
$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install waypoint
Remove the helm install (if installed via helm)
$ helm delete waypoint
release "waypoint" uninstalled
Now install using the waypoint binary:
$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!
The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1646140543".
Use the "waypoint context" CLI to manage CLI contexts.
The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".
To launch and authenticate into the Web UI, run:
waypoint ui -authenticate
Advertise Address: 192.168.1.12:9701
Web UI Address: https://192.168.1.12:9702
We see that Waypoint is indeed up at that address
We need a token to auth. you can use waypoint ui -authenticate
or just use the following to fetch it:
$ kubectl get secret `kubectl get secret | grep waypoint | sed 's/ .*//' | tr -d '\n'` -o json
| jq -r .data.token
ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltMVZUakI0VTBOUVdtWmtNblJvTkc5UE9GQkRXRlZ3ZFdWZlVVbGZaWEUyZFdoZmVFczRlRGRYZG04aWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbmRoZVhCdmFXNTBMWEoxYm01bGNpMTBiMnRsYmkxNmNITTJaeUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG01aGJXVWlPaUozWVhsd2IybHVkQzF5ZFc1dVpYSWlMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObGNuWnBZMlV0WVdOamIzVnVkQzUxYVdRaU9pSXhaREJqTmpjNVlTMWpOakE0TFRRM1lqSXRPVFJrTkMxa1pqRTNPV05tT1dObE5EUWlMQ0p6ZFdJaU9pSnplWE4wWlcwNmMyVnlkbWxqWldGalkyOTFiblE2WkdWbVlYVnNkRHAzWVhsd2IybHVkQzF5ZFc1dVpYSWlmUS5HRVcycHBTLU5CbjE1ZWMxMXZFTTRJU3J5SWVJWmE2N1NuRzFzUklsclEtSldBTkZsYjkzekdNY2tKOE9Eckt3NGxKWUhhLXViZDg0ZkR4LXhvX3RnMlBtT0lXVG9JMFFuVENEZ2tfNi1DNFlXOFdGb0ZSbEtxNXJOVGFmUTdQVmM5enNxM1dLeGhPems2dUR4a3ZVVGxfQWVJU1FuMWhxQVliR3plbUJzTTlXYnJfcVpuSjJvYU5FSTg1MXpoMmtJbzZvQXBjaVl0d1BWZFUyanQyTFdtZURBemRBci1pWWhHblh1azRabDIzTVE0LWVPZ1N5cFJfa3FUOEZZOVhBak9vcDlvTFRRRGNGdGhFZ2NKek1STEdaY01CeHp6ZllrWE42aXBZdEZhUkNDR1FaaGk5S1NGMFJ0T1RRaDE3ZG5XSUJLa3RpNTBEQ2ZkMVJ2R2lPN0E=
Because we used the CLI to install, we see the context already saved:
$ waypoint context list
| NAME | PLATFORM | SERVER ADDRESS
----+--------------------+------------+--------------------
* | install-1646140543 | kubernetes | 192.168.1.12:9701
Let’s start with what we know works. We’ll use the NodeJS example from Hashi’s examples:
$ git clone https://github.com/hashicorp/waypoint-examples.git
$ cd waypoint-examples/kubernetes/nodejs
We now just need to waypoint init
to set up the project
$ waypoint init
✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "example-nodejs" and all apps are registered with the server.
Project initialized!
You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
builder@DESKTOP-QADGF36:~/Workspaces/waypoint-examples/kubernetes/nodejs$
Using waypoint up
it will build locally and then push. Since I’m not using Docker Desktop, I’m interested to see if it can push to a Container Registry of my choosing.
$ waypoint up
Performing operation locally
» Building example-nodejs...
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64".
We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
│ [exporter] Adding layer 'process-types'
│ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
│ [exporter] Adding label 'io.buildpacks.build.metadata'
│ [exporter] Adding label 'io.buildpacks.project.metadata'
│ [exporter] Setting default process type 'web'
│ [exporter] Saving example-nodejs...
│ [exporter] *** Images (e6a2eac76957):
│ [exporter] example-nodejs
│ [exporter] Adding cache layer 'heroku/nodejs-engine:nodejs'
│ [exporter] Adding cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image
Generated new Docker image: example-nodejs:latest
✓ Tagging Docker image: example-nodejs:latest => example-nodejs:1
✓ Docker image pushed: example-nodejs:1
» Deploying example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "example-nodejs-v1"
⠸ Waiting on deployment to become available: requested=1 running=1 ready=0
Detected pods having an issue starting - Pod unable to access Docker image: rpc error: code = Unknown desc = failed to pull and unpack image
"docker.io/library/example-nodejs:1": failed to resolve reference "docker.io/library/example-nodejs:1": pull access denied, repository does not
exist or may require authorization: server message: insufficient_scope: authorization failed
That is fair, I can’t push to the global example-nodejs (docker.io/library/example-nodejs).
Using local private container registry
We need to create a Registry credential:
$ cat /home/builder/harborAuth.json
{
"username": "isaac",
"password": "MYPASSWORD",
"email": "isaac.johnson@gmail.com"
}
Now we just change the waypoint.hcl to use the path (and set the full path to the image in the CR)
$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index e2d979e..6920de2 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -10,9 +10,9 @@ app "example-nodejs" {
use "pack" {}
registry {
use "docker" {
- image = "example-nodejs"
+ image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = "1"
- local = true
+ encoded_auth = filebase64("/home/builder/harborAuth.json")
}
}
}
This time we got further, but failed to pull the built image
$ waypoint up
Performing operation locally
» Building example-nodejs...
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64".
We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
│ [exporter] Reusing layer 'process-types'
│ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
│ [exporter] Adding label 'io.buildpacks.build.metadata'
│ [exporter] Adding label 'io.buildpacks.project.metadata'
│ [exporter] Setting default process type 'web'
│ [exporter] Saving example-nodejs...
│ [exporter] *** Images (7c9b873f381b):
│ [exporter] example-nodejs
│ [exporter] Reusing cache layer 'heroku/nodejs-engine:nodejs'
│ [exporter] Reusing cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image
Generated new Docker image: example-nodejs:latest
✓ Tagging Docker image: example-nodejs:latest => harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
✓ Pushing Docker image...
│ ab2b7fb6b8c0: Pushed
│ a2e78a35a946: Pushed
│ 6d3d39532668: Pushed
│ 438aa9470562: Pushed
│ 0724d7609fc4: Pushed
│ fd10542322e5: Pushed
│ ed348f24d8dd: Pushed
│ 36ffdceb4c77: Pushed
│ 1: digest: sha256:7b9409ab5a29bea3fa3599afc9a9a697b63f530cefbf73593147d537f6f414
│ 77 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
» Deploying example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "example-nodejs-v2"
⠧ Waiting on deployment to become available: requested=1 running=1 ready=0
Detected pods having an issue starting - Pod unable to access Docker image: rpc error: code = Unknown desc = failed to pull and unpack image
"harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1": failed to resolve reference
"harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1": pulling from host harbor.freshbrewed.science failed with status code
[manifests 1]: 401 Unauthorized
Adding image pull secret to waypoint HCL for private registries
We need to make note of the imagepull secret we have to pull the private container in Kubernetes. Add an “image_secret” to the deploy Kubernetes block as such:
$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index e2d979e..43c6c3b 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -10,15 +10,16 @@ app "example-nodejs" {
use "pack" {}
registry {
use "docker" {
- image = "example-nodejs"
+ image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = "1"
- local = true
+ encoded_auth = filebase64("/home/builder/harborAuth.json")
}
}
}
deploy {
use "kubernetes" {
+ image_secret = "myharborreg"
probe_path = "/"
}
}
Validation
Now this works:
$ waypoint up
Performing operation locally
» Building example-nodejs...
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64".
We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
│ [exporter] Reusing layer 'process-types'
│ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
│ [exporter] Adding label 'io.buildpacks.build.metadata'
│ [exporter] Adding label 'io.buildpacks.project.metadata'
│ [exporter] Setting default process type 'web'
│ [exporter] Saving example-nodejs...
│ [exporter] *** Images (fabfa63a64fb):
│ [exporter] example-nodejs
│ [exporter] Reusing cache layer 'heroku/nodejs-engine:nodejs'
│ [exporter] Reusing cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image
Generated new Docker image: example-nodejs:latest
✓ Tagging Docker image: example-nodejs:latest => harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
✓ Pushing Docker image...
│ ab2b7fb6b8c0: Layer already exists
│ c04687436ccf: Pushed
│ 6d3d39532668: Layer already exists
│ 438aa9470562: Layer already exists
│ 0724d7609fc4: Layer already exists
│ fd10542322e5: Layer already exists
│ ed348f24d8dd: Layer already exists
│ 36ffdceb4c77: Layer already exists
│ 1: digest: sha256:58a9b3a5eec2a8c4bf15a2879638838211c41ce24cdb2401f5d1bfe8a3ee0a
│ 77 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
» Deploying example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "example-nodejs-v3"
✓ Deployment successfully rolled out!
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes deployment resource
✓ Deployment "example-nodejs-v3" is reporting ready!
» Releasing example-nodejs...
✓ Kubernetes client connected to https://192.168.1.77:6443 with namespace default
✓ Creating service...
✓ Service is ready!
» Pruning old deployments...
Deployment: 01FX2V06WWVJT218A09ZGRS1K3 (v1)
✓ Finished building report for Kubernetes platform
✓ Finished building report for Kubernetes service resource
The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."
Release URL: http://192.168.1.214:3000
Deployment URL: https://blatantly-smiling-goldfish--v3.waypoint.run
The other URL works just as well (with valid TLS)
I now see the app in projects (I did end up needing to relogin with waypoint ui -authenticate
)
And from there I can get build details
Let’s make a small change to the webpage we could see
$ git diff views/pages/index.ejs
diff --git a/kubernetes/nodejs/views/pages/index.ejs b/kubernetes/nodejs/views/pages/index.ejs
index f6c45be..65f56ff 100644
--- a/kubernetes/nodejs/views/pages/index.ejs
+++ b/kubernetes/nodejs/views/pages/index.ejs
@@ -17,7 +17,8 @@
</div>
<h1>This Node.js app was deployed with Waypoint.</h1>
<p>
- Try making a change to this text locally and run <code>waypoint up</code> again to see it.
+ Try making a change to this text locally and run <code>waypoint up</code> again to see it.
+ Hey, I made a change. wow.
</p>
<p>
Read the <a href="https://waypointproject.io/docs">documentation</a> for more about Waypoint.
Also, I would really like to do this on a remote runner.
I need to see CI to believe it.
In order to really use a hosted node, we must define the GIT repo from which to pull. Otherwise, how would Waypoint know where to get sources (once it’s invoked) to docker build?
To do this, I first forked their examples repo: https://github.com/idjohnson/waypoint-examples.
Then I cloned and changed the waypoint.hcl file. The key here is to set the “runner” block. By default, “runner” is disabled. That is to say, by default, Waypoint assumes you want to build locally.
We can easily fix both these things by defining the runner block and the data_source:
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git diff
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index e2d979e..21437d7 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -1,5 +1,14 @@
project = "example-nodejs"
+runner {
+ enabled = true
+
+ data_source "git" {
+ url = "https://github.com/idjohnson/waypoint-examples.git"
+ path = "kubernetes/nodejs"
+ }
+}
+
app "example-nodejs" {
labels = {
"service" = "example-nodejs",
@@ -10,15 +19,16 @@ app "example-nodejs" {
use "pack" {}
registry {
use "docker" {
- image = "example-nodejs"
+ image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = "1"
- local = true
+ encoded_auth = filebase64("/home/builder/harborAuth.json")
}
}
}
deploy {
use "kubernetes" {
+ image_secret = "myharborreg"
probe_path = "/"
}
}
Handling Auth Secrets for Private Container Registries in Waypoint utilizing Remote build agents
okay, that was a mouthful…
Now, clearly, I am not going to check in my password unencrypted to push to my CR.
So we need to create a runner secret we could use.
$ cat /home/builder/harborAuth.json | base64 -w 0
bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=
$ waypoint config set -runner HARBORCRED=bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=
And we can verify it took by asking Waypoint for our configs:
$ waypoint config get -runner
SCOPE | NAME | VALUE
| WORKSPACE | LABELS
----------+------------+------------------------------------------------------------------------------------------------------------------------
project | HARBORCRED | bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo= | |
We can use that cred by referring to it in the waypoint.hcl file. If you use AzDO, the variable syntax should be familiar:
$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index 21437d7..2ad0cae 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -21,7 +21,7 @@ app "example-nodejs" {
use "docker" {
image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = "1"
- encoded_auth = filebase64("/home/builder/harborAuth.json")
+ encoded_auth = ${HARBORCRED}
}
}
}
I ended up needing to change the project name (since Waypoint had a different configuration and was not hearing a new datasource):
$ cat waypoint.hcl
project = "example-nodejs2"
runner {
enabled = true
data_source "git" {
url = "https://github.com/idjohnson/waypoint-examples.git"
path = "kubernetes/nodejs"
}
}
app "example-nodejs2" {
labels = {
"service" = "example-nodejs",
"env" = "dev"
}
build {
use "pack" {}
registry {
use "docker" {
image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = "1"
encoded_auth = "${HARBORCRED}"
}
}
}
deploy {
use "kubernetes" {
image_secret = "myharborreg"
probe_path = "/"
}
}
release {
use "kubernetes" {
// Sets up a load balancer to access released application
load_balancer = true
port = 3000
}
}
}
Now we can say waypoint up, and be explicit about not running locally with the local false flag.
However, I got errors from the remote server
$ waypoint up -local=false
! error reading from server: EOF
In fact, any attempt to build fell down
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=false
» Building example-nodejs2...
! error reading from server: EOF
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build
! failed to create client: context deadline exceeded
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build
! failed to create client: context deadline exceeded
Checking the pods, I can see it crashing the waypoint server:
$ kubectl get pods | tail -n3
waypoint-runner-84984bdcd5-mcgkj 1/1 Running 3 (5m5s ago) 62m
example-nodejs-v1-6d89c89657-24n5t 0/1 ImagePullBackOff 0 45m
waypoint-server-0 1/1 Running 4 (75s ago) 63m
Debugging….
I invoked the build command then checked the server pod before it crashed:
$ waypoint build -local=false
» Building example-nodejs2...
! error reading from server: EOF
$ kubectl logs waypoint-server-0
2022-03-01T14:18:56.063Z [INFO] waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-03-01T14:18:56.063Z [DEBUG] waypoint: home configuration directory: path=/data/.config/waypoint
2022-03-01T14:18:56.063Z [INFO] waypoint.server: opening DB: path=/data/data.db
2022-03-01T14:18:56.064Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-03-01T14:18:56.064Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-03-01T14:18:56.343Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-03-01T14:18:56.343Z [DEBUG] waypoint.server.singleprocess.url_service: using saved URL guest token
2022-03-01T14:18:56.597Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-03-01T14:18:56.597Z [INFO] waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-03-01T14:18:56.597Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-03-01T14:18:56.597Z [INFO] waypoint.server.singleprocess.poll_queuer.project: starting
2022-03-01T14:18:56.597Z [INFO] waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-03-01T14:18:56.597Z [INFO] waypoint.server.singleprocess.prune: starting
2022-03-01T14:18:56.597Z [INFO] waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-01T14:18:56.631Z [INFO] waypoint.server.grpc: listener is wrapped with TLS
2022-03-01T14:18:56.631Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-03-01T14:18:56.631Z [INFO] waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-01T14:18:56.674Z [INFO] waypoint.server.http: listener is wrapped with TLS
» Server configuration:
DB Path: /data/data.db
gRPC Address: [::]:9701
HTTP Address: [::]:9702
Auth Required: yes
Browser UI Enabled: yes
URL Service: api.waypoint.run:443 (account: guest)
» Server logs:
2022-03-01T14:18:56.675Z [INFO] waypoint.server: starting built-in server: addr=[::]:9701
2022-03-01T14:18:56.675Z [INFO] waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
2022-03-01T14:18:56.675Z [INFO] waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022/03/01 14:19:01 http: TLS handshake error from 10.42.2.0:25941: remote error: tls: unknown certificate
2022-03-01T14:19:20.204Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:19:20.204Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:19:20.431Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:19:20.431Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:19:21.819Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-01T14:19:21.820Z [ERROR] waypoint.server.grpc: unknown runner connected: runner_id=01FX2XAW8C9ENQ3PB4ARM5JBGQ id=01FX2XAW8C9ENQ3PB4ARM5JBGQ
2022-03-01T14:19:21.820Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream response: error="rpc error: code = NotFound desc = runner ID not found: 01FX2XAW8C9ENQ3PB4ARM5JBGQ" duration=1.101831ms
2022-03-01T14:20:16.640Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:20:16.640Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=32.142µs
2022-03-01T14:20:16.669Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerConfig request
2022-03-01T14:20:17.170Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-01T14:20:51.154Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:20:51.154Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=42.552µs
2022-03-01T14:20:51.157Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject request
2022-03-01T14:20:51.158Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject response: error=<nil> duration=158.219µs
2022-03-01T14:20:51.386Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/QueueJob request
2022-03-01T14:20:51.386Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2
2022-03-01T14:20:51.386Z [INFO] waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FX2XNWSTD0G31VXWS3MHGTXZ
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]
goroutine 151 [running]:
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0xc0000d6b40, 0x0)
/tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0xc0000d6b40)
/tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0xc0006c8040)
/tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc00093ed20, {0x37d25a8, 0xc0004af680}, 0x8000101)
/tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc0004af680}, {0x2fad5e0, 0xc0006c8040})
/tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b
github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000b03c20}, {0x2fad5e0, 0xc0006c8040}, 0x40, 0xc000962318)
/tmp/wp-src/internal/server/auth.go:80 +0x204
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000b03c20}, {0x2fad5e0, 0xc0006c8040})
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000b03ad0}, {0x2fad5e0, 0xc0006c8040}, 0xc0013f2060, 0xc0006c8340)
/tmp/wp-src/internal/server/grpc_version.go:35 +0x8e
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000b03ad0}, {0x2fad5e0, 0xc0006c8040})
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc000b032f0}, {0x2fad5e0, 0xc0006c8040}, 0xc0013f2060, 0xc0006c8280)
/tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3
google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc000b032f0}, {0x2fad5e0, 0xc0006c8040}, 0xc000c00bb8, 0x2ce8d00)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc00093ed20}, {0x37d25a8, 0xc000b032f0}, 0xc00087a1e0, 0xc00075eac0)
/tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000a65340, {0x382aaa8, 0xc00051f980}, 0xc000af2ea0, 0xc000e78120, 0xa8684a0, 0x0)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f
google.golang.org/grpc.(*Server).handleStream(0xc000a65340, {0x382aaa8, 0xc00051f980}, 0xc000af2ea0, 0x0)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294
Let’s switch the runner profile (first attempt shows it was in a bad state). We do this by changing the OCI URL to a different runner image (OCI URL):
$ waypoint runner profile list
! failed to create client: context deadline exceeded
$ waypoint runner profile list
Runner profiles
NAME | PLUGIN TYPE | OCI URL | TARGET RUNNER ID | DEFAULT
-------------+-------------+-------------------------------+------------------+----------
kubernetes | kubernetes | hashicorp/waypoint-odr:latest | * | yes
$ waypoint runner profile set -name=kubernetes -oci-url=ghcr.io/hashicorp/waypoint/alpha-odr:latest -plugin-type=kubernetes
✓ Runner profile updated
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint runner profile list
Runner profiles
NAME | PLUGIN TYPE | OCI URL | TARGET RUNNER ID | DEFAULT
-------------+-------------+---------------------------------------------+------------------+----------
kubernetes | kubernetes | ghcr.io/hashicorp/waypoint/alpha-odr:latest | * |
And again, when trying to build, we see the same thing
$ waypoint build -local=false
» Building example-nodejs2...
! error reading from server: EOF
with a crash
$ kubectl logs waypoint-server-0
2022-03-01T14:22:27.840Z [INFO] waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-03-01T14:22:27.841Z [DEBUG] waypoint: home configuration directory: path=/data/.config/waypoint
2022-03-01T14:22:27.841Z [INFO] waypoint.server: opening DB: path=/data/data.db
2022-03-01T14:22:27.841Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-03-01T14:22:27.841Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-03-01T14:22:27.966Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-03-01T14:22:27.966Z [DEBUG] waypoint.server.singleprocess.url_service: using saved URL guest token
2022-03-01T14:22:28.207Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-03-01T14:22:28.208Z [INFO] waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-03-01T14:22:28.208Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-03-01T14:22:28.208Z [INFO] waypoint.server.singleprocess.poll_queuer.project: starting
2022-03-01T14:22:28.208Z [INFO] waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-03-01T14:22:28.208Z [INFO] waypoint.server.singleprocess.prune: starting
2022-03-01T14:22:28.208Z [INFO] waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-01T14:22:28.243Z [INFO] waypoint.server.grpc: listener is wrapped with TLS
2022-03-01T14:22:28.243Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-03-01T14:22:28.243Z [INFO] waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created
» Server configuration:
2022-03-01T14:22:28.285Z [INFO] waypoint.server.http: listener is wrapped with TLS
2022-03-01T14:22:28.285Z [INFO] waypoint.server: starting built-in server: addr=[::]:9701
2022-03-01T14:22:28.285Z [INFO] waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
DB Path: /data/data.db
gRPC Address: [::]:9701
HTTP Address: [::]:9702
Auth Required: yes
Browser UI Enabled: yes
URL Service: api.waypoint.run:443 (account: guest)
» Server logs:
2022-03-01T14:22:28.285Z [INFO] waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022-03-01T14:22:32.138Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:22:32.138Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=52.514µs
2022-03-01T14:22:32.141Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs request
2022-03-01T14:22:32.142Z [INFO] waypoint.server.singleprocess: looking up ondemand runner config by id: id=01fx2t1dxtfvzn0e1ykvnryzjk
2022-03-01T14:22:32.142Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs response: error=<nil> duration=809.491µs
2022-03-01T14:22:36.750Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:22:36.750Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=95.275µs
2022-03-01T14:22:36.754Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetOnDemandRunnerConfig request
2022-03-01T14:22:36.754Z [INFO] waypoint.server.singleprocess: looking up ondemand runner config by name: name=kubernetes
2022-03-01T14:22:36.754Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetOnDemandRunnerConfig response: error=<nil> duration=239.325µs
2022-03-01T14:22:36.755Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/UpsertOnDemandRunnerConfig request
2022-03-01T14:22:36.846Z [INFO] waypoint.server.singleprocess: looking up ondemand runner config by id: id=01FX2T1DXTFVZN0E1YKVNRYZJK
2022-03-01T14:22:37.657Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/UpsertOnDemandRunnerConfig response: error=<nil> duration=901.154437ms
2022-03-01T14:22:38.695Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-01T14:22:38.696Z [ERROR] waypoint.server.grpc: unknown runner connected: runner_id=01FX2XMTWVXB6W3YN34Y9AZ2D8 id=01FX2XMTWVXB6W3YN34Y9AZ2D8
2022-03-01T14:22:38.696Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream response: error="rpc error: code = NotFound desc = runner ID not found: 01FX2XMTWVXB6W3YN34Y9AZ2D8" duration=877.234µs
2022-03-01T14:22:38.794Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:22:38.794Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:22:38.975Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-01T14:22:38.975Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-01T14:22:40.594Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:22:40.594Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=103.629µs
2022-03-01T14:22:40.597Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs request
2022-03-01T14:22:40.597Z [INFO] waypoint.server.singleprocess: looking up ondemand runner config by id: id=01fx2t1dxtfvzn0e1ykvnryzjk
2022-03-01T14:22:40.597Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListOnDemandRunnerConfigs response: error=<nil> duration=132.514µs
2022-03-01T14:23:12.891Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-01T14:23:12.891Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=48.728µs
2022-03-01T14:23:12.895Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject request
2022-03-01T14:23:12.895Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject response: error=<nil> duration=134.615µs
2022-03-01T14:23:13.138Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/QueueJob request
2022-03-01T14:23:13.139Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2
2022-03-01T14:23:13.139Z [INFO] waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FX2XT77K36ERRBNECY2ETYRX
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]
goroutine 95 [running]:
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0xc0009efb80, 0x0)
/tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0xc0009efb80)
/tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0xc0005dd800)
/tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc0008015e0, {0x37d25a8, 0xc000368810}, 0x8000107)
/tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc000368810}, {0x2fad5e0, 0xc0005dd800})
/tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b
github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000368720}, {0x2fad5e0, 0xc0005dd800}, 0x40, 0xc0008c0858)
/tmp/wp-src/internal/server/auth.go:80 +0x204
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000368720}, {0x2fad5e0, 0xc0005dd800})
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000368690}, {0x2fad5e0, 0xc0005dd800}, 0xc000311640, 0xc0005dd980)
/tmp/wp-src/internal/server/grpc_version.go:35 +0x8e
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000368690}, {0x2fad5e0, 0xc0005dd800})
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc0003682d0}, {0x2fad5e0, 0xc0005dd800}, 0xc000311640, 0xc0005dd940)
/tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3
google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc0003682d0}, {0x2fad5e0, 0xc0005dd800}, 0xc000b5cbb8, 0x2ce8d00)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc0008015e0}, {0x37d25a8, 0xc0003682d0}, 0xc0008cef60, 0xc00120b000)
/tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000899340, {0x382aaa8, 0xc0005f2d80}, 0xc000875200, 0xc000afeab0, 0xa8684a0, 0x0)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f
google.golang.org/grpc.(*Server).handleStream(0xc000899340, {0x382aaa8, 0xc0005f2d80}, 0xc000875200, 0x0)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294
It’s now been several weeks. Checking the Alpha ODR listing of iamges: https://github.com/hashicorp/waypoint/pkgs/container/waypoint%2Falpha-odr I see there have been many since my last attempt.
I respect using “latest” on images. But I really need to know that when it pulls the new alpha-ocr, it gets the actual latest. So I’ll use the tagged SHA instead of “latest” this time
$ waypoint runner profile set -name=kubernetes -oci-url=ghcr.io/hashicorp/waypoint/alpha-odr:f38e20450 -plugin-type=kubernetes
✓ Runner profile updated
$ waypoint runner profile list
Runner profiles
NAME | PLUGIN TYPE | OCI URL | TARGET RUNNER ID | DEFAULT
-------------+-------------+------------------------------------------------+------------------+----------
kubernetes | kubernetes | ghcr.io/hashicorp/waypoint/alpha-odr:f38e20450 | * |
However, that crashes waypoint-server-0 right away as well
waypoint-server-0 0/1 Error 8 (87s ago) 18d
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ kubectl logs waypoint-server-0
2022-03-19T15:41:34.689Z [INFO] waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-03-19T15:41:34.690Z [DEBUG] waypoint: home configuration directory: path=/data/.config/waypoint
2022-03-19T15:41:34.690Z [INFO] waypoint.server: opening DB: path=/data/data.db
2022-03-19T15:41:34.690Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-03-19T15:41:34.690Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-03-19T15:41:34.791Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-03-19T15:41:34.791Z [DEBUG] waypoint.server.singleprocess.url_service: using saved URL guest token
2022-03-19T15:41:34.997Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-03-19T15:41:34.997Z [INFO] waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-03-19T15:41:34.997Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-03-19T15:41:34.997Z [INFO] waypoint.server.singleprocess.poll_queuer.project: starting
2022-03-19T15:41:34.998Z [INFO] waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-03-19T15:41:34.998Z [INFO] waypoint.server.singleprocess.prune: starting
2022-03-19T15:41:34.998Z [INFO] waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-03-19T15:41:35.032Z [INFO] waypoint.server.grpc: listener is wrapped with TLS
2022-03-19T15:41:35.032Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-03-19T15:41:35.032Z [INFO] waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created
» Server configuration:
DB Path: /data/data.db
gRPC Address: [::]:9701
HTTP Address: [::]:9702
Auth Required: yes
Browser UI Enabled: yes
URL Service: api.waypoint.run:443 (account: guest)
» Server logs:
2022-03-19T15:41:35.065Z [INFO] waypoint.server.http: listener is wrapped with TLS
2022-03-19T15:41:35.066Z [INFO] waypoint.server: starting built-in server: addr=[::]:9701
2022-03-19T15:41:35.066Z [INFO] waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
2022-03-19T15:41:35.066Z [INFO] waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022-03-19T15:41:54.649Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-19T15:41:54.649Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-19T15:41:58.346Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-19T15:41:58.383Z [ERROR] waypoint.server.grpc: unknown runner connected: runner_id=01FX2Y19MFEP6KQC35P7BZWWE8 id=01FX2Y19MFEP6KQC35P7BZWWE8
2022-03-19T15:41:58.383Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream response: error="rpc error: code = NotFound desc = runner ID not found: 01FX2Y19MFEP6KQC35P7BZWWE8" duration=36.81884ms
2022-03-19T15:41:58.749Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointLogStream request
2022-03-19T15:41:58.749Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/EntrypointConfig request
2022-03-19T15:42:09.352Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-19T15:42:09.352Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=29.643µs
2022-03-19T15:42:09.480Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerConfig request
2022-03-19T15:42:09.991Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/RunnerJobStream request
2022-03-19T15:42:42.109Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-03-19T15:42:42.109Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=58.049µs
2022-03-19T15:42:42.114Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject request
2022-03-19T15:42:42.114Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetProject response: error=<nil> duration=240.114µs
2022-03-19T15:42:42.367Z [INFO] waypoint.server.grpc: /hashicorp.waypoint.Waypoint/QueueJob request
2022-03-19T15:42:42.367Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2
2022-03-19T15:42:42.367Z [INFO] waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FYHDGPNZK2S3EYC408CN57Y0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d]
goroutine 134 [running]:
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0xc0006ff040, 0x0)
/tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0xc0006ff040)
/tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0xc000b75400)
/tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b
github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc0008cb360, {0x37d25a8, 0xc000e84f00}, 0x8000102)
/tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc000e84f00}, {0x2fad5e0, 0xc000b75400})
/tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b
github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000e84d50}, {0x2fad5e0, 0xc000b75400}, 0x40, 0xc000e581f8)
/tmp/wp-src/internal/server/auth.go:80 +0x204
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000e84d50}, {0x2fad5e0, 0xc000b75400})
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000e84cc0}, {0x2fad5e0, 0xc000b75400}, 0xc001129a20, 0xc000b75540)
/tmp/wp-src/internal/server/grpc_version.go:35 +0x8e
google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000e84cc0}, {0x2fad5e0, 0xc000b75400})
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9
github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc000e84660}, {0x2fad5e0, 0xc000b75400}, 0xc001129a20, 0xc000b754c0)
/tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3
google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc000e84660}, {0x2fad5e0, 0xc000b75400}, 0xc000f0ebb8, 0x2ce8d00)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f
github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc0008cb360}, {0x37d25a8, 0xc000e84660}, 0xc000268600, 0xc0013db640)
/tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000aeb340, {0x382aaa8, 0xc000c6ac00}, 0xc0008fe7e0, 0xc000c0d8f0, 0xa8684a0, 0x0)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f
google.golang.org/grpc.(*Server).handleStream(0xc000aeb340, {0x382aaa8, 0xc000c6ac00}, 0xc0008fe7e0, 0x0)
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294
Uninstall
We showed earlier how to uninstall with helm. Here, since we installed with the waypoint binary, we need to use the binary to uninstall as well.
$ waypoint server uninstall -platform kubernetes
! Uninstalling Waypoint server requires approval.
Rerun the command with -auto-approve to continue with the uninstall.
$ waypoint server uninstall -platform kubernetes -auto-approve
Uninstalling Waypoint server on platform "kubernetes" with context "install-1646140543"
✓ Snapshot "waypoint-server-snapshot-1647704697" generated
✓ Runner deployment deleted
✓ Statefulset and pods deleted
✓ Persistent Volume Claim deleted
✓ Service deleted
✓ Role binding deleted
✓ Cluster role deleted
✓ Cluster role binding deleted
Waypoint server successfully uninstalled for kubernetes platform
I checked to see if perhaps waypoint needed an update
$ sudo apt update
$ waypoint version
CLI: v0.7.2 (6e06e1a2d)
$ sudo apt install waypoint
Reading package lists... Done
Building dependency tree
Reading state information... Done
waypoint is already the newest version (0.7.2).
The following packages were automatically installed and are no longer required:
liblttng-ust-ctl4 liblttng-ust0
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 134 not upgraded.
And install it again
$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!
The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1647704975".
Use the "waypoint context" CLI to manage CLI contexts.
The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".
To launch and authenticate into the Web UI, run:
waypoint ui -authenticate
Advertise Address: 192.168.1.12:9701
Web UI Address: https://192.168.1.12:9702
We see our context has moved to the new IP
$ waypoint context list
| NAME | PLATFORM | SERVER ADDRESS
----+--------------------+------------+--------------------
* | install-1647704975 | kubernetes | 192.168.1.12:9701
Also, we can see Waypoint runners are set to the default latest image
$ waypoint runner profile list
Runner profiles
NAME | PLUGIN TYPE | OCI URL | TARGET RUNNER ID | DEFAULT
-------------+-------------+-------------------------------+------------------+----------
kubernetes | kubernetes | hashicorp/waypoint-odr:latest | * | yes
We need to set the HarborCred before trying again
$ cat /home/builder/harborAuth.json | base64 -w 0
bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=
$ waypoint config set -runner HARBORCRED=bm90IHJlYWxseSBteSBmaWxlLCBidXQgZ29vZCBvZiB5b3UgdG8gY2hlY2sgYnVkZHlcIQo=
We waypoint init to setup our local project
$ waypoint init
✓ Configuration file appears valid
✓ Connection to Waypoint server was successful
✓ Project "example-nodejs2" and all apps are registered with the server.
Project initialized!
You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
I want to check the project, so I’ll the auth token
$ waypoint user token
rHhYzVXQBcJzrGzb5wK8WSUWSEY7FXs3Gm4XuEEuXSmP6jQDi1kwLuyyQQap9GxwcZmTQGUnnbxKFMQuwRpLVMtZUuMu9bCm9vSAtu9YyLPsLbysTW4YU1t7DKA2g2Qjfa9qaaiPALtczTDjQ2KQCj4Jt4HFFEjzx
And login via the UI
And we see our project defined
Let’s try and fire it up.
and we see the crash
2022-03-19T16:26:06.956Z [DEBUG] waypoint.server.grpc: checking job project: project=example-nodejs2 ││ 2022-03-19T16:26:06.956Z [INFO] waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FYHG067C1AB2MR2304HP0WJP ││ panic: runtime error: invalid memory address or nil pointer dereference ││ [signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d] ││ ││ goroutine 1648 [running]: ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0xc000a50000, 0x0) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0xc000a50000) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0xc000c14380) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc00094a280, {0x37d25a8, 0xc0000f8c00}, 0x8000103) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc0000f8c00}, {0x2fad5e0, 0xc000c14380}) ││ /tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b ││ github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc0000f8b10}, {0x2fad5e0, 0xc000c14380}, 0x40, 0xc000d46978) ││ /tmp/wp-src/internal/server/auth.go:80 +0x204 ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc0000f8b10}, {0x2fad5e0, 0xc000c14380}) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9 ││ github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc0000f8a80}, {0x2fad5e0, 0xc000c14380}, 0xc0001bf2e0, 0xc000c14580) ││ /tmp/wp-src/internal/server/grpc_version.go:35 +0x8e ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc0000f8a80}, {0x2fad5e0, 0xc000c14380}) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9 ││ github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc0000f8180}, {0x2fad5e0, 0xc000c14380}, 0xc0001bf2e0, 0xc000c14540) ││ /tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3 ││ google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc0000f8180}, {0x2fad5e0, 0xc000c14380}, 0xc00125ebb8, 0x2ce8d00) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc00094a280}, {0x37d25a8, 0xc0000f8180}, 0xc000314b40, 0xc0010f5d20) ││ /tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138 ││ google.golang.org/grpc.(*Server).processUnaryRPC(0xc000b40000, {0x382aaa8, 0xc0008de600}, 0xc0007daa20, 0xc000e0c0f0, 0xa8684a0, 0x0) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f ││ google.golang.org/grpc.(*Server).handleStream(0xc000b40000, {0x382aaa8, 0xc0008de600}, 0xc0007daa20, 0x0) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a ││ google.golang.org/grpc.(*Server).serveStreams.func1.2() ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98 ││ created by google.golang.org/grpc.(*Server).serveStreams.func1 ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294 ││ Stream closed EOF for default/waypoint-server-0 (server)
I tried several more things…
I tried:
- forcing a new project
- hardcoding creds
- adding a git ref
the last run used (and the password has since been reset, but that is valid b64 string):
$ cat waypoint.hcl
project = "example-nodejs3"
runner {
enabled = true
data_source "git" {
url = "https://github.com/idjohnson/waypoint-examples.git"
ref = "main"
path = "kubernetes/nodejs"
}
}
app "example-nodejs3" {
labels = {
"service" = "example-nodejs",
"env" = "dev"
}
build {
use "pack" {}
registry {
use "docker" {
image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = "1"
encoded_auth = "ewoJInVzZXJuYW1lIjogImNocmRlbW8iLAoJInBhc3N3b3JkIjogIlRlbXBwYXNzMSIsCgkiZW1haWwiOiAiaXNhYWMuam9obnNvbkBnbWFpbC5jb20iCn0K"
}
}
}
deploy {
use "kubernetes" {
image_secret = "myharborreg"
probe_path = "/"
}
}
release {
use "kubernetes" {
// Sets up a load balancer to access released application
load_balancer = true
port = 3000
}
}
}
But in each case, waypoint-server-0 crashed
2022-03-19T16:38:53.176Z [INFO] waypoint.server.grpc: requesting ondemand runner via task start: runner-id=01FYHGQJFRV4VDD4XEEJS999ME ││ panic: runtime error: invalid memory address or nil pointer dereference ││ [signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x274e38d] ││ ││ goroutine 216 [running]: ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).onDemandRunnerStartJob(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0xc000ca32c0, 0x0) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:329 +0x52d ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).wrapJobWithRunner(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0xc000ca32c0) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:237 +0x7f ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).queueJobReqToJob(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0xc000023b40) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:197 +0x58b ││ github.com/hashicorp/waypoint/internal/server/singleprocess.(*service).QueueJob(0xc000ca0780, {0x37d25a8, 0xc000ba59e0}, 0x8000103) ││ /tmp/wp-src/internal/server/singleprocess/service_job.go:210 +0x2c ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler.func1({0x37d25a8, 0xc000ba59e0}, {0x2fad5e0, 0xc000023b40}) ││ /tmp/wp-src/internal/server/gen/server.pb.go:35058 +0x7b ││ github.com/hashicorp/waypoint/internal/server.authUnaryInterceptor.func1({0x37d25a8, 0xc000ba58c0}, {0x2fad5e0, 0xc000023b40}, 0x40, 0xc0006eaf48) ││ /tmp/wp-src/internal/server/auth.go:80 +0x204 ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000ba58c0}, {0x2fad5e0, 0xc000023b40}) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9 ││ github.com/hashicorp/waypoint/internal/server.versionUnaryInterceptor.func1({0x37d25a8, 0xc000ba5830}, {0x2fad5e0, 0xc000023b40}, 0xc000661360, 0xc000023d40) ││ /tmp/wp-src/internal/server/grpc_version.go:35 +0x8e ││ google.golang.org/grpc.getChainUnaryHandler.func1({0x37d25a8, 0xc000ba5830}, {0x2fad5e0, 0xc000023b40}) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1134 +0xb9 ││ github.com/hashicorp/waypoint/internal/server.logUnaryInterceptor.func1({0x37d25a8, 0xc000ba5440}, {0x2fad5e0, 0xc000023b40}, 0xc000661360, 0xc000023c40) ││ /tmp/wp-src/internal/server/grpc_log.go:36 +0x1e3 ││ google.golang.org/grpc.chainUnaryServerInterceptors.func1({0x37d25a8, 0xc000ba5440}, {0x2fad5e0, 0xc000023b40}, 0xc000fc7bb8, 0x2ce8d00) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1120 +0x8f ││ github.com/hashicorp/waypoint/internal/server/gen._Waypoint_QueueJob_Handler({0x31d3fe0, 0xc000ca0780}, {0x37d25a8, 0xc000ba5440}, 0xc00094c8a0, 0xc00029dd20) ││ /tmp/wp-src/internal/server/gen/server.pb.go:35060 +0x138 ││ google.golang.org/grpc.(*Server).processUnaryRPC(0xc000cb61c0, {0x382aaa8, 0xc000baec00}, 0xc0009ae000, 0xc0008dca80, 0xa8684a0, 0x0) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1293 +0xc6f ││ google.golang.org/grpc.(*Server).handleStream(0xc000cb61c0, {0x382aaa8, 0xc000baec00}, 0xc0009ae000, 0x0) ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:1618 +0xa2a ││ google.golang.org/grpc.(*Server).serveStreams.func1.2() ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:941 +0x98 ││ created by google.golang.org/grpc.(*Server).serveStreams.func1 ││ /go/pkg/mod/google.golang.org/grpc@v1.39.1/server.go:939 +0x294 ││ Stream closed EOF for default/waypoint-server-0 (server)
Crashing
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build
» Building example-nodejs3...
There are local changes that do not match the remote repository. By default,
Waypoint will perform this operation using a remote runner that will use the
remote repository’s git ref and not these local changes. For these changes
to be used for future operations, either commit and push, or run the operation
locally with the -local flag.
! error reading from server: EOF
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build
! failed to create client: context deadline exceeded
But if i change to local, this works just fine
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=true
» Building example-nodejs3...
Performing operation locally
Warning! Buildpacks are known to have issues on architectures other than amd64. The architecure being reported by the Docker server is "x86_64". We will still attempt to build the image, but may run into issues.
Creating new buildpack-based image using builder: heroku/buildpacks:20
✓ Creating pack client
✓ Building image
│ [exporter] Adding layer 'process-types'
│ [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
│ [exporter] Adding label 'io.buildpacks.build.metadata'
│ [exporter] Adding label 'io.buildpacks.project.metadata'
│ [exporter] Setting default process type 'web'
│ [exporter] Saving example-nodejs3...
│ [exporter] *** Images (56ddadcdd1b9):
│ [exporter] example-nodejs3
│ [exporter] Adding cache layer 'heroku/nodejs-engine:nodejs'
│ [exporter] Adding cache layer 'heroku/nodejs-engine:toolbox'
✓ Injecting entrypoint binary to image
Generated new Docker image: example-nodejs3:latest
✓ Tagging Docker image: example-nodejs3:latest => harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
✓ Pushing Docker image...
│ ab2b7fb6b8c0: Layer already exists
│ 5de7c0ba1e7a: Pushed
│ 4c226e279fd5: Pushed
│ aada41e93513: Layer already exists
│ c06c935628e1: Layer already exists
│ 4db08ac7c633: Layer already exists
│ 8891c4369930: Layer already exists
│ 68a85fa9d77e: Layer already exists
│ 1: digest: sha256:3bdc39376bd4e26c76785cca34555b844e1ec8f5b6b3c1a0c7e6d99d03fc61
│ 14 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$
And we see IF we build locally, we are fine
But as you can see, if I try again with remote, it crashes
│ 68a85fa9d77e: Layer already exists
│ 1: digest: sha256:3bdc39376bd4e26c76785cca34555b844e1ec8f5b6b3c1a0c7e6d99d03fc61
│ 14 size: 2624
✓ Docker image pushed: harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=false
» Building example-nodejs3...
There are local changes that do not match the remote repository. By default,
Waypoint will perform this operation using a remote runner that will use the
remote repository’s git ref and not these local changes. For these changes
to be used for future operations, either commit and push, or run the operation
locally with the -local flag.
! error reading from server: EOF
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint build -local=false
! failed to create client: context deadline exceeded
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$
Adding local runner
Let me try adding a local runner on a linux box
Install waypoint
builder@builder-HP-EliteBook-745-G5:~$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
OK
builder@builder-HP-EliteBook-745-G5:~$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Get:2 https://apt.releases.hashicorp.com focal InRelease [14.6 kB]
Hit:3 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:6 https://baltocdn.com/helm/stable/debian all InRelease
Get:7 https://apt.releases.hashicorp.com focal/main amd64 Packages [50.4 kB]
Fetched 65.0 kB in 1s (48.9 kB/s)
Reading package lists... Done
builder@builder-HP-EliteBook-745-G5:~$ sudo apt-get update && sudo apt-get install waypoint
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 https://apt.releases.hashicorp.com focal InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 https://baltocdn.com/helm/stable/debian all InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libfprint-2-tod1 libllvm10
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
waypoint
0 upgraded, 1 newly installed, 0 to remove and 73 not upgraded.
Need to get 112 MB of archives.
After this operation, 173 MB of additional disk space will be used.
Get:1 https://apt.releases.hashicorp.com focal/main amd64 waypoint amd64 0.7.2 [112 MB]
Fetched 112 MB in 5s (23.3 MB/s)
Selecting previously unselected package waypoint.
(Reading database ... 232177 files and directories currently installed.)
Preparing to unpack .../waypoint_0.7.2_amd64.deb ...
Unpacking waypoint (0.7.2) ...
Setting up waypoint (0.7.2) ...
If you try and add an agent with self signed certs (as is the default), you’ll see errors in the waypoint logs
│ 2022/03/19 16:55:09 http: TLS handshake error from 10.42.2.0:7886: remote error: tls: bad certificate ││ 2022/03/19 16:55:10 http: TLS handshake error from 10.42.2.0:30927: remote error: tls: bad certificate ││ 2022/03/19 16:55:12 http: TLS handshake error from 10.42.2.0:6183: remote error: tls: bad certificate ││ 2022/03/19 16:55:17 http: TLS handshake error from 10.42.2.0:27513: remote error: tls: bad certificate
So make sure to add TLK SKIP VERIFY
$ WAYPOINT_SERVER_TLS_SKIP_VERIFY=1 WAYPOINT_SERVER_ADDR=192.168.1.12:9701 WAYPOINT_SERVER_TLS=true WAYPOINT_SERVER_TOKEN=rHhYzVXQBcJzrGzb5wK8WSUW
SEY7FXs3Gm4XuEEuXSmP6jQDi1kwLuyyQQap9GxwcZmTQGUnnbxKFMQuwRpLVMtZUuMu9bCm9vSAtu9YyLPsLbysTW4YU1t7DKA2g2Qjfa9qaaiPALtczTDjQ2KQCj4Jt4HFFEjzx waypoint runner agent
2022-03-19T11:58:20.947-0500 [INFO] waypoint.runner.agent: initializing the runner
2022-03-19T11:58:20.947-0500 [INFO] waypoint.runner.agent: starting runner: id=01FYHHV6WK76ZK1SH109VECJ6M
2022-03-19T11:58:20.949-0500 [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
» Runner configuration:
Server address: 192.168.1.12:9701
» Runner logs:
Operating as a static Runner
2022-03-19T11:58:21.450-0500 [INFO] waypoint.runner.agent.runner: runner registered with server and ready
2022-03-19T11:58:21.450-0500 [INFO] waypoint.runner.agent.runner: waiting for job assignment
However, despite seeing the waypoint runner in my shell, asking waypoint for active runners does not show any
$ waypoint runner profile list
Runner profiles
NAME | PLUGIN TYPE | OCI URL | TARGET RUNNER ID | DEFAULT
-------------+-------------+-------------------------------+------------------+----------
kubernetes | kubernetes | hashicorp/waypoint-odr:latest | * | yes
But that too crashes things (and we crash the agent on the local linux host)
builder@builder-HP-EliteBook-745-G5:~$ WAYPOINT_SERVER_TLS_SKIP_VERIFY=1 WAYPOINT_SERVER_ADDR=192.168.1.12:9701 WAYPOINT_SERVER_TLS=true WAYPOINT_SERVER_TOKEN=rHhYzVXQBcJzrGzb5wK8WSUW
SEY7FXs3Gm4XuEEuXSmP6jQDi1kwLuyyQQap9GxwcZmTQGUnnbxKFMQuwRpLVMtZUuMu9bCm9vSAtu9YyLPsLbysTW4YU1t7DKA2g2Qjfa9qaaiPALtczTDjQ2KQCj4Jt4HFFEjzx waypoint runner agent
2022-03-19T11:58:20.947-0500 [INFO] waypoint.runner.agent: initializing the runner
2022-03-19T11:58:20.947-0500 [INFO] waypoint.runner.agent: starting runner: id=01FYHHV6WK76ZK1SH109VECJ6M
2022-03-19T11:58:20.949-0500 [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
» Runner configuration:
Server address: 192.168.1.12:9701
» Runner logs:
Operating as a static Runner
2022-03-19T11:58:21.450-0500 [INFO] waypoint.runner.agent.runner: runner registered with server and ready
2022-03-19T11:58:21.450-0500 [INFO] waypoint.runner.agent.runner: waiting for job assignment
2022-03-19T12:03:20.453-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-03-19T12:03:20.453-0500 [WARN] waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:20.453-0500 [ERROR] waypoint.runner.agent.runner.config_recv: error receiving configuration, exiting: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-03-19T12:03:22.454-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.1.12:9701: connect: connection refused""
2022-03-19T12:03:22.454-0500 [WARN] waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:43.230-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.1.12:9701: i/o timeout""
2022-03-19T12:03:43.230-0500 [WARN] waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:45.231-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.1.12:9701: i/o timeout""
2022-03-19T12:03:45.231-0500 [WARN] waypoint.runner.agent: server unavailable, sleeping before retry
2022-03-19T12:03:47.232-0500 [INFO] waypoint.runner.agent.runner: waiting for job assignment
2022-03-19T12:03:47.234-0500 [ERROR] waypoint.runner.agent: error running job: err="rpc error: code = NotFound desc = runner ID not found: 01FYHHV6WK76ZK1SH109VECJ6M"
2022-03-19T12:03:47.234-0500 [ERROR] waypoint.runner.agent: runner unexpectedly deregistered, exiting
2022-03-19T12:03:47.234-0500 [INFO] waypoint.runner.agent: quit request received, gracefully stopping runner
2022-03-19T12:03:47.234-0500 [WARN] waypoint.runner.agent.runner.watch_config: exiting due to context ended
builder@builder-HP-EliteBook-745-G5:~$
Cleanup
We’ll remove waypoint locally again
$ waypoint server uninstall -platform kubernetes -auto-approve
Uninstalling Waypoint server on platform "kubernetes" with context "install-1647704975"
✓ Snapshot "waypoint-server-snapshot-1647709511" generated
✓ Runner deployment deleted
✓ Statefulset and pods deleted
✓ Persistent Volume Claim deleted
✓ Service deleted
✓ Role binding deleted
✓ Cluster role deleted
✓ Cluster role binding deleted
Waypoint server successfully uninstalled for kubernetes platform
And, of course, change passwords (as i exposed via plain text)
Via Github Actions
First, we need to setup an external k8s base64 encoded env var
You can get the value if authed locally
$ cat ~/.kube/config | base64 -w 0
We can then define a Github Actions workflow to build and deploy our hcl using a waypoint binary we pull down on the fly.
I built the Github Actions yaml based largely on this Gitlab example
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ cat .github/workflows/waypoint.yml
name: GitHub To Waypoint
on: [pull_request, push]
env:
WAYPOINT_VERSION: '0.7.2'
WAYPOINT_SERVER_ADDR: ''
WAYPOINT_SERVER_TOKEN: ''
WAYPOINT_SERVER_TLS: '1'
WAYPOINT_SERVER_TLS_SKIP_VERIFY: '1'
jobs:
HostedActions:
runs-on: ubuntu-latest
steps:
- run: echo "🎉 The job was automatically triggered by a $ event."
- run: echo "🐧 This job is now running on a $ server hosted by GitHub!"
- run: echo "🔎 The name of your branch is $ and your repository is $."
- name: Check out repository code
uses: actions/checkout@v2
- run: echo "💡 The $ repository has been cloned to the runner."
- run: echo "🖥️ The workflow is now ready to test your code on the runner."
- name: install waypoint
run: |
wget -q -O /tmp/waypoint.zip https://releases.hashicorp.com/waypoint/$/waypoint_$_linux_amd64.zip
unzip -d /usr/local/bin /tmp/waypoint.zip
rm -rf /tmp/waypoint*
- name: set kubernetes config
run: |
mkdir -p ~/.kube
set -x
echo $ | base64 --decode > ~/.kube/config
- name: install waypoint
run: |
# go to src dir
cd kubernetes/nodejs
waypoint init
waypoint build
waypoint deploy
waypoint release
In this model we do not have a waypoint server to interrogate or check. We are just using an on-the-fly waypoint to build and release.
We can see some output (though it appears hung at the check service step)
We see it deployed, but again, it seems confused on how to properly expose a service on my k3s 1.23 cluster
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl get pods | grep example
example-nodejs-v3-ffdd6fd68-8bqlv 1/1 Running 0 18d
svclb-example-nodejs-vzrcb 1/1 Running 0 18d
svclb-example-nodejs-8tv7n 1/1 Running 0 18d
svclb-example-nodejs-xsvgc 1/1 Running 0 18d
svclb-example-nodejs-h5t9t 1/1 Running 0 18d
example-nodejs-v2-78b4dd576d-tc9vr 1/1 Running 0 18d
svclb-example-nodejs-c7th7 1/1 Running 0 18d
svclb-example-nodejs-tznc4 1/1 Running 0 18d
svclb-example-nodejs-sds7v 1/1 Running 0 18d
example-nodejs2-v1-5d4dd7cc4-vmbv4 1/1 Running 0 3h37m
example-nodejs-v1-6d89c89657-ss5fm 0/1 ImagePullBackOff 0 4d8h
example-nodejs3-v1-df8dff4cd-mpjq8 1/1 Running 0 4m27s
svclb-example-nodejs3-pmszp 0/1 Pending 0 2m34s
svclb-example-nodejs3-mntnh 0/1 Pending 0 2m34s
svclb-example-nodejs3-ffhsp 0/1 Pending 0 2m34s
svclb-example-nodejs3-92l9r 0/1 Pending 0 2m34s
svclb-example-nodejs3-2dw28 0/1 Pending 0 2m34s
svclb-example-nodejs3-wdt86 0/1 Pending 0 2m34s
svclb-example-nodejs3-tldxw 0/1 Pending 0 2m34s
As we see, indeed the service is stuck in pending (since it cannot grab port 3000 on all my servers)
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl get svc | grep example
example-auth ClusterIP 10.43.106.217 <none> 3000/TCP 299d
example-nodejs LoadBalancer 10.43.80.174 192.168.1.12,192.168.1.159,192.168.1.205,192.168.1.214,192.168.1.33,192.168.1.57,192.168.1.77 3000:32482/TCP 18d
example-nodejs3 LoadBalancer 10.43.225.183 <pending> 3000:32244/TCP 3m41s
I can unblock by removing the last service
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl delete svc example-nodejs
service "example-nodejs" deleted
Then we can see it pull port 3000
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ kubectl get svc | grep example
example-auth ClusterIP 10.43.106.217 <none> 3000/TCP 299d
example-nodejs3 LoadBalancer 10.43.225.183 192.168.1.12,192.168.1.159,192.168.1.205,192.168.1.214,192.168.1.33,192.168.1.57,192.168.1.77 3000:32244/TCP 4m47s
And this then succeeds
and we can indeed see the app is exposed on port 3000
Let’s push a change and see if that works.
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples$ git diff kubernetes/nodejs/views/pages/index.ejs
diff --git a/kubernetes/nodejs/views/pages/index.ejs b/kubernetes/nodejs/views/pages/index.ejs
index f6c45be..f487206 100644
--- a/kubernetes/nodejs/views/pages/index.ejs
+++ b/kubernetes/nodejs/views/pages/index.ejs
@@ -18,6 +18,7 @@
<h1>This Node.js app was deployed with Waypoint.</h1>
<p>
Try making a change to this text locally and run <code>waypoint up</code> again to see it.
+ <b>I changed it!</b>
</p>
<p>
Read the <a href="https://waypointproject.io/docs">documentation</a> for more about Waypoint.
It made a change, but I see no evidence it actually changed
I can confirm it did build and push to my private CR:
from the logs
and on Harbor
I port-forwarded to the new deploy and it too wasn’t updated
I fear it’s doing to re-using tags. Looking at the events on the pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m34s default-scheduler Successfully assigned default/example-nodejs3-v1-6867f6fc48-mswv4 to builder-macbookpro2
Normal Pulled 6m33s kubelet Container image "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:1" already present on machine
Normal Created 6m32s kubelet Created container example-nodejs3
We can fix this by not reusing the same tag. This is configured in the HCL
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git diff waypoint.hcl
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index 31227ba..fedba49 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -22,7 +22,7 @@ app "example-nodejs3" {
registry {
use "docker" {
image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
- tag = "1"
+ tag = gitrefpretty()
encoded_auth = "ewoJInVzZXJuYW1lIjogImNocmRlbW8iLAoJInBhc3N3b3JkIjogIlRlbXBwYXNzMSIsCgkiZW1haWwiOiAiaXNhYWMuam9obnNvbkBnbWFpbC5jb20iCn0K"
}
}
Now we commit and push
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git add waypoint.hcl
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git commit -m "use ref tag"
[main 544220a] use ref tag
1 file changed, 1 insertion(+), 1 deletion(-)
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 16 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 430 bytes | 430.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/idjohnson/waypoint-examples.git
c6df4b1..544220a main -> main
This time it worked
$ kubectl describe pod example-nodejs3-v1-7c77f4697c-5qkm6 | grep image
Normal Pulling 2m57s kubelet Pulling image "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:544220adb33cf427d4bcf27519f31dc1486f59c7"
Normal Pulled 2m49s kubelet Successfully pulled image "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs:544220adb33cf427d4bcf27519f31dc1486f59c7" in 7.415101299s
One last change; as you recall, to debug, we set the Harbor Reg creds inline.
Let’s fix those to use a GH secret as well.
First add a secret with the creds
Then change the waypoint HCL to use a config that we will set in the Github Actions workflow to the secret
$ git diff
diff --git a/.github/workflows/waypoint.yml b/.github/workflows/waypoint.yml
index adfe239..0605bcf 100644
--- a/.github/workflows/waypoint.yml
+++ b/.github/workflows/waypoint.yml
@@ -36,6 +36,9 @@ jobs:
# go to src dir
cd kubernetes/nodejs
waypoint init
+ set +x
+ waypoint config set -runner HARBORCRED=$
+ set -x
waypoint build
waypoint deploy
waypoint release
diff --git a/kubernetes/nodejs/waypoint.hcl b/kubernetes/nodejs/waypoint.hcl
index fedba49..020033d 100644
--- a/kubernetes/nodejs/waypoint.hcl
+++ b/kubernetes/nodejs/waypoint.hcl
@@ -23,7 +23,7 @@ app "example-nodejs3" {
use "docker" {
image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = gitrefpretty()
- encoded_auth = "ewoJInVzZXJuYW1lIjogImNocmRlbW8iLAoJInBhc3N3b3JkIjogIlRlbXBwYXNzMSIsCgkiZW1haWwiOiAiaXNhYWMuam9obnNvbkBnbWFpbC5jb20iCn0K"
+ encoded_auth = ${HARBORCRED}
}
}
}
Integrating GH Actions and a persistent Waypoint server
The final path I wanted to explore, and likely the one most would want to emulate, was to use a waypoint server on which teams could collaborate but build using a PR workflow using Github ephemeral agents.
I spun up a fresh waypoint server in my local cluster using the waypoint CLI
```bash
$ waypoint install -platform=kubernetes -accept-tos
✓ Gathering information about the Kubernetes cluster...
✓ Service account for on-demand runner initialized!
✓ Creating Kubernetes resources...
✓ Kubernetes StatefulSet reporting ready
✓ Service "waypoint" is ready
✓ Server installed and configured!
✓ Successfully connected to Waypoint server in Kubernetes!
✓ Installing runner...
✓ Registered ondemand runner!
✓ Creating Deployment for Runner
✓ Kubernetes Deployment for Waypoint runner reporting ready
Waypoint server successfully installed and configured!
The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1645496978".
Use the "waypoint context" CLI to manage CLI contexts.
The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".
To launch and authenticate into the Web UI, run:
waypoint ui -authenticate
Advertise Address: 192.168.1.12:9701
Web UI Address: https://1192.168.1.12:9702
I then punched a hole in my router so we could ingress from 9701 and 9702 to my Kubernetes service running locally
Verification:
I used waypoint user token
to get a new token and added that as a secret in Github Actions secrets.
Lastly, I updated my Github Actions workflow to build and deploy locally, but use the remote server for tracking.
name: GitHub To Waypoint
on: [pull_request, push]
env:
WAYPOINT_VERSION: '0.7.2'
WAYPOINT_SERVER_ADDR: '73.242.50.46:9701'
WAYPOINT_SERVER_TLS: '1'
WAYPOINT_SERVER_TLS_SKIP_VERIFY: '1'
jobs:
HostedActions:
runs-on: ubuntu-latest
steps:
- run: echo "🎉 The job was automatically triggered by a $ event."
- run: echo "🐧 This job is now running on a $ server hosted by GitHub!"
- run: echo "🔎 The name of your branch is $ and your repository is $."
- name: Check out repository code
uses: actions/checkout@v2
- run: echo "💡 The $ repository has been cloned to the runner."
- run: echo "🖥️ The workflow is now ready to test your code on the runner."
- name: install waypoint
run: |
wget -q -O /tmp/waypoint.zip https://releases.hashicorp.com/waypoint/$/waypoint_$_linux_amd64.zip
unzip -d /usr/local/bin /tmp/waypoint.zip
rm -rf /tmp/waypoint*
- name: set kubernetes config
run: |
mkdir -p ~/.kube
set -x
echo $ | base64 --decode > ~/.kube/config
- name: install waypoint
run: |
# go to src dir
cd kubernetes/nodejs
# sed -i "s/encoded_auth =.*/encoded_auth = \"$\"/g" waypoint.hcl
set +x
waypoint config set HARBORCRED=$
set -x
waypoint init
waypoint build -local=true
waypoint deploy -local=true
waypoint release
env: # Or as an environment variable
WAYPOINT_SERVER_TOKEN: $
WP_VAR_harborcred: $
One nuance to using the local runner that depended on a set credential was that it would get validated and fail on parsing the waypoint.hcl
Run # go to src dir
-> Validating configuration file...
-> Configuration file appears valid
-> Validating server credentials...
-> Connection to Waypoint server was successful
-> Checking if project "example-nodejs3" is registered...
-> Project "example-nodejs3" and all apps are registered with the server.
Project initialized!
You may now call 'waypoint up' to deploy your project or
commands such as 'waypoint build' to perform steps individually.
+ waypoint build -local=true
» Building example-nodejs3...
Performing operation locally
! /home/runner/work/waypoint-examples/waypoint-examples/kubernetes/nodejs/waypoint.hcl:26,27-37:
Unknown variable; There is no variable named "HARBORCRED"., and 1 other
diagnostic(s)
One issue was the syntax; having “=” in the real value threw it off. I could mitigate by wrapping the value in double quotes.
encoded_auth = "SGV5LCBob3cgaXMgaXQgZ29pbmcuICBTdXJlIHdpbGwgYmUgbmljZSB3aGVuIHdpbnRlciBpcyBkb25lLiBMb29raW5nIGZvcndhcmQgdG8gZ2V0dGluZyBvdXQgYW5kIGRvaW5nIHNvbWUgY2FtcGluZy4K=="
But clearly, we need to solve variables that can be at build time in an ephemeral Github runner and still be used in Waypoint as a Kubernetes runner.
To do this, I moved it to a proper waypoint “variable”.
project = "example-nodejs3"
runner {
enabled = true
data_source "git" {
url = "https://github.com/idjohnson/waypoint-examples.git"
ref = "main"
path = "kubernetes/nodejs"
}
}
app "example-nodejs3" {
labels = {
"service" = "example-nodejs",
"env" = "dev"
}
build {
use "pack" {}
registry {
use "docker" {
image = "harbor.freshbrewed.science/freshbrewedprivate/example-nodejs"
tag = gitrefpretty()
encoded_auth = var.harborcred
}
}
}
deploy {
use "kubernetes" {
image_secret = "myharborreg"
probe_path = "/"
}
}
release {
use "kubernetes" {
// Sets up a load balancer to access released application
load_balancer = true
port = 3000
}
}
}
variable "harborcred" {
type = string
default = null
}
Now there is a var that if I skip, it just defaults to blank.
As you saw above the Github Actions, we passed the var in using the WP_VAR environment variable syntax of waypoint that will turn “WP_VAR_harborcred” into the variable “harborcred”. (The other option would be to pass in a file or use the “-var” command line variable).
env: # Or as an environment variable
WAYPOINT_SERVER_TOKEN: $
WP_VAR_harborcred: $
Now when we run we see it built in Github
-> Kubernetes client connected to https://73.242.50.46:12345 with namespace default
-> Preparing deployment...
->
-> Expected "http" port "3000" for app "example-nodejs3-v1"
-> Updating deployment...
-> Waiting for deployment...
-> Waiting on deployment to become available: requested=1 running=2 ready=1
-> Deployment successfully rolled out!
Performing operation locally
-> Gathering health report for Kubernetes deployment...
-> Checking status of the Kubernetes deployment resource...
-> Finished building report for Kubernetes deployment resource
Determining overall container health...
-> Finished building report for Kubernetes platform
*: Deployment "example-nodejs3-v1" is reporting partially available!
*: Waypoint detected that the current deployment is not ready, however your application
might be available or still starting up.
*: pod "example-nodejs3-v1-6dfb9ffc96-2xwpm" is reporting "ALIVE"
» Releasing...
Performing operation locally
-> Initializing Kubernetes client...
-> Kubernetes client connected to https://73.242.50.46:12345 with namespace default
-> Preparing service...
-> Updating existing service...
-> Waiting for service to become ready...
-> Service is ready!
Performing operation locally
-> Gathering health report for Kubernetes release...
-> Checking status of Kubernetes service resource "example-nodejs3"...
-> Finished building report for Kubernetes service resource
-> Finished building report for Kubernetes platform
The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."
Release URL: http://192.168.1.12:3000
Deployment URL: https://deadly-pure-ladybird--v1.waypoint.run
+ waypoint release
» Releasing example-nodejs3...
The deployment "1" is already the released deployment. Nothing to be done.
The app is deployed and running
The Complete Flow
We see a change made:
diff --git a/kubernetes/nodejs/views/pages/index.ejs b/kubernetes/nodejs/views/pages/index.ejs
index f487206..3f7abbf 100644
--- a/kubernetes/nodejs/views/pages/index.ejs
+++ b/kubernetes/nodejs/views/pages/index.ejs
@@ -18,7 +18,7 @@
<h1>This Node.js app was deployed with Waypoint.</h1>
<p>
Try making a change to this text locally and run <code>waypoint up</code> again to see it.
- <b>I changed it!</b>
+ <b>I changed it again!</b>
</p>
<p>
Read the <a href="https://waypointproject.io/docs">documentation</a> for more about Waypoint.
Triggers a build which results in a live site update
And yet we can still see our build and deployment logs
However, until we change deployments back to Waypoint, we will see errors in the deployment log (albeit the deployment happened)
I noted that even if I set the project with -p, it would crash waypoint-server-0 if I tried to release remotely. Only “-local” worked
- name: install waypoint
run: |
# go to src dir
cd kubernetes/nodejs
# sed -i "s/encoded_auth =.*/encoded_auth = \"$\"/g" waypoint.hcl
set +x
waypoint config set HARBORCRED=$
waypoint config set -runner HARBORCRED=$
set -x
waypoint init
waypoint build -p example-nodejs3 -local=true
waypoint deploy -p example-nodejs3
waypoint release -p example-nodejs3
I also overcame one other slight annoyance. By default, Waypoint leaves all the prior releases running:
$ kubectl get pods | grep example-node
example-nodejs-v3-ffdd6fd68-8bqlv 1/1 Running 0 19d
example-nodejs-v2-78b4dd576d-tc9vr 1/1 Running 0 19d
example-nodejs2-v1-5d4dd7cc4-vmbv4 1/1 Running 0 31h
svclb-example-nodejs3-tldxw 1/1 Running 0 28h
svclb-example-nodejs3-pmszp 1/1 Running 0 28h
svclb-example-nodejs3-2dw28 1/1 Running 0 28h
svclb-example-nodejs3-92l9r 1/1 Running 0 28h
svclb-example-nodejs3-ffhsp 1/1 Running 0 28h
svclb-example-nodejs3-wdt86 1/1 Running 0 28h
svclb-example-nodejs3-mntnh 1/1 Running 0 28h
example-nodejs3-v2-758ddc49dc-stlhv 1/1 Running 1 (13m ago) 7h38m
example-nodejs3-v3-854dd4d55f-cr79v 1/1 Running 1 (13m ago) 3h54m
example-nodejs-v1-6d89c89657-ss5fm 0/1 ImagePullBackOff 0 5d12h
That is kind of a waste of resources for active development.
Howver, adding “-prune” to the release step will get rid of the former unused releases.
- name: install waypoint
run: |
# go to src dir
cd kubernetes/nodejs
# sed -i "s/encoded_auth =.*/encoded_auth = \"$\"/g" waypoint.hcl
set +x
waypoint config set HARBORCRED=$
waypoint config set -runner HARBORCRED=$
set -x
waypoint init
waypoint build -local=true
waypoint deploy -local=true
waypoint release -prune
And we can see it went from 3 versions running
$ kubectl get pods | grep ^example-nodejs3
example-nodejs3-v2-758ddc49dc-stlhv 1/1 Running 1 (17m ago) 7h43m
example-nodejs3-v3-854dd4d55f-cr79v 1/1 Running 1 (17m ago) 3h59m
example-nodejs3-v5-7b57f688b9-5t6zz 0/1 Running 0 27s
to just the one
$ kubectl get pods | grep ^example-nodejs3
example-nodejs3-v5-7b57f688b9-5t6zz 1/1 Running 0 2m21s
Auth tokens
If you are on the computer that installed waypoint, you can get fresh tokens easily
However, if you lose the token, you are in a real pickle.
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ waypoint user token
! failed to create client: no server connection configuration found
builder@DESKTOP-QADGF36:~/Workspaces/idj-waypoint-examples/kubernetes/nodejs$ WAYPOINT_SERVER_TLS_SKIP_VERIFY=1 WAYPOINT_SERVER_ADDR=192.168.1.12:9701 WAYPOINT_SERVER_TLS=true waypoint user token
! Authorization token is not supplied
There appears to be no other options in the docs. The only thread I could find showed the user just reinstalling when it was lost.
So guard that token as if it’s lost and you’ve not added other users, you are in a stuck spot.
I should point out you can invite others which generates a pre-authed link with a token
However, it should be worth noting that there aren’t really different types of users. Either you are in Waypoint or not. So I could not use that user, for instance, to reset the token for another. Nor can I create a “read-only” type of user either.
Summary
I really wanted to find a path to liking Waypoint. I watched a year old DevOps Toolkit Review of Waypoint to try and see if I missed something. In his review, Victor found errors and could not quite see a value.
While some of his points have counters; like the too many current releases running (use “-prune” to remove old ones), the fact is that he raises the real underlying question: why would we want to deploy to Kubernetes in a way different than we might promote? That is, if we use Helm for releasing, why would use anything else for deploying to lower environments?
In the last example we worked through, we explored using Github Actions to build our application but then Waypoint to track the releases. It was nice to see our build logs archived for past releases outside our CICD tool (Github).
However, our build log was just the buildpack steps from Waypoint. This constitutes just a small portion of the overall workflow.
If I wish to track externally, I might use something like Datadog or SumoLogic which could capture both logs
pipeline metrics
and tests.
The last item there brings up the other point Victor raised in his review; where are the tests?
The waypoint tool clearly covers build, deploy and release, but completely absent is testing. This issue has been raised by others.
Perhaps one might argue that Waypoint isn’t for Kubernetes, it’s for other deployment models, If that was the case, I would lean towards using Nomad or using regular release pipelines including ones such as Azure DevOps Release pipelines. AzDO Release Pipelines can have complex logic with forked stages, approval gates and deployment groups. All things that would be hard if not impossible in Waypoint.
Waypoint lacks user administration, access levels and built-in notification systems. In all cases, they would want you to use a webhook or plugin. Presently, there are no notifcation plugins.
So again, I ask, what is the point of Waypoint? I just cannot see where it fits in the picture outside of adding a rudimentary CI tool to the Hashi suite. With it’s general lack of stability and full stack coverage (namely tests), it doesn’t seem it’s quite ready for widespread adoption at this time. I find it telling, as well, that the tool itself uses Github Actions and Circle CI to build and release (instead of Waypoint).