Published: Dec 20, 2022 by Isaac Johnson
Today we’ll use ansible to rebuild an on-prem k3s cluster. We’ll then load it with a Redis instance and Dapr. We’ll deploy the pub-sub and calculator apps followed by zipkin.
We’ll use Zipkin (the service/app) to collect zipkin (the format) trace data, then pivot to setting up a zipkin forwarder to GCP Cloud Trace. This will require a service principal with the proper role setup.
Lastly, we’ll explore the data collected in Cloud Trace, how we can use it, and lastly some cost considerations.
Let’s dig in!
Resetting the cluster
Before I do anything, I should clean up the Kubernetes cluster with a quick refresh. Luckily, I’ve created playbooks to do just that.
I’ll go to my AWX instance and fire the reloadk3s template. (If you want to read more about that process, you can see a detailed step-by-step here)
This will use the reloadk3s playbook in git to rebuild the cluster fresh.
I had a bit of a hiccup in that my az CLI was logged out, blocking the playbook. At some point I’ll need to sort out refreshing Azure CLI logins on hosts. Until such time, I just hopped onto the k8s master (192.168.1.81) and used az login --use-device-code to refresh (and as root as well).
Once done, I could see my fresh cluster
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
isaac-macbookpro Ready <none> 4m58s v1.23.10+k3s1
anna-macbookair Ready control-plane,master 45m v1.23.10+k3s1
builder-macbookpro2 Ready <none> 2m6s v1.23.10+k3s1
Updating Dapr CLI
It’s always a good idea to get the latest and greatest Dapr.io. (Dapr.io is one of my favourite OS frameworks, you can see quite a few past articles here)
builder@DESKTOP-QADGF36:~$ dapr version
CLI version: 1.8.0
Runtime version: n/a
This is a bit out of date. We can easily upgrade our local CLI
builder@DESKTOP-QADGF36:~$ wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
Getting the latest Dapr CLI...
Your system is linux_amd64
Dapr CLI is detected:
CLI version: 1.8.0
Runtime version: n/a
Reinstalling Dapr CLI - /usr/local/bin/dapr...
Installing v1.9.1 Dapr CLI...
Downloading https://github.com/dapr/cli/releases/download/v1.9.1/dapr_linux_amd64.tar.gz ...
[sudo] password for builder:
dapr installed into /usr/local/bin successfully.
CLI version: 1.9.1
Runtime version: n/a
To get started with Dapr, please visit https://docs.dapr.io/getting-started/
builder@DESKTOP-QADGF36:~$ dapr version
CLI version: 1.9.1
Runtime version: n/a
Installing Dapr to K8s
The easiest way (and the recommended) is to just use dapr init -k. This ensures you have the latest and greatest Dapr runtime installed to your Kubernetes cluster
builder@DESKTOP-QADGF36:~$ dapr init -k
⌛ Making the jump to hyperspace...
ℹ️ Note: To install Dapr using Helm, see here: https://docs.dapr.io/getting-started/install-dapr-kubernetes/#install-with-helm-advanced
ℹ️ Container images will be pulled from Docker Hub
✅ Deploying the Dapr control plane to your cluster...
✅ Success! Dapr has been installed to namespace dapr-system. To verify, run `dapr status -k' in your terminal. To get started, go here: https://aka.ms/dapr-getting-started
We can now check our local version and Kubernetes version
builder@DESKTOP-QADGF36:~$ dapr version
CLI version: 1.9.1
Runtime version: n/a
builder@DESKTOP-QADGF36:~$ dapr status -k
NAME NAMESPACE HEALTHY STATUS REPLICAS VERSION AGE CREATED
dapr-placement-server dapr-system True Running 1 1.9.5 53s 2022-12-16 06:51.10
dapr-dashboard dapr-system True Running 1 0.11.0 53s 2022-12-16 06:51.10
dapr-sidecar-injector dapr-system True Running 1 1.9.5 53s 2022-12-16 06:51.10
dapr-operator dapr-system True Running 1 1.9.5 53s 2022-12-16 06:51.10
dapr-sentry dapr-system True Running 1 1.9.5 53s 2022-12-16 06:51.10
builder@DESKTOP-QADGF36:~$ kubectl get pods --namespace dapr-system
NAME READY STATUS RESTARTS AGE
dapr-dashboard-6b868c46fb-ldplf 1/1 Running 0 98s
dapr-sidecar-injector-75948876c-xfwk9 1/1 Running 0 98s
dapr-operator-6bd7f7bfdd-xft76 1/1 Running 0 98s
dapr-placement-server-0 1/1 Running 0 98s
dapr-sentry-7f8db4ff47-qvmr7 1/1 Running 0 98s
Dapr Quickstarts
We’ll want to get the quickstarts from Dapr. These get updated pretty often (as you can see below)
To pull fresh
$ git clone https://github.com/dapr/quickstarts.git
Or, in my case, updating
builder@DESKTOP-QADGF36:~/Workspaces$ cd quickstarts/
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts$ git pull
remote: Enumerating objects: 913, done.
remote: Counting objects: 100% (226/226), done.
remote: Total 913 (delta 226), reused 226 (delta 226), pack-reused 687
Receiving objects: 100% (913/913), 75.13 MiB | 8.28 MiB/s, done.
Resolving deltas: 100% (473/473), completed with 70 local objects.
From https://github.com/dapr/quickstarts
95f81e6..b0b9a5a master -> origin/master
* [new branch] dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/express-4.17.3 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/express-4.17.3
* [new branch] dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/qs-6.11.0 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/qs-6.11.0
* [new branch] dependabot/npm_and_yarn/pub_sub/javascript/sdk/checkout/qs-and-body-parser-6.11.0 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/sdk/checkout/qs-and-body-parser-6.11.0
* [new branch] dependabot/npm_and_yarn/pub_sub/javascript/sdk/order-processor/qs-and-body-parser-6.11.0 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/sdk/order-processor/qs-and-body-parser-6.11.0
* [new branch] dependabot/npm_and_yarn/secrets_management/javascript/sdk/order-processor/qs-and-body-parser-6.11.0 -> origin/dependabot/npm_and_yarn/secrets_management/javascript/sdk/order-processor/qs-and-body-parser-6.11.0
* [new branch] dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/express-4.18.2 -> origin/dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/express-4.18.2
* [new branch] dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/qs-and-express-6.11.0 -> origin/dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/qs-and-express-6.11.0
* [new branch] dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/express-4.18.2 -> origin/dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/express-4.18.2
* [new branch] dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/qs-and-express-6.11.0 -> origin/dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/qs-and-express-6.11.0
* [new branch] greenie-msft-patch-1 -> origin/greenie-msft-patch-1
* [new branch] release-1.9 -> origin/release-1.9
* [new branch] yaron2-patch-1 -> origin/yaron2-patch-1
* [new tag] v1.9.0 -> v1.9.0
Updating 95f81e6..b0b9a5a
Fast-forward
.github/workflows/validate.yaml | 11 +-
.github/workflows/validate_new_quickstarts_bindings.yaml | 20 +-
.github/workflows/validate_new_quickstarts_configuration.yaml | 103 +
.github/workflows/validate_new_quickstarts_pubsub.yaml | 68 +-
.github/workflows/validate_new_quickstarts_secrets_management.yaml | 65 +-
.github/workflows/validate_new_quickstarts_service_invo.yaml | 47 +-
.github/workflows/validate_new_quickstarts_state_management.yaml | 65 +-
README.md | 5 +-
bindings/csharp/sdk/batch/batch.csproj | 2 +-
bindings/go/http/README.md | 4 +-
bindings/go/http/batch/app.go | 10 +-
bindings/go/http/batch/dapr_output_binding | Bin 0 -> 7212034 bytes
bindings/go/http/batch/go.mod | 15 +-
bindings/go/sdk/README.md | 4 +-
bindings/go/sdk/batch/app.go | 10 +-
bindings/go/sdk/batch/dapr_output_binding | Bin 0 -> 14505250 bytes
bindings/go/sdk/batch/go.mod | 27 +-
bindings/go/sdk/batch/go.sum | 94 +-
bindings/java/sdk/batch/pom.xml | 4 +-
bindings/javascript/sdk/README.md | 2 +-
configuration/components/configstore.yaml | 12 +
configuration/csharp/http/README.md | 71 +
configuration/csharp/http/makefile | 2 +
configuration/csharp/http/order-processor/Program.cs | 94 +
configuration/csharp/http/order-processor/Program.csproj | 15 +
configuration/csharp/http/order-processor/Properties/launchSettings.json | 15 +
configuration/csharp/http/order-processor/appsettings.Development.json | 9 +
configuration/csharp/http/order-processor/appsettings.json | 10 +
configuration/csharp/sdk/README.md | 69 +
configuration/csharp/sdk/makefile | 2 +
configuration/csharp/sdk/order-processor/Program.cs | 57 +
configuration/csharp/sdk/order-processor/Program.csproj | 12 +
configuration/go/http/README.md | 73 +
configuration/go/http/makefile | 2 +
configuration/go/http/order-processor/app.go | 138 ++
configuration/go/http/order-processor/go.mod | 5 +
configuration/go/http/order-processor/go.sum | 2 +
configuration/go/sdk/README.md | 69 +
configuration/go/sdk/makefile | 2 +
configuration/go/sdk/order-processor/app.go | 67 +
configuration/go/sdk/order-processor/go.mod | 18 +
configuration/go/sdk/order-processor/go.sum | 156 ++
configuration/java/http/README.md | 92 +
configuration/java/http/makefile | 2 +
configuration/java/http/order-processor/pom.xml | 41 +
.../http/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java | 113 ++
.../src/main/java/com/service/controller/OrderProcessingServiceController.java | 24 +
configuration/java/sdk/README.md | 89 +
configuration/java/sdk/makefile | 2 +
configuration/java/sdk/order-processor/pom.xml | 51 +
.../sdk/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java | 60 +
configuration/javascript/http/.gitignore | 5 +
configuration/javascript/http/README.md | 83 +
configuration/javascript/http/makefile | 2 +
configuration/javascript/http/order-processor/index.js | 82 +
configuration/javascript/http/order-processor/package-lock.json | 4258 +++++++++++++++++++++++++++++++++++++++
configuration/javascript/http/order-processor/package.json | 21 +
configuration/javascript/sdk/.gitignore | 5 +
configuration/javascript/sdk/README.md | 80 +
configuration/javascript/sdk/makefile | 2 +
configuration/javascript/sdk/order-processor/index.js | 44 +
configuration/javascript/sdk/order-processor/package-lock.json | 4762 ++++++++++++++++++++++++++++++++++++++++++++
configuration/javascript/sdk/order-processor/package.json | 20 +
configuration/python/http/README.md | 82 +
configuration/python/http/makefile | 2 +
configuration/python/http/order-processor/app.py | 59 +
configuration/python/http/order-processor/requirements.txt | 2 +
configuration/python/sdk/README.md | 80 +
configuration/python/sdk/makefile | 2 +
configuration/python/sdk/order-processor/app.py | 49 +
configuration/python/sdk/order-processor/requirements.txt | 1 +
pub_sub/csharp/http/README.md | 7 +-
pub_sub/csharp/http/makefile | 3 -
pub_sub/csharp/sdk/README.md | 7 +-
pub_sub/csharp/sdk/checkout/checkout.csproj | 2 +-
pub_sub/csharp/sdk/makefile | 3 -
pub_sub/csharp/sdk/order-processor/Properties/launchSettings.json | 2 +-
pub_sub/csharp/sdk/order-processor/order-processor.csproj | 2 +-
pub_sub/go/http/README.md | 10 +-
pub_sub/go/http/checkout/dapr_example | Bin 0 -> 6523040 bytes
pub_sub/go/http/checkout/go.mod | 15 +-
pub_sub/go/http/checkout/go.sum | 0
pub_sub/go/http/makefile | 3 -
pub_sub/go/http/order-processor/app.go | 2 +-
pub_sub/go/http/order-processor/dapr_example | Bin 0 -> 6976880 bytes
pub_sub/go/http/order-processor/go.mod | 15 +-
pub_sub/go/http/order-processor/go.sum | 109 -
pub_sub/go/sdk/README.md | 12 +-
pub_sub/go/sdk/checkout/dapr_example | Bin 0 -> 13237984 bytes
pub_sub/go/sdk/checkout/go.mod | 16 +-
pub_sub/go/sdk/checkout/go.sum | 65 +-
pub_sub/go/sdk/makefile | 3 -
pub_sub/go/sdk/order-processor/app.go | 2 +-
pub_sub/go/sdk/order-processor/dapr_example | Bin 0 -> 15150672 bytes
pub_sub/go/sdk/order-processor/go.mod | 16 +-
pub_sub/go/sdk/order-processor/go.sum | 65 +-
pub_sub/java/http/README.md | 8 +-
pub_sub/java/http/makefile | 3 -
pub_sub/java/sdk/README.md | 8 +-
pub_sub/java/sdk/checkout/pom.xml | 2 +-
pub_sub/java/sdk/makefile | 3 -
pub_sub/java/sdk/order-processor/pom.xml | 4 +-
pub_sub/javascript/http/README.md | 8 +-
pub_sub/javascript/http/makefile | 3 -
pub_sub/javascript/sdk/README.md | 8 +-
pub_sub/javascript/sdk/makefile | 3 -
pub_sub/javascript/sdk/order-processor/index.js | 2 +-
pub_sub/python/http/README.md | 8 +-
pub_sub/python/http/makefile | 3 -
pub_sub/python/http/order-processor/app.py | 5 +-
pub_sub/python/sdk/README.md | 8 +-
pub_sub/python/sdk/makefile | 3 -
pub_sub/python/sdk/order-processor/app.py | 4 +-
resiliency/README.md | 9 +
resiliency/service-to-component-resiliency.md | 123 ++
resiliency/service-to-service-resiliency.md | 189 ++
secrets_management/csharp/http/makefile | 3 -
secrets_management/csharp/sdk/makefile | 3 -
secrets_management/csharp/sdk/order-processor/Program.csproj | 2 +-
secrets_management/go/http/README.md | 2 +-
secrets_management/go/http/makefile | 3 -
secrets_management/go/http/order-processor/dapr_example | Bin 0 -> 6166178 bytes
secrets_management/go/http/order-processor/go.mod | 15 +-
secrets_management/go/http/order-processor/go.sum | 111 --
secrets_management/go/sdk/README.md | 2 +-
secrets_management/go/sdk/makefile | 3 -
secrets_management/go/sdk/order-processor/dapr_example | Bin 0 -> 12803138 bytes
secrets_management/go/sdk/order-processor/go.mod | 21 +-
secrets_management/go/sdk/order-processor/go.sum | 96 +-
secrets_management/java/http/makefile | 5 +-
secrets_management/java/http/order-processor/pom.xml | 1 +
secrets_management/java/sdk/makefile | 5 +-
secrets_management/java/sdk/order-processor/pom.xml | 3 +-
secrets_management/javascript/http/makefile | 3 -
secrets_management/javascript/sdk/makefile | 3 -
secrets_management/python/http/makefile | 3 -
secrets_management/python/sdk/makefile | 3 -
service_invocation/components/resiliency.yaml | 26 +
service_invocation/csharp/http/checkout/Program.cs | 2 +-
service_invocation/csharp/http/config.yaml | 8 +
service_invocation/csharp/http/makefile | 2 +
service_invocation/go/http/README.md | 8 +-
service_invocation/go/http/checkout/app.go | 2 +-
service_invocation/go/http/checkout/dapr_example | Bin 0 -> 6166130 bytes
service_invocation/go/http/checkout/go.mod | 15 +-
service_invocation/go/http/checkout/go.sum | 111 --
service_invocation/go/http/config.yaml | 8 +
service_invocation/go/http/makefile | 3 -
service_invocation/go/http/order-processor/dapr_example | Bin 0 -> 6311378 bytes
service_invocation/go/http/order-processor/go.mod | 15 +-
service_invocation/go/http/order-processor/go.sum | 109 -
service_invocation/java/http/checkout/pom.xml | 1 +
.../java/http/checkout/src/main/java/com/service/CheckoutServiceApplication.java | 2 +-
service_invocation/java/http/config.yaml | 8 +
service_invocation/java/http/makefile | 3 -
service_invocation/javascript/http/checkout/index.js | 2 +-
service_invocation/javascript/http/config.yaml | 8 +
service_invocation/javascript/http/makefile | 3 -
service_invocation/python/http/README.md | 2 +-
service_invocation/python/http/checkout/app.py | 2 +-
service_invocation/python/http/config.yaml | 8 +
service_invocation/python/http/makefile | 3 -
service_invocation/python/http/order-processor/app.py | 2 +-
state_management/components/resiliency.yaml | 27 +
state_management/csharp/http/makefile | 3 -
state_management/csharp/http/order-processor/Program.cs | 2 +-
state_management/csharp/sdk/config.yaml | 8 +
state_management/csharp/sdk/makefile | 3 -
state_management/csharp/sdk/order-processor/Program.cs | 2 +-
state_management/csharp/sdk/order-processor/Program.csproj | 2 +-
state_management/go/http/README.md | 4 +-
state_management/go/http/makefile | 3 -
state_management/go/http/order-processor/app.go | 2 +-
state_management/go/http/order-processor/dapr_example | Bin 0 -> 6309474 bytes
state_management/go/http/order-processor/go.mod | 15 +-
state_management/go/http/order-processor/go.sum | 111 --
state_management/go/sdk/README.md | 4 +-
state_management/go/sdk/config.yaml | 8 +
state_management/go/sdk/makefile | 3 -
state_management/go/sdk/order-processor/app.go | 8 +-
state_management/go/sdk/order-processor/dapr_example | Bin 0 -> 12803218 bytes
state_management/go/sdk/order-processor/go.mod | 21 +-
state_management/go/sdk/order-processor/go.sum | 96 +-
state_management/java/http/makefile | 3 -
state_management/java/http/order-processor/pom.xml | 3 +-
.../http/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java | 2 +-
state_management/java/sdk/config.yaml | 8 +
state_management/java/sdk/makefile | 3 -
state_management/java/sdk/order-processor/pom.xml | 3 +-
.../sdk/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java | 2 +-
state_management/javascript/http/makefile | 3 -
state_management/javascript/http/order-processor/index.js | 2 +-
state_management/javascript/sdk/config.yaml | 8 +
state_management/javascript/sdk/makefile | 3 -
state_management/javascript/sdk/order-processor/index.js | 2 +-
state_management/python/http/makefile | 3 -
state_management/python/http/order-processor/app.py | 2 +-
state_management/python/sdk/config.yaml | 8 +
state_management/python/sdk/makefile | 3 -
state_management/python/sdk/order-processor/app.py | 2 +-
tutorials/distributed-calculator/README.md | 48 +-
tutorials/distributed-calculator/csharp/Properties/launchSettings.json | 2 +-
tutorials/distributed-calculator/go/go | Bin 0 -> 6765968 bytes
tutorials/distributed-calculator/go/go.mod | 2 +-
tutorials/distributed-calculator/react-calculator/package.json | 2 +-
tutorials/distributed-calculator/react-calculator/server.js | 86 +-
tutorials/hello-kubernetes/README.md | 4 +-
tutorials/pub-sub/README.md | 6 +-
tutorials/pub-sub/csharp-subscriber/csharp-subscriber.csproj | 2 +-
209 files changed, 12208 insertions(+), 1300 deletions(-)
create mode 100644 .github/workflows/validate_new_quickstarts_configuration.yaml
create mode 100755 bindings/go/http/batch/dapr_output_binding
create mode 100755 bindings/go/sdk/batch/dapr_output_binding
create mode 100644 configuration/components/configstore.yaml
create mode 100644 configuration/csharp/http/README.md
create mode 100644 configuration/csharp/http/makefile
create mode 100644 configuration/csharp/http/order-processor/Program.cs
create mode 100644 configuration/csharp/http/order-processor/Program.csproj
create mode 100644 configuration/csharp/http/order-processor/Properties/launchSettings.json
create mode 100644 configuration/csharp/http/order-processor/appsettings.Development.json
create mode 100644 configuration/csharp/http/order-processor/appsettings.json
create mode 100644 configuration/csharp/sdk/README.md
create mode 100644 configuration/csharp/sdk/makefile
create mode 100644 configuration/csharp/sdk/order-processor/Program.cs
create mode 100644 configuration/csharp/sdk/order-processor/Program.csproj
create mode 100644 configuration/go/http/README.md
create mode 100644 configuration/go/http/makefile
create mode 100644 configuration/go/http/order-processor/app.go
create mode 100644 configuration/go/http/order-processor/go.mod
create mode 100644 configuration/go/http/order-processor/go.sum
create mode 100644 configuration/go/sdk/README.md
create mode 100644 configuration/go/sdk/makefile
create mode 100644 configuration/go/sdk/order-processor/app.go
create mode 100644 configuration/go/sdk/order-processor/go.mod
create mode 100644 configuration/go/sdk/order-processor/go.sum
create mode 100644 configuration/java/http/README.md
create mode 100644 configuration/java/http/makefile
create mode 100644 configuration/java/http/order-processor/pom.xml
create mode 100644 configuration/java/http/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java
create mode 100644 configuration/java/http/order-processor/src/main/java/com/service/controller/OrderProcessingServiceController.java
create mode 100644 configuration/java/sdk/README.md
create mode 100644 configuration/java/sdk/makefile
create mode 100644 configuration/java/sdk/order-processor/pom.xml
create mode 100644 configuration/java/sdk/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java
create mode 100644 configuration/javascript/http/.gitignore
create mode 100644 configuration/javascript/http/README.md
create mode 100644 configuration/javascript/http/makefile
create mode 100644 configuration/javascript/http/order-processor/index.js
create mode 100644 configuration/javascript/http/order-processor/package-lock.json
create mode 100644 configuration/javascript/http/order-processor/package.json
create mode 100644 configuration/javascript/sdk/.gitignore
create mode 100644 configuration/javascript/sdk/README.md
create mode 100644 configuration/javascript/sdk/makefile
create mode 100644 configuration/javascript/sdk/order-processor/index.js
create mode 100644 configuration/javascript/sdk/order-processor/package-lock.json
create mode 100644 configuration/javascript/sdk/order-processor/package.json
create mode 100644 configuration/python/http/README.md
create mode 100644 configuration/python/http/makefile
create mode 100644 configuration/python/http/order-processor/app.py
create mode 100644 configuration/python/http/order-processor/requirements.txt
create mode 100644 configuration/python/sdk/README.md
create mode 100644 configuration/python/sdk/makefile
create mode 100644 configuration/python/sdk/order-processor/app.py
create mode 100644 configuration/python/sdk/order-processor/requirements.txt
create mode 100755 pub_sub/go/http/checkout/dapr_example
create mode 100644 pub_sub/go/http/checkout/go.sum
create mode 100755 pub_sub/go/http/order-processor/dapr_example
create mode 100755 pub_sub/go/sdk/checkout/dapr_example
create mode 100755 pub_sub/go/sdk/order-processor/dapr_example
create mode 100644 resiliency/README.md
create mode 100644 resiliency/service-to-component-resiliency.md
create mode 100644 resiliency/service-to-service-resiliency.md
create mode 100755 secrets_management/go/http/order-processor/dapr_example
create mode 100755 secrets_management/go/sdk/order-processor/dapr_example
create mode 100644 service_invocation/components/resiliency.yaml
create mode 100644 service_invocation/csharp/http/config.yaml
create mode 100644 service_invocation/csharp/http/makefile
create mode 100755 service_invocation/go/http/checkout/dapr_example
create mode 100644 service_invocation/go/http/config.yaml
create mode 100755 service_invocation/go/http/order-processor/dapr_example
create mode 100644 service_invocation/java/http/config.yaml
create mode 100644 service_invocation/javascript/http/config.yaml
create mode 100644 service_invocation/python/http/config.yaml
create mode 100644 state_management/components/resiliency.yaml
create mode 100644 state_management/csharp/sdk/config.yaml
create mode 100755 state_management/go/http/order-processor/dapr_example
create mode 100644 state_management/go/sdk/config.yaml
create mode 100755 state_management/go/sdk/order-processor/dapr_example
create mode 100644 state_management/java/sdk/config.yaml
create mode 100644 state_management/javascript/sdk/config.yaml
create mode 100644 state_management/python/sdk/config.yaml
create mode 100755 tutorials/distributed-calculator/go/go
Adding Pre-Reqs
For the rest to work, we’ll need some form of a state store. The easiest to start is a quick Redis instance.
I’ll add the chart and update
$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" already exists with the same configuration, skipping
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "longhorn" chart repository
...Successfully got an update from the "confluentinc" chart repository
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "dapr" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "adwerx" chart repository
...Successfully got an update from the "uptime-kuma" chart repository
...Successfully got an update from the "rhcharts" chart repository
...Successfully got an update from the "myharbor" chart repository
...Successfully got an update from the "sonarqube" chart repository
...Successfully got an update from the "novum-rgi-helm" chart repository
...Successfully got an update from the "kubecost" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "sumologic" chart repository
...Successfully got an update from the "epsagon" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "lifen-charts" chart repository
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "argo-cd" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "rancher-latest" chart repository
...Successfully got an update from the "crossplane-stable" chart repository
...Successfully got an update from the "newrelic" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Then install
$ helm install redis bitnami/redis
NAME: redis
LAST DEPLOYED: Fri Dec 16 07:03:10 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis
CHART VERSION: 17.3.16
APP VERSION: 7.0.6
** Please be patient while the chart is being deployed **
Redis® can be accessed on the following DNS names from within your cluster:
redis-master.default.svc.cluster.local for read/write operations (port 6379)
redis-replicas.default.svc.cluster.local for read-only operations (port 6379)
To get your password run:
export REDIS_PASSWORD=$(kubectl get secret --namespace default redis -o jsonpath="{.data.redis-password}" | base64 -d)
To connect to your Redis® server:
1. Run a Redis® pod that you can use as a client:
kubectl run --namespace default redis-client --restart='Never' --env REDIS_PASSWORD=$REDIS_PASSWORD --image docker.io/bitnami/redis:7.0.6-debian-11-r0 --command -- sleep infinity
Use the following command to attach to the pod:
kubectl exec --tty -i redis-client \
--namespace default -- bash
2. Connect using the Redis® CLI:
REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-master
REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-replicas
To connect to your database from outside the cluster execute the following commands:
kubectl port-forward --namespace default svc/redis-master 6379:6379 &
REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h 127.0.0.1 -p 6379
I can get the Redis password from the secret set
$ kubectl get secrets redis -o json | jq -r '.data | .["redis-password"]' | base64 --decode && echo
lXfNgkyt4K
This can be used in our pubsub component definition to create the pubsub.redis component:
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/pub_sub/components$ cat pubsub.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: orderpubsub
spec:
type: pubsub.redis
version: v1
metadata:
- name: redisHost
value: "redis-master:6379"
- name: redisPassword
value: "lXfNgkyt4K"
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/pub_sub/components$ kubectl apply -f pubsub.yaml
component.dapr.io/orderpubsub created
I should point out that you do not need to expose your password as such. You can just pull from the secret store directly
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/pub-sub/deploy$ cat redis.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.redis
version: v1
metadata:
# These settings will work out of the box if you use `helm install
# bitnami/redis`. If you have your own setup, replace
# `redis-master:6379` with your own Redis master address, and the
# Redis password with your own Secret's name. For more information,
# see https://docs.dapr.io/operations/components/component-secrets .
- name: redisHost
value: redis-master:6379
- name: redisPassword
secretKeyRef:
name: redis
key: redis-password
auth:
secretStore: kubernetes
Deploy the Kubernetes quickstart
If you want to try the basic Dapr hello-kubernetes quickstart, you can do so using the “hello-kubernetes” tutorial. While I will install it, we won’t be using it for testing APM.
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/hello-kubernetes/deploy$ kubectl apply -f .
service/nodeapp created
deployment.apps/nodeapp created
deployment.apps/pythonapp created
component.dapr.io/statestore created
I now have the tutorial running
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/hello-kubernetes/deploy$ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-master-0 1/1 Running 0 18m
redis-replicas-0 1/1 Running 0 18m
redis-replicas-1 1/1 Running 0 17m
redis-replicas-2 1/1 Running 0 17m
csharp-subscriber-66b7c5bcbc-jrxf4 2/2 Running 0 3m46s
pythonapp-7c9b7f7966-6jkfc 2/2 Running 0 36s
nodeapp-679885bdf8-4d4sc 2/2 Running 0 36s
And I can see it running
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/hello-kubernetes/deploy$ kubectl logs nodeapp-679885bdf8-4d4sc | tail -n 10
Defaulted container "node" out of: node, daprd
Got a new order! Order ID: 77
Successfully persisted state.
Got a new order! Order ID: 78
Successfully persisted state.
Got a new order! Order ID: 79
Successfully persisted state.
Got a new order! Order ID: 80
Successfully persisted state.
Got a new order! Order ID: 81
Successfully persisted state.
Observability
We’ll want to also setup the basic calculator app with zipkin.
Initially, we can send traces to zipkin (the app/service) using the zipkin format.
The zipkin service is defined in the zipkin.yaml
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ cat zipkin.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: zipkin
labels:
app: zipkin
spec:
replicas: 1
selector:
matchLabels:
app: zipkin
template:
metadata:
labels:
app: zipkin
spec:
containers:
- name: zipkin
image: openzipkin/zipkin
ports:
- containerPort: 9411
---
kind: Service
apiVersion: v1
metadata:
name: zipkin
labels:
app: zipkin
spec:
selector:
app: zipkin
ports:
- protocol: TCP
port: 9411
targetPort: 9411
type: ClusterIP
The “appconfig” configuration will be setup to send our zipkin (format) traces to the Zipkin service
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ cat appconfig.yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
name: appconfig
spec:
tracing:
samplingRate: "1"
zipkin:
endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
Later, should we jump into OpenTelemetry, we will use the endpointAddress of our OTel collector.
The app we’ll use to test this setup is the Calculator app.
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ cat python-multiplier.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: multiplyapp
labels:
app: multiply
spec:
replicas: 1
selector:
matchLabels:
app: multiply
template:
metadata:
labels:
app: multiply
annotations:
dapr.io/enabled: "true"
dapr.io/app-id: "multiplyapp"
dapr.io/app-port: "5001"
dapr.io/config: "appconfig"
spec:
containers:
- name: multiply
image: ghcr.io/dapr/samples/distributed-calculator-slow-python:latest
env:
- name: APP_PORT
value: "5001"
ports:
- containerPort: 5001
imagePullPolicy: Always
The part that makes it actually send trace data is setup in the annotatations
annotations:
dapr.io/enabled: "true"
dapr.io/app-id: "multiplyapp"
dapr.io/app-port: "5001"
dapr.io/config: "appconfig"
Should we wish to stop sending trace data, we merely remove or change the value for config
dapr.io/config: "somethingelse"
We can disable Dapr on it altogether (provided we did not annotate the namespace) with
dapr.io/enabled: "false"
We’ll apply the whole lot of YAML
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ kubectl apply -f .
configuration.dapr.io/appconfig created
deployment.apps/multiplyapp created
deployment.apps/zipkin created
service/zipkin created
We can the test the Zipkin service with a port-forward
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ kubectl port-forward svc/zipkin 19111:9411
Forwarding from 127.0.0.1:19111 -> 9411
Forwarding from [::1]:19111 -> 9411
Handling connection for 19111
Handling connection for 19111
Handling connection for 19111
Handling connection for 19111
We can then port forward to the calculator and run some operations.
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/distributed-calculator/deploy$ kubectl port-forward service/calculator-front-end 8000:80
Forwarding from 127.0.0.1:8000 -> 8080
Forwarding from [::1]:8000 -> 8080
Going back to the Zipkin service, we can see some maps in the dependencies page
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/distributed-calculator/deploy$ !1965
kubectl port-forward svc/zipkin 19111:9411
Forwarding from 127.0.0.1:19111 -> 9411
Forwarding from [::1]:19111 -> 9411
Handling connection for 19111
Handling connection for 19111
We can also query the serviceName
We can view any given trace details as well
For instance, with the above we see the front end called addapp which lasted 1.717 seconds.
Setting up for GCP Cloud Trace
For the collector, we’ll need a service account
We’ll set a name
Then we can add the Trace Agent role
Then click through to Done.
We can find the Service Account in the list and click “Manage Keys”
Then “Create new key”
And chose JSON
This will create and download the JSON
I can use that to create the configmap.
(Note: had I used a more generic name, like “credentials.json”, my later env variable for GOOGLE_APPLICATION_CREDENTIALS would use the same name. As I used my downloaded file, we’ll see I have to match filenames for GOOGLE_APPLICATION_CREDENTIALS in my Deployment YAML)
$ kubectl create configmap gcptracecred --from-file=/mnt/c/Users/isaac/Downloads/myanthosproject2-eea0dc5b11be.json
configmap/gcptracecred created
Now that we loaded the configmap, we can use it in the deployment yaml for the zip collector. Only this time we have some environment variables to set and a volume to mount
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/distributed-calculator/deploy$ cat gcp-zipkin-agent.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: zipkin
labels:
app: zipkin
spec:
replicas: 1
selector:
matchLabels:
app: zipkin
template:
metadata:
labels:
app: zipkin
spec:
containers:
- name: zipkin
image: openzipkin/zipkin-gcp
env:
- name: STORAGE_TYPE
value: stackdriver
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /zipkin/.gcp/myanthosproject2-eea0dc5b11be.json
- name: STACKDRIVER_PROJECT_ID
value: myanthosproject2
volumeMounts:
- name: gcp-volume
mountPath: /zipkin/.gcp
ports:
- containerPort: 9411
volumes:
- name: gcp-volume
configMap:
name: gcptracecred
---
kind: Service
apiVersion: v1
metadata:
name: zipkin
labels:
app: zipkin
spec:
selector:
app: zipkin
ports:
- protocol: TCP
port: 9411
targetPort: 9411
type: ClusterIP
$ kubectl apply -f gcp-zipkin-agent.yaml
deployment.apps/zipkin configured
service/zipkin unchanged
I can see it running
$ kubectl logs zipkin-57797dd5db-mjl9v
oo
oooo
oooooo
oooooooo
oooooooooo
oooooooooooo
ooooooo ooooooo
oooooo ooooooo
oooooo ooooooo
oooooo o o oooooo
oooooo oo oo oooooo
ooooooo oooo oooo ooooooo
oooooo ooooo ooooo ooooooo
oooooo oooooo oooooo ooooooo
oooooooo oo oo oooooooo
ooooooooooooo oo oo ooooooooooooo
oooooooooooo oooooooooooo
oooooooo oooooooo
oooo oooo
________ ____ _ _____ _ _
|__ /_ _| _ \| |/ /_ _| \ | |
/ / | || |_) | ' / | || \| |
/ /_ | || __/| . \ | || |\ |
|____|___|_| |_|\_\___|_| \_|
:: version 2.23.16 :: commit b90f2b3 ::
2022-12-18 22:42:35.990 INFO [/] 1 --- [oss-http-*:9411] c.l.a.s.Server : Serving HTTP at /[0:0:0:0:0:0:0:0%0]:9411 - http://127.0.0.1:9411/
I’ll run a few more operations on the calculator
kubectl port-forward service/calculator-front-end 8000:80
Forwarding from 127.0.0.1:8000 -> 8080
Forwarding from [::1]:8000 -> 8080
Handling connection for 8000
Handling connection for 8000
Handling connection for 8000
Handling connection for 8000
Viewing trace data in Cloud Trace
We can now go back to our GCP project and see it properly captured traces
I can view specific traces
In GCP, Traces have logs. So one can pull up trace data in Log explorer as well
clicking view shows the logs:
Trace Reports
I made a few traces so I could create a reasonable report
On Analysis reports, I created one for the last hour
We can see latency as it breaks down by percentage of requests
The above tells us the median is just shy of 2ms.
Usage Metrics
We can view usage metrics (from the last month) in Metrics Explorer.
We can look at just Spans Ingested as well in Metrics Explorer
Alerting
We could use the metric of “Trace Spans Ingested” to trigger an alert policy.
For instance, say I wanted to check the max (total) spans in a given hour
If it exceeds 150, for instance
I could then email myself with a note
I give it a name
Then lastly review the policy
I can click “Create Policy” if i wish to create it, or view the JSON with “view code”
{
"displayName": "TooManySpans",
"documentation": {
"content": "Something is getting noisy - more than our allowed max spans in an hour!",
"mimeType": "text/markdown"
},
"userLabels": {},
"conditions": [
{
"displayName": "Global - Trace spans ingested",
"conditionThreshold": {
"filter": "resource.type = \"global\" AND metric.type = \"cloudtrace.googleapis.com/billing/spans_ingested\"",
"aggregations": [
{
"alignmentPeriod": "3600s",
"crossSeriesReducer": "REDUCE_NONE",
"perSeriesAligner": "ALIGN_MAX"
}
],
"comparison": "COMPARISON_GT",
"duration": "0s",
"trigger": {
"count": 1
},
"thresholdValue": 150
}
}
],
"alertStrategy": {
"autoClose": "604800s"
},
"combiner": "OR",
"enabled": true,
"notificationChannels": [
"projects/myanthosproject2/notificationChannels/14548337521871198017"
]
}
I should note that while I have email setup presently, there are plenty of Notification Channels including Slack, Webhooks and PagerDuty.
Dashboard
We can turn any metric into a Dashboard.
Just save your metric from the Metrics Explorer to a new (or existing) dashboard.
In the dashboard, we can view for different time spans
Pricing
We really have to hammer Cloud Trace to incur costs. The first 2.5 million spans a month are free and it’s just US$0.20 a million beyond that.
While GCP Monitoring is not free, it’s charged my data ingested for monitoring. So, I believe, in our case it’s not going to cost us.
Summary
Today we respun an on-prem K3s cluster then loaded it with Dapr. We used the Observability tutorial to show an auto-instrumented calculator app sending zipkin formatted traces to zipkin. We then pivoted to using the GCP Zipkin collector partnered with a proper Service Account to route the traces to GCP Cloud Trace. Lastly, we explored several features of cloud trace including trace details, reports, metrics and alerting.
GCP Cloud Trace is a nice low-cost option for viewing traces. It can gather metrics and use alerting. We can make a very basic report, graph and dashboard.
For a product that is essentially free for moderate use, I’ll not complain. I can’t create warn vs alert thresholds nor can I see service maps
