GCP Cloud Trace

Published: Dec 20, 2022 by Isaac Johnson

Today we’ll use ansible to rebuild an on-prem k3s cluster. We’ll then load it with a Redis instance and Dapr. We’ll deploy the pub-sub and calculator apps followed by zipkin.

We’ll use Zipkin (the service/app) to collect zipkin (the format) trace data, then pivot to setting up a zipkin forwarder to GCP Cloud Trace. This will require a service principal with the proper role setup.

Lastly, we’ll explore the data collected in Cloud Trace, how we can use it, and lastly some cost considerations.

Let’s dig in!

Resetting the cluster

Before I do anything, I should clean up the Kubernetes cluster with a quick refresh. Luckily, I’ve created playbooks to do just that.

I’ll go to my AWX instance and fire the reloadk3s template. (If you want to read more about that process, you can see a detailed step-by-step here)

/content/images/2022/12/cloudtrace-01.png

This will use the reloadk3s playbook in git to rebuild the cluster fresh.

I had a bit of a hiccup in that my az CLI was logged out, blocking the playbook. At some point I’ll need to sort out refreshing Azure CLI logins on hosts. Until such time, I just hopped onto the k8s master (192.168.1.81) and used az login --use-device-code to refresh (and as root as well).

Once done, I could see my fresh cluster

$ kubectl get nodes
NAME                  STATUS   ROLES                  AGE     VERSION
isaac-macbookpro      Ready    <none>                 4m58s   v1.23.10+k3s1
anna-macbookair       Ready    control-plane,master   45m     v1.23.10+k3s1
builder-macbookpro2   Ready    <none>                 2m6s    v1.23.10+k3s1

Updating Dapr CLI

It’s always a good idea to get the latest and greatest Dapr.io. (Dapr.io is one of my favourite OS frameworks, you can see quite a few past articles here)

builder@DESKTOP-QADGF36:~$ dapr version
CLI version: 1.8.0
Runtime version: n/a

This is a bit out of date. We can easily upgrade our local CLI

builder@DESKTOP-QADGF36:~$ wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
Getting the latest Dapr CLI...
Your system is linux_amd64

Dapr CLI is detected:
CLI version: 1.8.0
Runtime version: n/a
Reinstalling Dapr CLI - /usr/local/bin/dapr...

Installing v1.9.1 Dapr CLI...
Downloading https://github.com/dapr/cli/releases/download/v1.9.1/dapr_linux_amd64.tar.gz ...
[sudo] password for builder:
dapr installed into /usr/local/bin successfully.
CLI version: 1.9.1
Runtime version: n/a

To get started with Dapr, please visit https://docs.dapr.io/getting-started/
builder@DESKTOP-QADGF36:~$ dapr version
CLI version: 1.9.1
Runtime version: n/a

Installing Dapr to K8s

The easiest way (and the recommended) is to just use dapr init -k. This ensures you have the latest and greatest Dapr runtime installed to your Kubernetes cluster

builder@DESKTOP-QADGF36:~$ dapr init -k
⌛  Making the jump to hyperspace...
ℹ️  Note: To install Dapr using Helm, see here: https://docs.dapr.io/getting-started/install-dapr-kubernetes/#install-with-helm-advanced

ℹ️  Container images will be pulled from Docker Hub
✅  Deploying the Dapr control plane to your cluster...
✅  Success! Dapr has been installed to namespace dapr-system. To verify, run `dapr status -k' in your terminal. To get started, go here: https://aka.ms/dapr-getting-started

We can now check our local version and Kubernetes version

builder@DESKTOP-QADGF36:~$ dapr version
CLI version: 1.9.1
Runtime version: n/a
builder@DESKTOP-QADGF36:~$ dapr status -k
  NAME                   NAMESPACE    HEALTHY  STATUS   REPLICAS  VERSION  AGE  CREATED
  dapr-placement-server  dapr-system  True     Running  1         1.9.5    53s  2022-12-16 06:51.10
  dapr-dashboard         dapr-system  True     Running  1         0.11.0   53s  2022-12-16 06:51.10
  dapr-sidecar-injector  dapr-system  True     Running  1         1.9.5    53s  2022-12-16 06:51.10
  dapr-operator          dapr-system  True     Running  1         1.9.5    53s  2022-12-16 06:51.10
  dapr-sentry            dapr-system  True     Running  1         1.9.5    53s  2022-12-16 06:51.10

builder@DESKTOP-QADGF36:~$ kubectl get pods --namespace dapr-system
NAME                                    READY   STATUS    RESTARTS   AGE
dapr-dashboard-6b868c46fb-ldplf         1/1     Running   0          98s
dapr-sidecar-injector-75948876c-xfwk9   1/1     Running   0          98s
dapr-operator-6bd7f7bfdd-xft76          1/1     Running   0          98s
dapr-placement-server-0                 1/1     Running   0          98s
dapr-sentry-7f8db4ff47-qvmr7            1/1     Running   0          98s

Dapr Quickstarts

We’ll want to get the quickstarts from Dapr. These get updated pretty often (as you can see below)

To pull fresh

$ git clone https://github.com/dapr/quickstarts.git

Or, in my case, updating

builder@DESKTOP-QADGF36:~/Workspaces$ cd quickstarts/
builder@DESKTOP-QADGF36:~/Workspaces/quickstarts$ git pull
remote: Enumerating objects: 913, done.
remote: Counting objects: 100% (226/226), done.
remote: Total 913 (delta 226), reused 226 (delta 226), pack-reused 687
Receiving objects: 100% (913/913), 75.13 MiB | 8.28 MiB/s, done.
Resolving deltas: 100% (473/473), completed with 70 local objects.
From https://github.com/dapr/quickstarts
   95f81e6..b0b9a5a  master               -> origin/master
 * [new branch]      dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/express-4.17.3 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/express-4.17.3
 * [new branch]      dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/qs-6.11.0 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/http/order-processor/qs-6.11.0
 * [new branch]      dependabot/npm_and_yarn/pub_sub/javascript/sdk/checkout/qs-and-body-parser-6.11.0 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/sdk/checkout/qs-and-body-parser-6.11.0
 * [new branch]      dependabot/npm_and_yarn/pub_sub/javascript/sdk/order-processor/qs-and-body-parser-6.11.0 -> origin/dependabot/npm_and_yarn/pub_sub/javascript/sdk/order-processor/qs-and-body-parser-6.11.0
 * [new branch]      dependabot/npm_and_yarn/secrets_management/javascript/sdk/order-processor/qs-and-body-parser-6.11.0 -> origin/dependabot/npm_and_yarn/secrets_management/javascript/sdk/order-processor/qs-and-body-parser-6.11.0
 * [new branch]      dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/express-4.18.2 -> origin/dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/express-4.18.2
 * [new branch]      dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/qs-and-express-6.11.0 -> origin/dependabot/npm_and_yarn/tutorials/distributed-calculator/react-calculator/client/qs-and-express-6.11.0
 * [new branch]      dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/express-4.18.2 -> origin/dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/express-4.18.2
 * [new branch]      dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/qs-and-express-6.11.0 -> origin/dependabot/npm_and_yarn/tutorials/pub-sub/react-form/client/qs-and-express-6.11.0
 * [new branch]      greenie-msft-patch-1 -> origin/greenie-msft-patch-1
 * [new branch]      release-1.9          -> origin/release-1.9
 * [new branch]      yaron2-patch-1       -> origin/yaron2-patch-1
 * [new tag]         v1.9.0               -> v1.9.0
Updating 95f81e6..b0b9a5a
Fast-forward
 .github/workflows/validate.yaml                                                            |   11 +-
 .github/workflows/validate_new_quickstarts_bindings.yaml                                   |   20 +-
 .github/workflows/validate_new_quickstarts_configuration.yaml                              |  103 +
 .github/workflows/validate_new_quickstarts_pubsub.yaml                                     |   68 +-
 .github/workflows/validate_new_quickstarts_secrets_management.yaml                         |   65 +-
 .github/workflows/validate_new_quickstarts_service_invo.yaml                               |   47 +-
 .github/workflows/validate_new_quickstarts_state_management.yaml                           |   65 +-
 README.md                                                                                  |    5 +-
 bindings/csharp/sdk/batch/batch.csproj                                                     |    2 +-
 bindings/go/http/README.md                                                                 |    4 +-
 bindings/go/http/batch/app.go                                                              |   10 +-
 bindings/go/http/batch/dapr_output_binding                                                 |  Bin 0 -> 7212034 bytes
 bindings/go/http/batch/go.mod                                                              |   15 +-
 bindings/go/sdk/README.md                                                                  |    4 +-
 bindings/go/sdk/batch/app.go                                                               |   10 +-
 bindings/go/sdk/batch/dapr_output_binding                                                  |  Bin 0 -> 14505250 bytes
 bindings/go/sdk/batch/go.mod                                                               |   27 +-
 bindings/go/sdk/batch/go.sum                                                               |   94 +-
 bindings/java/sdk/batch/pom.xml                                                            |    4 +-
 bindings/javascript/sdk/README.md                                                          |    2 +-
 configuration/components/configstore.yaml                                                  |   12 +
 configuration/csharp/http/README.md                                                        |   71 +
 configuration/csharp/http/makefile                                                         |    2 +
 configuration/csharp/http/order-processor/Program.cs                                       |   94 +
 configuration/csharp/http/order-processor/Program.csproj                                   |   15 +
 configuration/csharp/http/order-processor/Properties/launchSettings.json                   |   15 +
 configuration/csharp/http/order-processor/appsettings.Development.json                     |    9 +
 configuration/csharp/http/order-processor/appsettings.json                                 |   10 +
 configuration/csharp/sdk/README.md                                                         |   69 +
 configuration/csharp/sdk/makefile                                                          |    2 +
 configuration/csharp/sdk/order-processor/Program.cs                                        |   57 +
 configuration/csharp/sdk/order-processor/Program.csproj                                    |   12 +
 configuration/go/http/README.md                                                            |   73 +
 configuration/go/http/makefile                                                             |    2 +
 configuration/go/http/order-processor/app.go                                               |  138 ++
 configuration/go/http/order-processor/go.mod                                               |    5 +
 configuration/go/http/order-processor/go.sum                                               |    2 +
 configuration/go/sdk/README.md                                                             |   69 +
 configuration/go/sdk/makefile                                                              |    2 +
 configuration/go/sdk/order-processor/app.go                                                |   67 +
 configuration/go/sdk/order-processor/go.mod                                                |   18 +
 configuration/go/sdk/order-processor/go.sum                                                |  156 ++
 configuration/java/http/README.md                                                          |   92 +
 configuration/java/http/makefile                                                           |    2 +
 configuration/java/http/order-processor/pom.xml                                            |   41 +
 .../http/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java  |  113 ++
 .../src/main/java/com/service/controller/OrderProcessingServiceController.java             |   24 +
 configuration/java/sdk/README.md                                                           |   89 +
 configuration/java/sdk/makefile                                                            |    2 +
 configuration/java/sdk/order-processor/pom.xml                                             |   51 +
 .../sdk/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java   |   60 +
 configuration/javascript/http/.gitignore                                                   |    5 +
 configuration/javascript/http/README.md                                                    |   83 +
 configuration/javascript/http/makefile                                                     |    2 +
 configuration/javascript/http/order-processor/index.js                                     |   82 +
 configuration/javascript/http/order-processor/package-lock.json                            | 4258 +++++++++++++++++++++++++++++++++++++++
 configuration/javascript/http/order-processor/package.json                                 |   21 +
 configuration/javascript/sdk/.gitignore                                                    |    5 +
 configuration/javascript/sdk/README.md                                                     |   80 +
 configuration/javascript/sdk/makefile                                                      |    2 +
 configuration/javascript/sdk/order-processor/index.js                                      |   44 +
 configuration/javascript/sdk/order-processor/package-lock.json                             | 4762 ++++++++++++++++++++++++++++++++++++++++++++
 configuration/javascript/sdk/order-processor/package.json                                  |   20 +
 configuration/python/http/README.md                                                        |   82 +
 configuration/python/http/makefile                                                         |    2 +
 configuration/python/http/order-processor/app.py                                           |   59 +
 configuration/python/http/order-processor/requirements.txt                                 |    2 +
 configuration/python/sdk/README.md                                                         |   80 +
 configuration/python/sdk/makefile                                                          |    2 +
 configuration/python/sdk/order-processor/app.py                                            |   49 +
 configuration/python/sdk/order-processor/requirements.txt                                  |    1 +
 pub_sub/csharp/http/README.md                                                              |    7 +-
 pub_sub/csharp/http/makefile                                                               |    3 -
 pub_sub/csharp/sdk/README.md                                                               |    7 +-
 pub_sub/csharp/sdk/checkout/checkout.csproj                                                |    2 +-
 pub_sub/csharp/sdk/makefile                                                                |    3 -
 pub_sub/csharp/sdk/order-processor/Properties/launchSettings.json                          |    2 +-
 pub_sub/csharp/sdk/order-processor/order-processor.csproj                                  |    2 +-
 pub_sub/go/http/README.md                                                                  |   10 +-
 pub_sub/go/http/checkout/dapr_example                                                      |  Bin 0 -> 6523040 bytes
 pub_sub/go/http/checkout/go.mod                                                            |   15 +-
 pub_sub/go/http/checkout/go.sum                                                            |    0
 pub_sub/go/http/makefile                                                                   |    3 -
 pub_sub/go/http/order-processor/app.go                                                     |    2 +-
 pub_sub/go/http/order-processor/dapr_example                                               |  Bin 0 -> 6976880 bytes
 pub_sub/go/http/order-processor/go.mod                                                     |   15 +-
 pub_sub/go/http/order-processor/go.sum                                                     |  109 -
 pub_sub/go/sdk/README.md                                                                   |   12 +-
 pub_sub/go/sdk/checkout/dapr_example                                                       |  Bin 0 -> 13237984 bytes
 pub_sub/go/sdk/checkout/go.mod                                                             |   16 +-
 pub_sub/go/sdk/checkout/go.sum                                                             |   65 +-
 pub_sub/go/sdk/makefile                                                                    |    3 -
 pub_sub/go/sdk/order-processor/app.go                                                      |    2 +-
 pub_sub/go/sdk/order-processor/dapr_example                                                |  Bin 0 -> 15150672 bytes
 pub_sub/go/sdk/order-processor/go.mod                                                      |   16 +-
 pub_sub/go/sdk/order-processor/go.sum                                                      |   65 +-
 pub_sub/java/http/README.md                                                                |    8 +-
 pub_sub/java/http/makefile                                                                 |    3 -
 pub_sub/java/sdk/README.md                                                                 |    8 +-
 pub_sub/java/sdk/checkout/pom.xml                                                          |    2 +-
 pub_sub/java/sdk/makefile                                                                  |    3 -
 pub_sub/java/sdk/order-processor/pom.xml                                                   |    4 +-
 pub_sub/javascript/http/README.md                                                          |    8 +-
 pub_sub/javascript/http/makefile                                                           |    3 -
 pub_sub/javascript/sdk/README.md                                                           |    8 +-
 pub_sub/javascript/sdk/makefile                                                            |    3 -
 pub_sub/javascript/sdk/order-processor/index.js                                            |    2 +-
 pub_sub/python/http/README.md                                                              |    8 +-
 pub_sub/python/http/makefile                                                               |    3 -
 pub_sub/python/http/order-processor/app.py                                                 |    5 +-
 pub_sub/python/sdk/README.md                                                               |    8 +-
 pub_sub/python/sdk/makefile                                                                |    3 -
 pub_sub/python/sdk/order-processor/app.py                                                  |    4 +-
 resiliency/README.md                                                                       |    9 +
 resiliency/service-to-component-resiliency.md                                              |  123 ++
 resiliency/service-to-service-resiliency.md                                                |  189 ++
 secrets_management/csharp/http/makefile                                                    |    3 -
 secrets_management/csharp/sdk/makefile                                                     |    3 -
 secrets_management/csharp/sdk/order-processor/Program.csproj                               |    2 +-
 secrets_management/go/http/README.md                                                       |    2 +-
 secrets_management/go/http/makefile                                                        |    3 -
 secrets_management/go/http/order-processor/dapr_example                                    |  Bin 0 -> 6166178 bytes
 secrets_management/go/http/order-processor/go.mod                                          |   15 +-
 secrets_management/go/http/order-processor/go.sum                                          |  111 --
 secrets_management/go/sdk/README.md                                                        |    2 +-
 secrets_management/go/sdk/makefile                                                         |    3 -
 secrets_management/go/sdk/order-processor/dapr_example                                     |  Bin 0 -> 12803138 bytes
 secrets_management/go/sdk/order-processor/go.mod                                           |   21 +-
 secrets_management/go/sdk/order-processor/go.sum                                           |   96 +-
 secrets_management/java/http/makefile                                                      |    5 +-
 secrets_management/java/http/order-processor/pom.xml                                       |    1 +
 secrets_management/java/sdk/makefile                                                       |    5 +-
 secrets_management/java/sdk/order-processor/pom.xml                                        |    3 +-
 secrets_management/javascript/http/makefile                                                |    3 -
 secrets_management/javascript/sdk/makefile                                                 |    3 -
 secrets_management/python/http/makefile                                                    |    3 -
 secrets_management/python/sdk/makefile                                                     |    3 -
 service_invocation/components/resiliency.yaml                                              |   26 +
 service_invocation/csharp/http/checkout/Program.cs                                         |    2 +-
 service_invocation/csharp/http/config.yaml                                                 |    8 +
 service_invocation/csharp/http/makefile                                                    |    2 +
 service_invocation/go/http/README.md                                                       |    8 +-
 service_invocation/go/http/checkout/app.go                                                 |    2 +-
 service_invocation/go/http/checkout/dapr_example                                           |  Bin 0 -> 6166130 bytes
 service_invocation/go/http/checkout/go.mod                                                 |   15 +-
 service_invocation/go/http/checkout/go.sum                                                 |  111 --
 service_invocation/go/http/config.yaml                                                     |    8 +
 service_invocation/go/http/makefile                                                        |    3 -
 service_invocation/go/http/order-processor/dapr_example                                    |  Bin 0 -> 6311378 bytes
 service_invocation/go/http/order-processor/go.mod                                          |   15 +-
 service_invocation/go/http/order-processor/go.sum                                          |  109 -
 service_invocation/java/http/checkout/pom.xml                                              |    1 +
 .../java/http/checkout/src/main/java/com/service/CheckoutServiceApplication.java           |    2 +-
 service_invocation/java/http/config.yaml                                                   |    8 +
 service_invocation/java/http/makefile                                                      |    3 -
 service_invocation/javascript/http/checkout/index.js                                       |    2 +-
 service_invocation/javascript/http/config.yaml                                             |    8 +
 service_invocation/javascript/http/makefile                                                |    3 -
 service_invocation/python/http/README.md                                                   |    2 +-
 service_invocation/python/http/checkout/app.py                                             |    2 +-
 service_invocation/python/http/config.yaml                                                 |    8 +
 service_invocation/python/http/makefile                                                    |    3 -
 service_invocation/python/http/order-processor/app.py                                      |    2 +-
 state_management/components/resiliency.yaml                                                |   27 +
 state_management/csharp/http/makefile                                                      |    3 -
 state_management/csharp/http/order-processor/Program.cs                                    |    2 +-
 state_management/csharp/sdk/config.yaml                                                    |    8 +
 state_management/csharp/sdk/makefile                                                       |    3 -
 state_management/csharp/sdk/order-processor/Program.cs                                     |    2 +-
 state_management/csharp/sdk/order-processor/Program.csproj                                 |    2 +-
 state_management/go/http/README.md                                                         |    4 +-
 state_management/go/http/makefile                                                          |    3 -
 state_management/go/http/order-processor/app.go                                            |    2 +-
 state_management/go/http/order-processor/dapr_example                                      |  Bin 0 -> 6309474 bytes
 state_management/go/http/order-processor/go.mod                                            |   15 +-
 state_management/go/http/order-processor/go.sum                                            |  111 --
 state_management/go/sdk/README.md                                                          |    4 +-
 state_management/go/sdk/config.yaml                                                        |    8 +
 state_management/go/sdk/makefile                                                           |    3 -
 state_management/go/sdk/order-processor/app.go                                             |    8 +-
 state_management/go/sdk/order-processor/dapr_example                                       |  Bin 0 -> 12803218 bytes
 state_management/go/sdk/order-processor/go.mod                                             |   21 +-
 state_management/go/sdk/order-processor/go.sum                                             |   96 +-
 state_management/java/http/makefile                                                        |    3 -
 state_management/java/http/order-processor/pom.xml                                         |    3 +-
 .../http/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java  |    2 +-
 state_management/java/sdk/config.yaml                                                      |    8 +
 state_management/java/sdk/makefile                                                         |    3 -
 state_management/java/sdk/order-processor/pom.xml                                          |    3 +-
 .../sdk/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java   |    2 +-
 state_management/javascript/http/makefile                                                  |    3 -
 state_management/javascript/http/order-processor/index.js                                  |    2 +-
 state_management/javascript/sdk/config.yaml                                                |    8 +
 state_management/javascript/sdk/makefile                                                   |    3 -
 state_management/javascript/sdk/order-processor/index.js                                   |    2 +-
 state_management/python/http/makefile                                                      |    3 -
 state_management/python/http/order-processor/app.py                                        |    2 +-
 state_management/python/sdk/config.yaml                                                    |    8 +
 state_management/python/sdk/makefile                                                       |    3 -
 state_management/python/sdk/order-processor/app.py                                         |    2 +-
 tutorials/distributed-calculator/README.md                                                 |   48 +-
 tutorials/distributed-calculator/csharp/Properties/launchSettings.json                     |    2 +-
 tutorials/distributed-calculator/go/go                                                     |  Bin 0 -> 6765968 bytes
 tutorials/distributed-calculator/go/go.mod                                                 |    2 +-
 tutorials/distributed-calculator/react-calculator/package.json                             |    2 +-
 tutorials/distributed-calculator/react-calculator/server.js                                |   86 +-
 tutorials/hello-kubernetes/README.md                                                       |    4 +-
 tutorials/pub-sub/README.md                                                                |    6 +-
 tutorials/pub-sub/csharp-subscriber/csharp-subscriber.csproj                               |    2 +-
 209 files changed, 12208 insertions(+), 1300 deletions(-)
 create mode 100644 .github/workflows/validate_new_quickstarts_configuration.yaml
 create mode 100755 bindings/go/http/batch/dapr_output_binding
 create mode 100755 bindings/go/sdk/batch/dapr_output_binding
 create mode 100644 configuration/components/configstore.yaml
 create mode 100644 configuration/csharp/http/README.md
 create mode 100644 configuration/csharp/http/makefile
 create mode 100644 configuration/csharp/http/order-processor/Program.cs
 create mode 100644 configuration/csharp/http/order-processor/Program.csproj
 create mode 100644 configuration/csharp/http/order-processor/Properties/launchSettings.json
 create mode 100644 configuration/csharp/http/order-processor/appsettings.Development.json
 create mode 100644 configuration/csharp/http/order-processor/appsettings.json
 create mode 100644 configuration/csharp/sdk/README.md
 create mode 100644 configuration/csharp/sdk/makefile
 create mode 100644 configuration/csharp/sdk/order-processor/Program.cs
 create mode 100644 configuration/csharp/sdk/order-processor/Program.csproj
 create mode 100644 configuration/go/http/README.md
 create mode 100644 configuration/go/http/makefile
 create mode 100644 configuration/go/http/order-processor/app.go
 create mode 100644 configuration/go/http/order-processor/go.mod
 create mode 100644 configuration/go/http/order-processor/go.sum
 create mode 100644 configuration/go/sdk/README.md
 create mode 100644 configuration/go/sdk/makefile
 create mode 100644 configuration/go/sdk/order-processor/app.go
 create mode 100644 configuration/go/sdk/order-processor/go.mod
 create mode 100644 configuration/go/sdk/order-processor/go.sum
 create mode 100644 configuration/java/http/README.md
 create mode 100644 configuration/java/http/makefile
 create mode 100644 configuration/java/http/order-processor/pom.xml
 create mode 100644 configuration/java/http/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java
 create mode 100644 configuration/java/http/order-processor/src/main/java/com/service/controller/OrderProcessingServiceController.java
 create mode 100644 configuration/java/sdk/README.md
 create mode 100644 configuration/java/sdk/makefile
 create mode 100644 configuration/java/sdk/order-processor/pom.xml
 create mode 100644 configuration/java/sdk/order-processor/src/main/java/com/service/OrderProcessingServiceApplication.java
 create mode 100644 configuration/javascript/http/.gitignore
 create mode 100644 configuration/javascript/http/README.md
 create mode 100644 configuration/javascript/http/makefile
 create mode 100644 configuration/javascript/http/order-processor/index.js
 create mode 100644 configuration/javascript/http/order-processor/package-lock.json
 create mode 100644 configuration/javascript/http/order-processor/package.json
 create mode 100644 configuration/javascript/sdk/.gitignore
 create mode 100644 configuration/javascript/sdk/README.md
 create mode 100644 configuration/javascript/sdk/makefile
 create mode 100644 configuration/javascript/sdk/order-processor/index.js
 create mode 100644 configuration/javascript/sdk/order-processor/package-lock.json
 create mode 100644 configuration/javascript/sdk/order-processor/package.json
 create mode 100644 configuration/python/http/README.md
 create mode 100644 configuration/python/http/makefile
 create mode 100644 configuration/python/http/order-processor/app.py
 create mode 100644 configuration/python/http/order-processor/requirements.txt
 create mode 100644 configuration/python/sdk/README.md
 create mode 100644 configuration/python/sdk/makefile
 create mode 100644 configuration/python/sdk/order-processor/app.py
 create mode 100644 configuration/python/sdk/order-processor/requirements.txt
 create mode 100755 pub_sub/go/http/checkout/dapr_example
 create mode 100644 pub_sub/go/http/checkout/go.sum
 create mode 100755 pub_sub/go/http/order-processor/dapr_example
 create mode 100755 pub_sub/go/sdk/checkout/dapr_example
 create mode 100755 pub_sub/go/sdk/order-processor/dapr_example
 create mode 100644 resiliency/README.md
 create mode 100644 resiliency/service-to-component-resiliency.md
 create mode 100644 resiliency/service-to-service-resiliency.md
 create mode 100755 secrets_management/go/http/order-processor/dapr_example
 create mode 100755 secrets_management/go/sdk/order-processor/dapr_example
 create mode 100644 service_invocation/components/resiliency.yaml
 create mode 100644 service_invocation/csharp/http/config.yaml
 create mode 100644 service_invocation/csharp/http/makefile
 create mode 100755 service_invocation/go/http/checkout/dapr_example
 create mode 100644 service_invocation/go/http/config.yaml
 create mode 100755 service_invocation/go/http/order-processor/dapr_example
 create mode 100644 service_invocation/java/http/config.yaml
 create mode 100644 service_invocation/javascript/http/config.yaml
 create mode 100644 service_invocation/python/http/config.yaml
 create mode 100644 state_management/components/resiliency.yaml
 create mode 100644 state_management/csharp/sdk/config.yaml
 create mode 100755 state_management/go/http/order-processor/dapr_example
 create mode 100644 state_management/go/sdk/config.yaml
 create mode 100755 state_management/go/sdk/order-processor/dapr_example
 create mode 100644 state_management/java/sdk/config.yaml
 create mode 100644 state_management/javascript/sdk/config.yaml
 create mode 100644 state_management/python/sdk/config.yaml
 create mode 100755 tutorials/distributed-calculator/go/go

Adding Pre-Reqs

For the rest to work, we’ll need some form of a state store. The easiest to start is a quick Redis instance.

I’ll add the chart and update

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" already exists with the same configuration, skipping

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "longhorn" chart repository
...Successfully got an update from the "confluentinc" chart repository
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "dapr" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "adwerx" chart repository
...Successfully got an update from the "uptime-kuma" chart repository
...Successfully got an update from the "rhcharts" chart repository
...Successfully got an update from the "myharbor" chart repository
...Successfully got an update from the "sonarqube" chart repository
...Successfully got an update from the "novum-rgi-helm" chart repository
...Successfully got an update from the "kubecost" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "sumologic" chart repository
...Successfully got an update from the "epsagon" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "lifen-charts" chart repository
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "argo-cd" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "rancher-latest" chart repository
...Successfully got an update from the "crossplane-stable" chart repository
...Successfully got an update from the "newrelic" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

Then install

$ helm install redis bitnami/redis
NAME: redis
LAST DEPLOYED: Fri Dec 16 07:03:10 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis
CHART VERSION: 17.3.16
APP VERSION: 7.0.6

** Please be patient while the chart is being deployed **

Redis&reg; can be accessed on the following DNS names from within your cluster:

    redis-master.default.svc.cluster.local for read/write operations (port 6379)
    redis-replicas.default.svc.cluster.local for read-only operations (port 6379)



To get your password run:

    export REDIS_PASSWORD=$(kubectl get secret --namespace default redis -o jsonpath="{.data.redis-password}" | base64 -d)

To connect to your Redis&reg; server:

1. Run a Redis&reg; pod that you can use as a client:

   kubectl run --namespace default redis-client --restart='Never'  --env REDIS_PASSWORD=$REDIS_PASSWORD  --image docker.io/bitnami/redis:7.0.6-debian-11-r0 --command -- sleep infinity

   Use the following command to attach to the pod:

   kubectl exec --tty -i redis-client \
   --namespace default -- bash

2. Connect using the Redis&reg; CLI:
   REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-master
   REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-replicas

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/redis-master 6379:6379 &
    REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h 127.0.0.1 -p 6379

I can get the Redis password from the secret set

$ kubectl get secrets redis -o json | jq -r '.data | .["redis-password"]' | base64 --decode && echo
lXfNgkyt4K

This can be used in our pubsub component definition to create the pubsub.redis component:

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/pub_sub/components$ cat pubsub.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: orderpubsub
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: "redis-master:6379"
  - name: redisPassword
    value: "lXfNgkyt4K"

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/pub_sub/components$ kubectl apply -f pubsub.yaml
component.dapr.io/orderpubsub created

I should point out that you do not need to expose your password as such. You can just pull from the secret store directly

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/pub-sub/deploy$ cat redis.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
spec:
  type: pubsub.redis
  version: v1
  metadata:
  # These settings will work out of the box if you use `helm install
  # bitnami/redis`.  If you have your own setup, replace
  # `redis-master:6379` with your own Redis master address, and the
  # Redis password with your own Secret's name. For more information,
  # see https://docs.dapr.io/operations/components/component-secrets .
  - name: redisHost
    value: redis-master:6379
  - name: redisPassword
    secretKeyRef:
      name: redis
      key: redis-password
auth:
  secretStore: kubernetes

Deploy the Kubernetes quickstart

If you want to try the basic Dapr hello-kubernetes quickstart, you can do so using the “hello-kubernetes” tutorial. While I will install it, we won’t be using it for testing APM.

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/hello-kubernetes/deploy$ kubectl apply -f .
service/nodeapp created
deployment.apps/nodeapp created
deployment.apps/pythonapp created
component.dapr.io/statestore created

I now have the tutorial running

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/hello-kubernetes/deploy$ kubectl get pods
NAME                                 READY   STATUS             RESTARTS      AGE
redis-master-0                       1/1     Running            0             18m
redis-replicas-0                     1/1     Running            0             18m
redis-replicas-1                     1/1     Running            0             17m
redis-replicas-2                     1/1     Running            0             17m
csharp-subscriber-66b7c5bcbc-jrxf4   2/2     Running            0             3m46s
pythonapp-7c9b7f7966-6jkfc           2/2     Running            0             36s
nodeapp-679885bdf8-4d4sc             2/2     Running            0             36s

And I can see it running

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/hello-kubernetes/deploy$ kubectl logs nodeapp-679885bdf8-4d4sc | tail -n 10
Defaulted container "node" out of: node, daprd
Got a new order! Order ID: 77
Successfully persisted state.
Got a new order! Order ID: 78
Successfully persisted state.
Got a new order! Order ID: 79
Successfully persisted state.
Got a new order! Order ID: 80
Successfully persisted state.
Got a new order! Order ID: 81
Successfully persisted state.

Observability

We’ll want to also setup the basic calculator app with zipkin.

Initially, we can send traces to zipkin (the app/service) using the zipkin format.

The zipkin service is defined in the zipkin.yaml

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ cat zipkin.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zipkin
  template:
    metadata:
      labels:
        app: zipkin
    spec:
      containers:
      - name: zipkin
        image: openzipkin/zipkin
        ports:
        - containerPort: 9411

---

kind: Service
apiVersion: v1
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  selector:
    app: zipkin
  ports:
  - protocol: TCP
    port: 9411
    targetPort: 9411
  type: ClusterIP

The “appconfig” configuration will be setup to send our zipkin (format) traces to the Zipkin service

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ cat appconfig.yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
spec:
  tracing:
    samplingRate: "1"
    zipkin:
      endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"

Later, should we jump into OpenTelemetry, we will use the endpointAddress of our OTel collector.

The app we’ll use to test this setup is the Calculator app.

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ cat python-multiplier.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multiplyapp
  labels:
    app: multiply
spec:
  replicas: 1
  selector:
    matchLabels:
      app: multiply
  template:
    metadata:
      labels:
        app: multiply
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "multiplyapp"
        dapr.io/app-port: "5001"
        dapr.io/config: "appconfig"
    spec:
      containers:
      - name: multiply
        image: ghcr.io/dapr/samples/distributed-calculator-slow-python:latest
        env:
        - name: APP_PORT
          value: "5001"
        ports:
        - containerPort: 5001
        imagePullPolicy: Always

The part that makes it actually send trace data is setup in the annotatations

      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "multiplyapp"
        dapr.io/app-port: "5001"
        dapr.io/config: "appconfig"

Should we wish to stop sending trace data, we merely remove or change the value for config

dapr.io/config: "somethingelse"

We can disable Dapr on it altogether (provided we did not annotate the namespace) with

dapr.io/enabled: "false"

We’ll apply the whole lot of YAML

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ kubectl apply -f .
configuration.dapr.io/appconfig created
deployment.apps/multiplyapp created
deployment.apps/zipkin created
service/zipkin created

We can the test the Zipkin service with a port-forward

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/observability/deploy$ kubectl port-forward svc/zipkin 19111:9411
Forwarding from 127.0.0.1:19111 -> 9411
Forwarding from [::1]:19111 -> 9411
Handling connection for 19111
Handling connection for 19111
Handling connection for 19111
Handling connection for 19111

/content/images/2022/12/cloudtrace-02.png

We can then port forward to the calculator and run some operations.

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/distributed-calculator/deploy$ kubectl port-forward service/calculator-front-end 8000:80
Forwarding from 127.0.0.1:8000 -> 8080
Forwarding from [::1]:8000 -> 8080

/content/images/2022/12/cloudtrace-06.png

Going back to the Zipkin service, we can see some maps in the dependencies page

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/distributed-calculator/deploy$ !1965
kubectl port-forward svc/zipkin 19111:9411
Forwarding from 127.0.0.1:19111 -> 9411
Forwarding from [::1]:19111 -> 9411
Handling connection for 19111
Handling connection for 19111

/content/images/2022/12/cloudtrace-03.png

We can also query the serviceName

/content/images/2022/12/cloudtrace-04.png

We can view any given trace details as well

/content/images/2022/12/cloudtrace-05.png

For instance, with the above we see the front end called addapp which lasted 1.717 seconds.

Setting up for GCP Cloud Trace

For the collector, we’ll need a service account

/content/images/2022/12/cloudtrace-07.png

We’ll set a name

/content/images/2022/12/cloudtrace-08.png

Then we can add the Trace Agent role

/content/images/2022/12/cloudtrace-09.png

Then click through to Done.

We can find the Service Account in the list and click “Manage Keys”

/content/images/2022/12/cloudtrace-10.png

Then “Create new key”

/content/images/2022/12/cloudtrace-11.png

And chose JSON

/content/images/2022/12/cloudtrace-12.png

This will create and download the JSON

I can use that to create the configmap.

(Note: had I used a more generic name, like “credentials.json”, my later env variable for GOOGLE_APPLICATION_CREDENTIALS would use the same name. As I used my downloaded file, we’ll see I have to match filenames for GOOGLE_APPLICATION_CREDENTIALS in my Deployment YAML)

$ kubectl create configmap gcptracecred --from-file=/mnt/c/Users/isaac/Downloads/myanthosproject2-eea0dc5b11be.json
configmap/gcptracecred created

Now that we loaded the configmap, we can use it in the deployment yaml for the zip collector. Only this time we have some environment variables to set and a volume to mount

builder@DESKTOP-QADGF36:~/Workspaces/quickstarts/tutorials/distributed-calculator/deploy$ cat gcp-zipkin-agent.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zipkin
  template:
    metadata:
      labels:
        app: zipkin
    spec:
      containers:
      - name: zipkin
        image: openzipkin/zipkin-gcp
        env:
        - name: STORAGE_TYPE
          value: stackdriver
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /zipkin/.gcp/myanthosproject2-eea0dc5b11be.json
        - name: STACKDRIVER_PROJECT_ID
          value: myanthosproject2
        volumeMounts:
        - name: gcp-volume
          mountPath: /zipkin/.gcp
        ports:
        - containerPort: 9411
      volumes:
      - name: gcp-volume
        configMap:
          name: gcptracecred
---

kind: Service
apiVersion: v1
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  selector:
    app: zipkin
  ports:
  - protocol: TCP
    port: 9411
    targetPort: 9411
  type: ClusterIP


$ kubectl apply -f gcp-zipkin-agent.yaml
deployment.apps/zipkin configured
service/zipkin unchanged

I can see it running

$ kubectl logs zipkin-57797dd5db-mjl9v

                  oo
                 oooo
                oooooo
               oooooooo
              oooooooooo
             oooooooooooo
           ooooooo  ooooooo
          oooooo     ooooooo
         oooooo       ooooooo
        oooooo   o  o   oooooo
       oooooo   oo  oo   oooooo
     ooooooo  oooo  oooo  ooooooo
    oooooo   ooooo  ooooo  ooooooo
   oooooo   oooooo  oooooo  ooooooo
  oooooooo      oo  oo      oooooooo
  ooooooooooooo oo  oo ooooooooooooo
      oooooooooooo  oooooooooooo
          oooooooo  oooooooo
              oooo  oooo

     ________ ____  _  _____ _   _
    |__  /_ _|  _ \| |/ /_ _| \ | |
      / / | || |_) | ' / | ||  \| |
     / /_ | ||  __/| . \ | || |\  |
    |____|___|_|   |_|\_\___|_| \_|

:: version 2.23.16 :: commit b90f2b3 ::

2022-12-18 22:42:35.990  INFO [/] 1 --- [oss-http-*:9411] c.l.a.s.Server                           : Serving HTTP at /[0:0:0:0:0:0:0:0%0]:9411 - http://127.0.0.1:9411/

I’ll run a few more operations on the calculator

kubectl port-forward service/calculator-front-end 8000:80
Forwarding from 127.0.0.1:8000 -> 8080
Forwarding from [::1]:8000 -> 8080
Handling connection for 8000
Handling connection for 8000
Handling connection for 8000
Handling connection for 8000

/content/images/2022/12/cloudtrace-13.png

Viewing trace data in Cloud Trace

We can now go back to our GCP project and see it properly captured traces

/content/images/2022/12/cloudtrace-14.png

I can view specific traces

/content/images/2022/12/cloudtrace-15.png

In GCP, Traces have logs. So one can pull up trace data in Log explorer as well

/content/images/2022/12/cloudtrace-17.png

clicking view shows the logs:

/content/images/2022/12/cloudtrace-16.png

Trace Reports

I made a few traces so I could create a reasonable report

/content/images/2022/12/cloudtrace-18.png

On Analysis reports, I created one for the last hour

/content/images/2022/12/cloudtrace-19.png

We can see latency as it breaks down by percentage of requests

/content/images/2022/12/cloudtrace-20.png

The above tells us the median is just shy of 2ms.

Usage Metrics

We can view usage metrics (from the last month) in Metrics Explorer.

/content/images/2022/12/cloudtrace-21.png

We can look at just Spans Ingested as well in Metrics Explorer

/content/images/2022/12/cloudtrace-22.png

Alerting

We could use the metric of “Trace Spans Ingested” to trigger an alert policy.

For instance, say I wanted to check the max (total) spans in a given hour

/content/images/2022/12/cloudtrace-23.png

If it exceeds 150, for instance

/content/images/2022/12/cloudtrace-24.png

I could then email myself with a note

/content/images/2022/12/cloudtrace-25.png

I give it a name

/content/images/2022/12/cloudtrace-26.png

Then lastly review the policy

/content/images/2022/12/cloudtrace-27.png

I can click “Create Policy” if i wish to create it, or view the JSON with “view code”

{
  "displayName": "TooManySpans",
  "documentation": {
    "content": "Something is getting noisy - more than our allowed max spans in an hour!",
    "mimeType": "text/markdown"
  },
  "userLabels": {},
  "conditions": [
    {
      "displayName": "Global - Trace spans ingested",
      "conditionThreshold": {
        "filter": "resource.type = \"global\" AND metric.type = \"cloudtrace.googleapis.com/billing/spans_ingested\"",
        "aggregations": [
          {
            "alignmentPeriod": "3600s",
            "crossSeriesReducer": "REDUCE_NONE",
            "perSeriesAligner": "ALIGN_MAX"
          }
        ],
        "comparison": "COMPARISON_GT",
        "duration": "0s",
        "trigger": {
          "count": 1
        },
        "thresholdValue": 150
      }
    }
  ],
  "alertStrategy": {
    "autoClose": "604800s"
  },
  "combiner": "OR",
  "enabled": true,
  "notificationChannels": [
    "projects/myanthosproject2/notificationChannels/14548337521871198017"
  ]
}

I should note that while I have email setup presently, there are plenty of Notification Channels including Slack, Webhooks and PagerDuty.

/content/images/2022/12/cloudtrace-28.png

Dashboard

We can turn any metric into a Dashboard.

Just save your metric from the Metrics Explorer to a new (or existing) dashboard.

/content/images/2022/12/cloudtrace-30.png

In the dashboard, we can view for different time spans

/content/images/2022/12/cloudtrace-31.png

Pricing

We really have to hammer Cloud Trace to incur costs. The first 2.5 million spans a month are free and it’s just US$0.20 a million beyond that.

/content/images/2022/12/cloudtrace-29.png

While GCP Monitoring is not free, it’s charged my data ingested for monitoring. So, I believe, in our case it’s not going to cost us.

Summary

Today we respun an on-prem K3s cluster then loaded it with Dapr. We used the Observability tutorial to show an auto-instrumented calculator app sending zipkin formatted traces to zipkin. We then pivoted to using the GCP Zipkin collector partnered with a proper Service Account to route the traces to GCP Cloud Trace. Lastly, we explored several features of cloud trace including trace details, reports, metrics and alerting.

GCP Cloud Trace is a nice low-cost option for viewing traces. It can gather metrics and use alerting. We can make a very basic report, graph and dashboard.

For a product that is essentially free for moderate use, I’ll not complain. I can’t create warn vs alert thresholds nor can I see service maps

gcp cloud trace trace

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes