Dapr Part 3: Digging into Secrets

Dapr provides a basic but very functional secrets abstraction component we can use in our services.  In our last topic we furthered the pub/sub knowledge with a custom perl subscriber.  Today we will follow that by examining secrets integration with AKV and AWS SSM.

Setup

We are going to build off of the last blog (Dapr : Part 2). However, if you want to create a quick secret store demo to use some of these building blocks, you can follow the Dapr secretstore quick start here.

We'll assume you already have a kubernetes cluster created and available for steps below.

Let's set our subscription and create a resource group for our Key Vault

$ az account set -s Pay-As-You-Go
$ az group create -n idjakvrg --location centralus
{
  "id": "/subscriptions/a283bc3e-01f1-4cec-8426-e04e9d02d95a/resourceGroups/idjakvrg",
  "location": "centralus",
  "managedBy": null,
  "name": "idjakvrg",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

Next, we can create our AKV instance

$ az keyvault create --location centralus --name idjakv --resource-group idjakvrg
{
  "id": "/subscriptions/a283bc3e-01f1-4cec-8426-e04e9d02d95a/resourceGroups/idjakvrg/providers/Microsoft.KeyVault/vaults/idjakv",
  "location": "centralus",
  "name": "idjakv",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "1f5d835c-b129-41e6-b2fe-5858a5f4e41a",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": null,
    "networkAcls": null,
    "provisioningState": "Succeeded",
    "sku": {
      "name": "standard"
    },
    "tenantId": "28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a",
    "vaultUri": "https://idjakv.vault.azure.net/"
  },
  "resourceGroup": "idjakvrg",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

I had some troubles creating the Service Principal with a cert credential

$ az ad sp create-for-rbac --name idjdaprsp --create-cert --cert idjdaprspcert --keyvault idjakv --skip-assignment --years 1
Changing "idjdaprsp" to a valid URI of "http://idjdaprsp", which is the required format used for service principal names
The command failed with an unexpected error. Here is the traceback:

cannot import name 'KeyVaultAuthentication' from 'azure.keyvault' (unknown location)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/knack/cli.py", line 206, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib/python3/dist-packages/azure/cli/core/commands/__init__.py", line 608, in execute
    raise ex...

until I realized that my local az cli was out of date.

$ az version
This command is in preview. It may be changed/removed in a future release.
{
  "azure-cli": "2.0.81",
  "azure-cli-core": "2.0.81",
  "azure-cli-telemetry": "1.0.4",
  "extensions": {
    "azure-devops": "0.17.0"
  }
}

Once I upgraded to a newer version

$ az version
{
  "azure-cli": "2.21.0",
  "azure-cli-core": "2.21.0",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {}
}

I was able to create

$ az ad sp create-for-rbac --name idjdaprsp --create-cert --cert idjdaprspcert --keyvault idjakv --skip-assignment --years 1
Changing "idjdaprsp" to a valid URI of "http://idjdaprsp", which is the required format used for service principal names
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
  "appId": "8049597e-4970-4430-8f9d-9d920f446f05",
  "displayName": "idjdaprsp",
  "name": "http://idjdaprsp",
  "password": null,
  "tenant": "28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a"
}

We can use the portal to view the keyvault and cert created

and via CLI

$ az ad sp show --id 8049597e-4970-4430-8f9d-9d920f446f05 -o json | jq -r '.objectId'
f17e37c4-6a48-4102-bd4c-3ec2bcb7947d

Now let’s grant that secrets.get permissions

$ az keyvault set-policy --name idjakv --object-id f17e37c4-6a48-4102-bd4c-3ec2bcb7947d --secret-permissions get
{- Finished ..
  "id": "/subscriptions/a283bc3e-01f1-4cec-8426-e04e9d02d95a/resourceGroups/idjakvrg/providers/Microsoft.KeyVault/vaults/idjakv",
  "location": "centralus",
  "name": "idjakv",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "1f5d835c-b129-41e6-b2fe-5858a5f4e41a",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a"
      },
      {
        "applicationId": null,
        "objectId": "f17e37c4-6a48-4102-bd4c-3ec2bcb7947d",
        "permissions": {
          "certificates": null,
          "keys": null,
          "secrets": [
            "get"
          ],
          "storage": null
        },
        "tenantId": "28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableRbacAuthorization": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": null,
    "networkAcls": null,
    "privateEndpointConnections": null,
    "provisioningState": "Succeeded",
    "sku": {
      "family": "A",
      "name": "standard"
    },
    "softDeleteRetentionInDays": null,
    "tenantId": "28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a",
    "vaultUri": "https://idjakv.vault.azure.net/"
  },
  "resourceGroup": "idjakvrg",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

We now need that pfx we created when we made the kv.

$ az keyvault secret download --vault-name idjakv --name idjdaprspcert --encoding base64 --file idjdaprspcert.pfx

$ ls -ltra | tail -n2
-rw-r--r-- 1 builder builder 2644 Apr  7 21:50 idjdaprspcert.pfx
drwxr-xr-x 2 builder builder 4096 Apr  7 21:50 .

Now save it as a secret in Kubernetes

$ kubectl create secret generic idjdaprsp --from-file=idjdaprspcert.pfx
secret/idjdaprsp created

For sanity, let’s verify the key name, we’ll need that next.

$ kubectl get secret idjdaprsp -o yaml | grep pfx | head -n1 | sed s/:.*//
  idjdaprspcert.pfx

We now have all the parts that we need to create the AKV component.  

$ cat akvcomp.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: azurekeyvault
  namespace: default
spec:
  type: secretstores.azure.keyvault
  version: v1
  metadata:
  - name: vaultName
    value: idjakv
  - name: spnTenantId
    value: "28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a"
  - name: spnClientId
    value: "8049597e-4970-4430-8f9d-9d920f446f05"
  - name: spnCertificate
    secretKeyRef:
      name: idjdaprsp
      key:  idjdaprspcert.pfx
auth:
    secretStore: kubernetes

$ kubectl apply -f akvcomp.yaml
component.dapr.io/azurekeyvault created

verification

$ kubectl describe component azurekeyvault
Name:         azurekeyvault
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  dapr.io/v1alpha1
Auth:
  Secret Store:  kubernetes
Kind:            Component
Metadata:
  Creation Timestamp:  2021-04-08T02:59:42Z
  Generation:          1
  Managed Fields:
    API Version:  dapr.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:auth:
        .:
        f:secretStore:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:metadata:
        f:type:
        f:version:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-04-08T02:59:42Z
  Resource Version:  23459341
  Self Link:         /apis/dapr.io/v1alpha1/namespaces/default/components/azurekeyvault
  UID:               3659e9be-857a-44a7-bf98-6fb84e0b4415
Spec:
  Metadata:
    Name:   vaultName
    Value:  idjakv
    Name:   spnTenantId
    Value:  28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a
    Name:   spnClientId
    Value:  8049597e-4970-4430-8f9d-9d920f446f05
    Name:   spnCertificate
    Secret Key Ref:
      Key:   idjdaprspcert.pfx
      Name:  idjdaprsp
  Type:      secretstores.azure.keyvault
  Version:   v1
Events:      <none>

Let's create a secret in our AKV instance

$ az keyvault secret set --name MySecret --value Testing --vault-name idjakv
{
  "attributes": {
    "created": "2021-04-08T03:12:28+00:00",
    "enabled": true,
    "expires": null,
    "notBefore": null,
    "recoveryLevel": "Purgeable",
    "updated": "2021-04-08T03:12:28+00:00"
  },
  "contentType": null,
  "id": "https://idjakv.vault.azure.net/secrets/MySecret/a0b2e0360e9c4795967ac9a5db83b9dc",
  "kid": null,
  "managed": null,
  "name": "MySecret",
  "tags": {
    "file-encoding": "utf-8"
  },
  "value": "Testing"
}

We can see that in the portal

Now, once we bounce a pod, or remove and reapply a deployment:

perl-subscriber-5f589bb996-t8vhq                     1/2     Running       0          20s

We can see the secret:

$ kubectl port-forward perl-subscriber-5f589bb996-t8vhq 3500:3500
Forwarding from 127.0.0.1:3500 -> 3500
Forwarding from [::1]:3500 -> 3500
Handling connection for 3500

# in another session
$ curl http://localhost:3500/v1.0/secrets/azurekeyvault/MySecret
{"MySecret":"Testing"}

It should be noted here that the Dapr side car is providing this secret, not my subscriber app.  The subscriber is serving port 8080 and Dapr defaults to 3500.

Updates

Let’s update the secret in the UI

And we can type in a value

Then we can check it

If I keep checking it, it does change

And we can see that i did not rotate that pod:

$ kubectl get pods | grep perl
perl-subscriber-5f589bb996-t8vhq                     2/2     Running   0          8h

We can mix secrets components as well

Create the json with a secret.

Note: local secrets ends up not working, but I'll take you through what i tried

$ cat testinglocal.json
{
   "localSecretYay" : "see, im a secret"
}

then apply

$ cat testinglocal.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: testinglocal
  namespace: default
spec:
  type: secretstores.local.file
  version: v1
  metadata:
  - name: secretsFile
    value: testinglocal.json
  - name: nestedSeparator
    value: ":"

$ kubectl apply -f testinglocal.yaml
component.dapr.io/testinglocal created

It crashed the pod.. Since that is a local file (which doesn't work in k8s)

time="2021-04-08T12:03:29.48793413Z" level=fatal msg="process component testinglocal error: open testinglocal.json: no such file or directory" app_id=perl-subscriber instance=perl-subscriber-5f589bb996-l8f99 scope=dapr.runtime type=log ver=1.1.0

I tried setting the files on the k8s nodes themselves:

isaac@isaac-MacBookAir:~$ cat /tmp/testing.json
{
   "localSecretYay" : "see, im a secret"
}
isaac@isaac-MacBookPro:~$ cat /tmp/testing.json
{
   "localSecretYay" : "see, im a secret"
}

then apply again

$ cat testinglocal.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: testinglocal
  namespace: default
spec:
  type: secretstores.local.file
  version: v1
  metadata:
  - name: secretsFile
    value: /tmp/testing.json
  - name: nestedSeparator
    value: ":"

$ kubectl apply -f testinglocal.yaml
component.dapr.io/testinglocal created
$ kubectl get component
NAME             AGE
pubsub           5d23h
azurekeyvault    10h
awssecretstore   15m
testinglocal     19s

I even came back later after AWS (below) and Azure as well as upgrading to the latest Dapr (1.1.1) and couldn't get localfile to work in Kubernetes

time="2021-04-08T13:12:56.892848028Z" level=info msg="component loaded. name: azurekeyvault, type: secretstores.azure.keyvault/v1" app_id=perl-subscriber instance=perl-subscriber-6d89cf54fc-4kj49 scope=dapr.runtime type=log ver=1.1.1
time="2021-04-08T13:12:56.893142116Z" level=info msg="component loaded. name: awssecretstore, type: secretstores.aws.secretmanager/v1" app_id=perl-subscriber instance=perl-subscriber-6d89cf54fc-4kj49 scope=dapr.runtime type=log ver=1.1.1
time="2021-04-08T13:12:56.893278429Z" level=warning msg="failed to init state store secretstores.local.file/v1 named testinglocal: open /tmp/testing.json: no such file or directory" app_id=perl-subscriber instance=perl-subscriber-6d89cf54fc-4kj49 scope=dapr.runtime type=log ver=1.1.1
time="2021-04-08T13:12:56.893321458Z" level=fatal msg="process component testinglocal error: open /tmp/testing.json: no such file or directory" app_id=perl-subscriber instance=perl-subscriber-6d89cf54fc-4kj49 scope=dapr.runtime type=log ver=1.1.1

AWS

Let’s create us-east-1 SSM parameter

Note: again, here is a try and fail. We'll go through what I did for Parameter Store, which just would not work (as of Dapr 1.1.1 for me) but then we'll use Secret Manager which does work (was GA in 1.0)

We can see it created after we create

We can create a secret store for AWS using an IAM Access and Secret key (and if needed, a token as well)

$ cat aws-secrets.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: awsparameterstore
  namespace: default
spec:
  type: secretstores.aws.parameterstore
  version: v1
  metadata:
  - name: region
    value: "us-east-1"
  - name: accessKey
    value: "A***********************O"
  - name: secretKey
    value: "*****************************************************"

$ kubectl apply -f aws-secrets.yaml
component.dapr.io/awsparameterstore created

I’m going to stop here, since i tried many times over to get parameterstore to work.  I even updated my cluster to the latest Dapr 1.1.1.  No matter what i did, i kept getting the error from the Dapr sidecar:

time="2021-04-08T12:44:18.474799416Z" level=fatal msg="process component awsparameterstore2 error: couldn't find secret store secretstores.aws.parameterstore/v1" app_id=perl-subscriber instance=perl-subscriber-5f589bb996-hjgtd scope=dapr.runtime type=log ver=1.1.0

And after upgrade, the same

time="2021-04-08T12:54:54.905300902Z" level=fatal msg="process component awsparameterstore2 error: couldn't find secret store secretstores.aws.parameterstore/v1" app_id=perl-subscriber instance=perl-subscriber-6d89cf54fc-t4xzn scope=dapr.runtime type=log ver=1.1.1

The documentation indicates parameterstore was introduced in v1.1 but i do not see it.

AWS Secret Store

Using secret store worked, however, while they said sessionToken was optional, for my case, not needing it, i needed to still set the value anyhow (to "")

$ cat aws-secretstore.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: awssecretstore
  namespace: default
spec:
  type: secretstores.aws.secretmanager
  version: v1
  metadata:
  - name: region
    value: "us-east-1"
  - name: accessKey
    value: "A******************"
  - name: secretKey
    value: "*****************************"
  - name: sessionToken
    value: ""

$ kubectl apply -f aws-secretstore.yaml
component.dapr.io/awssecretstore created

I created a quick value in AWS Secrets Manager

And when port-forwarding to a dapr pod, i can see the value returned now

$ curl http://localhost:3500/v1.0/secrets/awssecretstore/TestSecret
{"TestSecret":"{\"MySecret\":\"MyVault\"}"}

And this rather proves Dapr can easily pass forward from two engines at the same time

$ curl http://localhost:3500/v1.0/secrets/azurekeyvault/MySecret && echo && curl http://localhost:3500/v1.0/secrets/awssecretstore/TestSecret && echo
{"MySecret":"DarfDarf"}
{"TestSecret":"{\"MySecret\":\"MyVault\"}"}

In the pod

Let’s show how we can pull in the secrets from within the pod.  We can actually go to the sidecar and pull the secrets from the dapr runner.

Here we can create another endpoint and fetch secrets from AWS and AKV at the same time.

I'll add a "/D" dispatcher

my %dispatch = (
    '/dapr/subscribe' => \&resp_subscribe,
    '/hello' => \&resp_hello,
    '/A' => \&resp_A,
    '/B' => \&resp_B,
    '/C' => \&resp_C,
    '/D' => \&resp_D,
    # ...
);

And implementation of "resp_D"

sub resp_D {
    my $cgi  = shift;   # CGI.pm object
    return if !ref $cgi;
 
    my $secretsURL = "http://localhost:3500/v1.0/secrets";
 
    print $cgi->header('application/json');
    # azure kv
    my $cmd = "curl -H 'Content-Type: application/json' $secretsURL/azurekeyvault/MySecret";
    print STDERR "\ncmd: $cmd\n";
    my $rc =`$cmd`;
    print STDERR "\n$rc\n";
    print STDERR "\n";
    # aws
    $cmd = "curl -H 'Content-Type: application/json' $secretsURL/awssecretstore/TestSecret";
    print STDERR "\ncmd: $cmd\n";
    $rc =`$cmd`;
    print STDERR "\n$rc\n";
    print STDERR "\n";
}

When we build and push, we can port forward:

$ kubectl port-forward perl-subscriber-7b4457c4bf-4zghq 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080

Then hit the endpoint:
$ curl -X POST http://localhost:8080/D -H 'Content-Type: application/json'

And lastly check the logs:

$ kubectl logs perl-subscriber-7b4457c4bf-4zghq perl-subscriber
running on 8080
MyWebServer: You can connect to your server at http://localhost:8080/

cmd: curl -H 'Content-Type: application/json' http://localhost:3500/v1.0/secrets/azurekeyvault/MySecret
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    23  100    23    0     0     19      0  0:00:01  0:00:01 --:--:--    19

{"MySecret":"DarfDarf"}


cmd: curl -H 'Content-Type: application/json' http://localhost:3500/v1.0/secrets/awssecretstore/TestSecret
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    43  100    43    0     0    113      0 --:--:-- --:--:-- --:--:--   113

{"TestSecret":"{\"MySecret\":\"MyVault\"}"}

And again, this matches our values in AWS Secret Store

and AKV

Notes on Cloud Costs

And if we want a handle on costs, a few days of SecretsManager queries amounted to $0.11 for AWS

And AKV is effectively free at this level

Summary

Overall I really like the ease of which it took to add Dapr components for AWS SSM and Azure AKV.  I would have tried Hashi Vault as well, which I'm sure works fine, but will save that for another demo.  

I was a tad disappointed that i couldn't get localfile to work, however it would also be rather pointless in Kubernetes as there is a perfectly good basic secrets management reference to Kubernetes secrets already in Dapr and in Kubernetes itself.  That said, i was bothered by not being able to use Parameter Store.  For all my googling, i could not find others with similar issues.  I'm hoping i can sort it out in future demos.  While SSM is fine, for my daily work, i often use Parameter store for lightweight storage of key value pairs that don't need the heft of KMS keys for storage.