Kubewatch to sync secrets to AKV for DR

In our last blog entry we dived into kubernetes event bindings in Dapr.io, Kubewatch and then started on a path to use Azure DevOps Pipelines with webhooks. In this blog we will complete that journey by refreshing ourselves on the Kubewatch payload then working out Azure Pipelines that will store the values into Azure Key Vault.

Note: you can follow along with code in Github here

A plan

I thought of three ways we could do this:

The first was to use Dapr on a pubsub to trigger Azure DevOps as well as things like a notifier.

The next method I considered was using an Azure DevOps Agent. If we had a disconnected cluster, this would likely be the required approach.

However, in all things we need to consider simplifying

Approach 3: Orchestrating via Azure Pipelines

Let's use the third approach.

Working out the kubewatch payload

First thing we need to do is to sort out the payload. We could route the kubewatch through a webhook to a debug pod

routing to a service pointing to a debug pod

Using the perl debugger, i parsed out the webhook payload on Create, Update and Delete:

$ kubectl logs perl-debugger-5967f99ff6-fcdwv perl-debugger
ERROR: /dapr/config not found in handler.. HTTP/1.0 404 Not found
ERROR: /dapr/subscribe not found in handler.. HTTP/1.0 404 Not found
ERROR: /kubeevents not found in handler.. HTTP/1.0 404 Not found
ERROR: /snsnotify not found in handler.. HTTP/1.0 404 Not found
ERROR: / not found in handler.. HTTP/1.0 404 Not found
decdata : HASH(0x56450dc341b8)
dVal : Data::Dumper=HASH(0x56450dc34020)
$decdata = {
             'messageType' => 'A',
             'message' => 'Hello'
           };

decdata : HASH(0x56450dc33fa8)
dVal : Data::Dumper=HASH(0x56450dbfeab0)
$decdata = {
             'eventmeta' => {
                              'name' => 'my-secret5',
                              'kind' => 'secret',
                              'reason' => 'created',
                              'namespace' => 'default'
                            },
             'time' => '2021-06-10T13:11:05.603329135Z',
             'text' => 'A `secret` in namespace `default` has been `created`:
`my-secret5`'
           };

decdata : HASH(0x56450dc19de0)
dVal : Data::Dumper=HASH(0x56450dc18808)
$decdata = {
             'text' => 'A `secret` in namespace `` has been `updated`:
`default/my-secret5`',
             'time' => '2021-06-10T13:11:55.175407571Z',
             'eventmeta' => {
                              'reason' => 'updated',
                              'namespace' => '',
                              'kind' => 'secret',
                              'name' => 'default/my-secret5'
                            }
           };

decdata : HASH(0x56450dc17e48)
dVal : Data::Dumper=HASH(0x56450dc19df8)
$decdata = {
             'eventmeta' => {
                              'name' => 'default/my-secret5',
                              'kind' => 'secret',
                              'reason' => 'deleted',
                              'namespace' => 'default'
                            },
             'time' => '2021-06-10T13:12:51.552845793Z',
             'text' => 'A `secret` in namespace `default` has been `deleted`:
`default/my-secret5`'
           };

This was sent by creating, updating and then deleting my-secret.yaml

$ cat my-secret.yaml
apiVersion: v1
data:
  key1: Y2hhbmdlZHZhbHVlCg==
  key2: bGV0c3VwZGF0ZQo=
  key3: Y2hhbmdlZHZhbHVlCg==
kind: Secret
metadata:
  name: my-secret5
  namespace: default
type: Opaque

Webhooks in Azure DevOps

First i created the service hook in the project's service connections

Webhook Definition in Service Connections

Then i went and updated the helm values on kubewatch to use the webhook:

$ helm get values kubewatch
USER-SUPPLIED VALUES:
msteams:
  enabled: true
  webhookurl: https://princessking.webhook.office.com/webhookb2/….snip….
rbac:
  create: true
resourcesToWatch:
  clusterrole: false
  configmap: false
  daemonset: false
  deployment: false
  ingress: false
  job: false
  namespace: false
  node: false
  persistentvolume: false
  pod: false
  replicaset: false
  replicationcontroller: false
  secret: true
  serviceaccount: false
  services: false
slack:
  enabled: false
webhook:
  enabled: true
  url: https://dev.azure.com/princessking/_apis/public/distributedtask/webhooks/k8sevents?api-version=6.0-preview

Next I went and created a pipeline that would trigger on the event:

trigger:
- main
 
resources:
  webhooks:
    - webhook: k8sevents
      connection: k8sevents
      filters:
        - path: 'eventmeta.kind' 
          value: secret
      
pool:
  vmImage: ubuntu-latest
 
steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'
 
- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo "kind: ${{ parameters.k8sevents.eventmeta.kind }}"
    echo "reason: ${{ parameters.k8sevents.eventmeta.reason }}"
    echo "namespace: ${{ parameters.k8sevents.eventmeta.namespace }}"
    echo "name: ${{ parameters.k8sevents.eventmeta.name }}"
    echo "have a nice day."
  displayName: 'Check webhook payload'

Verification

we can update the secret and apply it

$ cat my-secret.yaml
apiVersion: v1
data:
  key1: Y2hhbmdlZHZhbHVlCg==
  key2: Y2hhbmdlZHZhbHVlCg==
kind: Secret
metadata:
  name: my-secret6
  namespace: default
type: Opaque

$ kubectl apply -f my-secret.yaml
secret/my-secret6 configured

Then check that our pipeline is invoked by way of the webhook

Saving to AKV

Next we should add a YAML that will save these to AKV.

First, we need to make sure our AzDO service principal has a secret access policy set for the Key Vault we plan to use:

Adding a Secret Access Policy for the AzDO SP

Then to actually fetch the values from Kubernetes, the AzDO agent will need to see the secrets. This requires a kubeconfig.

To keep it simple, we can just base64 our config (that has external IPs.. note, by default k3s uses internal IPs).

$ cat ~/.kube/config | base64 -w 0
YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6C……

Save that into a group variable for our use later:

One improvement we could do here for security is to create a service account with basic secret read policies on a rolebinding then create a kubeconfig just for that service account. However, we'll KISS on this for now.

Our new azure-pipelines.yaml looks like this:

trigger:
- main
 
 
resources:
  webhooks:
    - webhook: k8sevents
      connection: k8sevents
      filters:
        - path: 'eventmeta.kind' 
          value: secret
      
pool:
  vmImage: ubuntu-latest
 
variables:
- group: webhooklibrary
 
steps:
- script: (mkdir ~/.kube || true) && echo $(k8scfg) | base64 --decode > ~/.kube/config
  displayName: 'get k8s config'
 
- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo "kind: ${{ parameters.k8sevents.eventmeta.kind }}"
    echo "reason: ${{ parameters.k8sevents.eventmeta.reason }}"
    echo "namespace: ${{ parameters.k8sevents.eventmeta.namespace }}"
    echo "name: ${{ parameters.k8sevents.eventmeta.name }}"
    echo "have a nice day."
    
    export testKind=${{ parameters.k8sevents.eventmeta.kind }}
    export testReason=${{ parameters.k8sevents.eventmeta.reason }}
    export namespace=${{ parameters.k8sevents.eventmeta.namespace }}
    export name=${{ parameters.k8sevents.eventmeta.name }}
        
    touch $(Pipeline.Workspace)/KEYSETTING.sh
 
    if [[ $testReason == "created" ]]; then
        IFS=$'\n'
        for row in $(kubectl get secret -n $namespace $name -o json | jq '.data')
        do
            if [[ "$row" == *":"* ]]; then
                keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
                #echo $row
                #echo " $keyName and $keyValue"
                `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                echo "az keyvault secret set --name $namespace-$name-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
            fi
        done
    fi
 
 
    if [[ $testReason == "updated" ]]; then
        IFS='/'
        read -a strarr <<<"$name"
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"
 
        #kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data'
        IFS=$'\n'
        for row in $(kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data')
        do
            if [[ "$row" == *":"* ]]; then
                keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
                #echo $row
                #echo " $keyName and $keyValue"
                `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                echo "az keyvault secret set --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
            fi
        done
    fi
 
    # deleted
    if [[ $testReason == "deleted" ]]; then
        IFS='/'
        read -a strarr <<<"$name"
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"
 
        IFS=$'\n'
        for row in $(kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data')
        do
            if [[ "$row" == *":"* ]]; then
                keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g'`
                echo "az keyvault secret set-attributes --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name myiackv --enabled false" >> $(Pipeline.Workspace)/KEYSETTING.sh
            fi
        done
    fi
 
  displayName: 'Create Az KV File'
 
- task: AzureCLI@2
  inputs:
    azureSubscription: 'Pay-As-You-Go(d955c0ba-13dc-44cf-a29a-8fed74cbb22d)'
    scriptType: 'bash'
    scriptLocation: 'scriptPath'
    scriptPath: '$(Pipeline.Workspace)/KEYSETTING.sh'

To test my regular expressions (which are ugly, i know. I will try and smarten them up later) I created a basic test.sh i could run locally to see what it would create

#!/bin/bash
 
# create
export testKind=secret
export testReason=created
export namespace=default
#export name='dapr-workflows'
export name='my-secret6'
 
 
# update
#export testKind=secret
#export testReason=updated
#export namespace=`echo`
#export name='default/my-secret6'
#export name='default/dapr-workflows'
 
# delete
#export testKind=secret
#export testReason=deleted
#export namespace=`echo`
#export name='default/my-secret6'
#export name='default/dapr-workflows'
 
#### tested for update
if [[ $testReason == "created" ]]; then
    IFS=$'\n'
    for row in $(kubectl get secret -n $namespace $name -o json | jq '.data')
    do
        if [[ "$row" == *":"* ]]; then
            export keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
            export keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
            echo $row
            echo " $keyName and $keyValue"
            `echo $keyValue | base64 --decode > $keyName-kvDec.txt`
            echo "az keyvault secret set --name $namespace-$name-$keyName --vault-name TBDVAULT --file ./$keyName-kvDec.txt"
        fi
    done
fi
 
 
if [[ $testReason == "updated" ]]; then
    IFS='/'
    read -a strarr <<<"$name"
    echo "Namespace: ${strarr[0]}"
    echo "Secret: ${strarr[1]}"
 
    #kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data'
    IFS=$'\n'
    for row in $(kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data')
    do
        if [[ "$row" == *":"* ]]; then
            keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
            keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
            #echo $row
            #echo " $keyName and $keyValue"
            `echo $keyValue | base64 --decode > $keyName-kvDec.txt`
            echo "az keyvault secret set --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name TBDVAULT --file ./$keyName-kvDec.txt"
        fi
    done
fi
 
# deleted
if [[ $testReason == "deleted" ]]; then
    IFS='/'
    read -a strarr <<<"$name"
    echo "Namespace: ${strarr[0]}"
    echo "Secret: ${strarr[1]}"
 
    IFS=$'\n'
    for row in $(kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data')
    do
        if [[ "$row" == *":"* ]]; then
            keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
            keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g'`
            #echo $row
            #echo " $keyName and $keyValue"
            `echo $keyValue | base64 --decode > $keyName-kvDec.txt`
            echo "az keyvault secret set-attributes --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name TBDVAULT --enabled false"
        fi
    done
fi

One thing I put in to both was to decode the base64 values prior to inserting into KV.. I'm still debating if that is the right path.

If you want to store the base64 values verbatim, just replace this:

`echo $keyValue | base64 --decode > $keyName-kvDec.txt`

with

`echo $keyValue > $keyName-kvDec.txt`

A bug on delete… if i delete a secret then clearly its not there to query for its keys in data.

Thus i need to rewrite that to interrogate AKV for secrets that match the prefix. I'm guessing i need to leverage a quick get secret and grep on namespace-keyname.. e.g.

$ az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | .name' | grep '^default-my-secret6-'
default-my-secret6-key1
default-my-secret6-key2
default-my-secret6-key3

And indeed, that fixes deletions:

   # deleted
    if [[ $testReason == "deleted" ]]; then
        IFS='/'
        read -a strarr <<<"$name"
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"
 
        echo "az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | .name' | grep '^${strarr[0]}-${strarr[1]}-' | sed 's/^\(.*\)$/az keyvault secret set-attributes --name \1 --vault-name myiackv --enabled false/g' | bash"  >> $(Pipeline.Workspace)/KEYSETTING.sh
    fi

Testing

First, we can delete our secret to trigger kubewatch

$ kubectl delete -f my-secret.yaml
secret "my-secret6" deleted

Then verify the pipeline was invoked

One thing we may realize in our logs, is the log output shows the plain text password - not really something we want to keep in there

Let's implement some cleanup and set echo off in bash

$ git diff azure-pipelines.yml
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 8c5064e..44ab2e2 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -33,6 +33,8 @@ steps:
     echo "name: ${{ parameters.k8sevents.eventmeta.name }}"
     echo "have a nice day."

+    set +x
+
     export testKind=${{ parameters.k8sevents.eventmeta.kind }}
     export testReason=${{ parameters.k8sevents.eventmeta.reason }}
     export namespace=${{ parameters.k8sevents.eventmeta.namespace }}
@@ -47,8 +49,6 @@ steps:
             if [[ "$row" == *":"* ]]; then
                 keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                 keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
-                #echo $row
-                #echo " $keyName and $keyValue"
                 `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                 echo "az keyvault secret set --name $namespace-$name-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
             fi
@@ -69,8 +69,6 @@ steps:
             if [[ "$row" == *":"* ]]; then
                 keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                 keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
-                #echo $row
-                #echo " $keyName and $keyValue"
                 `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                 echo "az keyvault secret set --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
             fi
@@ -96,4 +94,15 @@ steps:
     azureSubscription: 'Pay-As-You-Go(d955c0ba-13dc-44cf-a29a-8fed74cbb22d)'
     scriptType: 'bash'
     scriptLocation: 'scriptPath'
-    scriptPath: '$(Pipeline.Workspace)/KEYSETTING.sh'
\ No newline at end of file
+    scriptPath: '$(Pipeline.Workspace)/KEYSETTING.sh'
+
+- task: Bash@3
+  displayName: 'Remove Generated Files (if exist)'
+  inputs:
+    targetType: 'inline'
+    script: |
+      set +x
+      rm -f ./*.txt || true
+      rm -f KEYSETTING.sh || true
+    workingDirectory: '${Pipeline.Workspaces)'
+  condition: always()
\ No newline at end of file

That renders a pipeline like this:

$ cat azure-pipelines.yml
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- main


resources:
  webhooks:
    - webhook: k8sevents
      connection: k8sevents
      filters:
        - path: 'eventmeta.kind'
          value: secret

pool:
  vmImage: ubuntu-latest

variables:
- group: webhooklibrary

steps:
- script: (mkdir ~/.kube || true) && echo $(k8scfg) | base64 --decode > ~/.kube/config
  displayName: 'get k8s config'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo "kind: ${{ parameters.k8sevents.eventmeta.kind }}"
    echo "reason: ${{ parameters.k8sevents.eventmeta.reason }}"
    echo "namespace: ${{ parameters.k8sevents.eventmeta.namespace }}"
    echo "name: ${{ parameters.k8sevents.eventmeta.name }}"
    echo "have a nice day."

    set +x

    export testKind=${{ parameters.k8sevents.eventmeta.kind }}
    export testReason=${{ parameters.k8sevents.eventmeta.reason }}
    export namespace=${{ parameters.k8sevents.eventmeta.namespace }}
    export name=${{ parameters.k8sevents.eventmeta.name }}

    touch $(Pipeline.Workspace)/KEYSETTING.sh

    if [[ $testReason == "created" ]]; then
        IFS=$'\n'
        for row in $(kubectl get secret -n $namespace $name -o json | jq '.data')
        do
            if [[ "$row" == *":"* ]]; then
                keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
                `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                echo "az keyvault secret set --name $namespace-$name-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
            fi
        done
    fi


    if [[ $testReason == "updated" ]]; then
        IFS='/'
        read -a strarr <<<"$name"
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"

        #kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data'
        IFS=$'\n'
        for row in $(kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data')
        do
            if [[ "$row" == *":"* ]]; then
                keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
                `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                echo "az keyvault secret set --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
            fi
        done
    fi

    # deleted
    if [[ $testReason == "deleted" ]]; then
        IFS='/'
        read -a strarr <<<"$name"
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"

        echo "az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | .name' | grep '^${strarr[0]}-${strarr[1]}-' | sed 's/^\(.*\)$/az keyvault secret set-attributes --name \1 --vault-name myiackv --enabled false/g' | bash"  >> $(Pipeline.Workspace)/KEYSETTING.sh
    fi

    chmod 755 $(Pipeline.Workspace)/KEYSETTING.sh

  displayName: 'Create Az KV File'

- task: AzureCLI@2
  inputs:
    azureSubscription: 'Pay-As-You-Go(d955c0ba-13dc-44cf-a29a-8fed74cbb22d)'
    scriptType: 'bash'
    scriptLocation: 'scriptPath'
    scriptPath: '$(Pipeline.Workspace)/KEYSETTING.sh'

- task: Bash@3
  displayName: 'Remove Generated Files (if exist)'
  inputs:
    targetType: 'inline'
    script: |
      set +x
      rm -f ./*.txt || true
      rm -f KEYSETTING.sh || true
    workingDirectory: '${Pipeline.Workspace)'
  condition: always()

Let's now test

$ kubectl apply -f my-secret.yaml
secret/my-secret6 created

And our triggered pipeline

A DR Strategy

What if we want to store the whole secret? Perhaps our goal is really just DR or replication.

We could add a block to store the keys as base64'ed entries into KV which renders a pipeline as such

 
trigger:
- main
 
 
resources:
  webhooks:
    - webhook: k8sevents
      connection: k8sevents
      filters:
        - path: 'eventmeta.kind' 
          value: secret
      
pool:
  vmImage: ubuntu-latest
 
variables:
- group: webhooklibrary
 
steps:
- script: (mkdir ~/.kube || true) && echo $(k8scfg) | base64 --decode > ~/.kube/config
  displayName: 'get k8s config'
 
- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo "kind: ${{ parameters.k8sevents.eventmeta.kind }}"
    echo "reason: ${{ parameters.k8sevents.eventmeta.reason }}"
    echo "namespace: ${{ parameters.k8sevents.eventmeta.namespace }}"
    echo "name: ${{ parameters.k8sevents.eventmeta.name }}"
    echo "have a nice day."
    
    set +x
 
    export testKind=${{ parameters.k8sevents.eventmeta.kind }}
    export testReason=${{ parameters.k8sevents.eventmeta.reason }}
    export namespace=${{ parameters.k8sevents.eventmeta.namespace }}
    export name=${{ parameters.k8sevents.eventmeta.name }}
        
    touch $(Pipeline.Workspace)/KEYSETTING.sh
 
    # store as b64 yaml
    IFS='/'
    read -a strarr <<<"$name"
    if [[ $testReason == "created" ]]; then
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"
    else
        echo "Namespace: $namespace"
        echo "Secret: $name"
    fi
 
    if [[ $testReason == "deleted" ]]; then
        echo "az keyvault secret set-attributes --name k8ssecret-${strarr[0]}-${strarr[1]} --vault-name myiackv --enabled false" >> $(Pipeline.Workspace)/KEYSETTING.sh
    elif [[ $testReason == "updated" ]]; then
        `kubectl get secret -n ${strarr[0]} ${strarr[1]} -o yaml | base64 -w 0 > $(Pipeline.Workspace)/val-b64.txt`
        echo "az keyvault secret set-attributes --name k8ssecret-${strarr[0]}-${strarr[1]} --vault-name myiackv --enabled true" >> $(Pipeline.Workspace)/KEYSETTING.sh
        echo "az keyvault secret set --name k8ssecret-${strarr[0]}-${strarr[1]} --vault-name myiackv --file $(Pipeline.Workspace)/val-b64.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
    else
        `kubectl get secret -n $namespace $name -o yaml | base64 -w 0 > $(Pipeline.Workspace)/val-b64.txt`
        echo "az keyvault secret set-attributes --name k8ssecret-$namespace-$name --vault-name myiackv --enabled true" >> $(Pipeline.Workspace)/KEYSETTING.sh
        echo "az keyvault secret set --name k8ssecret-$namespace-$name --vault-name myiackv --file $(Pipeline.Workspace)/val-b64.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
    fi
 
    # store as keys
    if [[ $testReason == "created" ]]; then
        IFS=$'\n'
        for row in $(kubectl get secret -n $namespace $name -o json | jq '.data')
        do
            if [[ "$row" == *":"* ]]; then
                keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
                `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                echo "az keyvault secret set --name $namespace-$name-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
                echo "az keyvault secret set-attributes --name $namespace-$name-$keyName --vault-name myiackv --enabled true" >> $(Pipeline.Workspace)/KEYSETTING.sh
            fi
        done
    fi
 
    if [[ $testReason == "updated" ]]; then
        IFS='/'
        read -a strarr <<<"$name"
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"
 
        #kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data'
        IFS=$'\n'
        for row in $(kubectl get secret -n ${strarr[0]}  ${strarr[1]} -o json | jq '.data')
        do
            if [[ "$row" == *":"* ]]; then
                keyName=`echo $row | sed 's/:.*//' | sed 's/"//g' | sed 's/ //g'`
                keyValue=`echo $row | sed 's/^.*://' | sed 's/"//g' | tr -d '\n' | sed 's/ //g' | sed 's/,$//'`
                `echo $keyValue | base64 --decode > $(Pipeline.Workspace)/$keyName-kvDec.txt`
                echo "az keyvault secret set --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name myiackv --file $(Pipeline.Workspace)/$keyName-kvDec.txt" >> $(Pipeline.Workspace)/KEYSETTING.sh
                echo "az keyvault secret set-attributes --name ${strarr[0]}-${strarr[1]}-$keyName --vault-name myiackv --enabled true" >> $(Pipeline.Workspace)/KEYSETTING.sh
            fi
        done
    fi
 
    # deleted
    if [[ $testReason == "deleted" ]]; then
        IFS='/'
        read -a strarr <<<"$name"
        echo "Namespace: ${strarr[0]}"
        echo "Secret: ${strarr[1]}"
 
        echo "az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | .name' | grep '^${strarr[0]}-${strarr[1]}-' | sed 's/^\(.*\)$/az keyvault secret set-attributes --name \1 --vault-name myiackv --enabled false/g' | bash"  >> $(Pipeline.Workspace)/KEYSETTING.sh
    fi
    
    chmod 755 $(Pipeline.Workspace)/KEYSETTING.sh
 
  displayName: 'Create Az KV File'
 
- task: AzureCLI@2
  inputs:
    azureSubscription: 'Pay-As-You-Go(d955c0ba-13dc-44cf-a29a-8fed74cbb22d)'
    scriptType: 'bash'
    scriptLocation: 'scriptPath'
    scriptPath: '$(Pipeline.Workspace)/KEYSETTING.sh'
 
- task: Bash@3
  displayName: 'Remove Generated Files (if exist)'
  inputs:
    targetType: 'inline'
    script: |
      set +x
      rm -f ./*.txt || true
      rm -f KEYSETTING.sh || true
    workingDirectory: '$(Pipeline.Workspace)'
  condition: always()

Testing

Now when i apply a secret with a new entry:

$ kubectl apply -f my-secret.yaml
secret/my-secret6 configured
$ cat my-secret.yaml
apiVersion: v1
data:
  key1: Y2hhbmdlZHZhbHVlCg==
  key2: Y2hhbmdlZHZhbHVlCg==
  key3: Y2hhbmdlZHZhbHVlCg==
kind: Secret
metadata:
  name: my-secret7
  namespace: default
type: Opaque

After kubewatch triggers the pipeline and completes, we see

Which we can see easily renders it back

$ az keyvault secret show --vault-name myiackv --name k8ssecret-default-my-secret7 -o json | jq -r '.value'
YXBpVmVyc2lvbjogdjEKZGF0YToKICBrZXkxOiBZMmhoYm1kbFpIWmhiSFZsQ2c9PQogIGtleTI6IFkyaGhibWRsWkhaaGJIVmxDZz09CiAga2V5MzogWTJoaGJtZGxaSFpoYkhWbENnPT0Ka2luZDogU2VjcmV0Cm1ldGFkYXRhOgogIGFubm90YXRpb25zOgogICAga3ViZWN0bC5rdWJlcm5ldGVzLmlvL2xhc3QtYXBwbGllZC1jb25maWd1cmF0aW9uOiB8CiAgICAgIHsiYXBpVmVyc2lvbiI6InYxIiwiZGF0YSI6eyJrZXkxIjoiWTJoaGJtZGxaSFpoYkhWbENnPT0iLCJrZXkyIjoiWTJoaGJtZGxaSFpoYkhWbENnPT0iLCJrZXkzIjoiWTJoaGJtZGxaSFpoYkhWbENnPT0ifSwia2luZCI6IlNlY3JldCIsIm1ldGFkYXRhIjp7ImFubm90YXRpb25zIjp7fSwibmFtZSI6Im15LXNlY3JldDciLCJuYW1lc3BhY2UiOiJkZWZhdWx0In0sInR5cGUiOiJPcGFxdWUifQogIGNyZWF0aW9uVGltZXN0YW1wOiAiMjAyMS0wNi0xM1QyMzo0NzoxMloiCiAgbWFuYWdlZEZpZWxkczoKICAtIGFwaVZlcnNpb246IHYxCiAgICBmaWVsZHNUeXBlOiBGaWVsZHNWMQogICAgZmllbGRzVjE6CiAgICAgIGY6ZGF0YToKICAgICAgICAuOiB7fQogICAgICAgIGY6a2V5MToge30KICAgICAgICBmOmtleTI6IHt9CiAgICAgICAgZjprZXkzOiB7fQogICAgICBmOm1ldGFkYXRhOgogICAgICAgIGY6YW5ub3RhdGlvbnM6CiAgICAgICAgICAuOiB7fQogICAgICAgICAgZjprdWJlY3RsLmt1YmVybmV0ZXMuaW8vbGFzdC1hcHBsaWVkLWNvbmZpZ3VyYXRpb246IHt9CiAgICAgIGY6dHlwZToge30KICAgIG1hbmFnZXI6IGt1YmVjdGwtY2xpZW50LXNpZGUtYXBwbHkKICAgIG9wZXJhdGlvbjogVXBkYXRlCiAgICB0aW1lOiAiMjAyMS0wNi0xM1QyMzo0NzoxMloiCiAgbmFtZTogbXktc2VjcmV0NwogIG5hbWVzcGFjZTogZGVmYXVsdAogIHJlc291cmNlVmVyc2lvbjogIjQ1NjE0MjEyIgogIHNlbGZMaW5rOiAvYXBpL3YxL25hbWVzcGFjZXMvZGVmYXVsdC9zZWNyZXRzL215LXNlY3JldDcKICB1aWQ6IDUyMGJjNGFmLTdkOTYtNGY2NC05ODFjLTMzYzBlNmViODBiMAp0eXBlOiBPcGFxdWUK

$ az keyvault secret show --vault-name myiackv --name k8ssecret-default-my-secret7 -o json | jq -r '.value' | base64 --decode
apiVersion: v1
data:
  key1: Y2hhbmdlZHZhbHVlCg==
  key2: Y2hhbmdlZHZhbHVlCg==
  key3: Y2hhbmdlZHZhbHVlCg==
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"key1":"Y2hhbmdlZHZhbHVlCg==","key2":"Y2hhbmdlZHZhbHVlCg==","key3":"Y2hhbmdlZHZhbHVlCg=="},"kind":"Secret","metadata":{"annotations":{},"name":"my-secret7","namespace":"default"},"type":"Opaque"}
  creationTimestamp: "2021-06-13T23:47:12Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:key1: {}
        f:key2: {}
        f:key3: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-06-13T23:47:12Z"
  name: my-secret7
  namespace: default
  resourceVersion: "45614212"
  selfLink: /api/v1/namespaces/default/secrets/my-secret7
  uid: 520bc4af-7d96-4f64-981c-33c0e6eb80b0
type: Opaque

This means we can easily get each active key value back with a simple JQ filter applied to the Key Vault query:

$ az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | select(.attributes.enabled==true) | .name' | grep '^default-'
default-my-secret6-key1
default-my-secret6-key2
default-my-secret6-key3
default-my-secret7-key1
default-my-secret7-key2
default-my-secret7-key3

Or the full active YAML values:

$ az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | select(.attributes.enabled==true) | .name' | grep '^k8ssecret-default-'
k8ssecret-default-my-secret6
k8ssecret-default-my-secret7

While I'm sure there are more slick ways to do this in bash, here is my quick DR one-liner that would recreate the secrets

$ (rm -f ./recreate.sh || true) && az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | select(.attributes.enabled==true) | .name' | grep '^k8ssecret-default-' | sed 's/^\(.*\)$/az keyvault secret show --vault-name myiackv --name \1 -o json | jq -r .value | base64 --decode >> recreate.yaml \&\& echo '---' >> recreate.yaml/' | bash

$ cat recreate.yaml
apiVersion: v1
data:
  key1: Y2hhbmdlZHZhbHVlCg==
  key2: Y2hhbmdlZHZhbHVlCg==
  key3: Y2hhbmdlZHZhbHVlCg==
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"key1":"Y2hhbmdlZHZhbHVlCg==","key2":"Y2hhbmdlZHZhbHVlCg==","key3":"Y2hhbmdlZHZhbHVlCg=="},"kind":"Secret","metadata":{"annotations":{},"name":"my-secret6","namespace":"default"},"type":"Opaque"}
  creationTimestamp: "2021-06-13T22:54:47Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:key1: {}
        f:key2: {}
        f:key3: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-06-13T22:54:47Z"
  name: my-secret6
  namespace: default
  resourceVersion: "45601878"
  selfLink: /api/v1/namespaces/default/secrets/my-secret6
  uid: 6fd9b3fb-c740-4b15-943d-e798dca41eaa
type: Opaque
---
apiVersion: v1
data:
  key1: Y2hhbmdlZHZhbHVlCg==
  key2: Y2hhbmdlZHZhbHVlCg==
  key3: Y2hhbmdlZHZhbHVlCg==
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"key1":"Y2hhbmdlZHZhbHVlCg==","key2":"Y2hhbmdlZHZhbHVlCg==","key3":"Y2hhbmdlZHZhbHVlCg=="},"kind":"Secret","metadata":{"annotations":{},"name":"my-secret7","namespace":"default"},"type":"Opaque"}
  creationTimestamp: "2021-06-13T23:47:12Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:key1: {}
        f:key2: {}
        f:key3: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-06-13T23:47:12Z"
  name: my-secret7
  namespace: default
  resourceVersion: "45614212"
  selfLink: /api/v1/namespaces/default/secrets/my-secret7
  uid: 520bc4af-7d96-4f64-981c-33c0e6eb80b0
type: Opaque
---

And of course you could do the whole of it as one-liner with:

$ (rm -f ./recreate.sh || true) && az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | select(.attributes.enabled==true) | .name' | grep '^k8ssecret-default-' | sed 's/^\(.*\)$/az keyvault secret show --vault-name myiackv --name \1 -o json | jq -r .value | base64 --decode >> recreate.yaml \&\& echo '---' >> recreate.yaml/' | bash && kubectl apply -f ./recreate.yaml
secret/my-secret6 configured
secret/my-secret7 configured

and re-running would have errors as we just re-applied:

$ (rm -f ./recreate.sh || true) && az keyvault secret list --vault-name myiackv -o json | jq -r '.[] | select(.attributes.enabled==true) | .name' | grep '^k8ssecret-default-' | sed 's/^\(.*\)$/az keyvault secret show --vault-name myiackv --name \1 -o json | jq -r .value | base64 --decode >> recreate.yaml \&\& echo '---' >> recreate.yaml/' | bash && kubectl apply -f ./recreate.yaml


Error from server (Conflict): error when applying patch:
{"metadata":{"resourceVersion":"45601878"}}
to:
Resource: "/v1, Resource=secrets", GroupVersionKind: "/v1, Kind=Secret"
Name: "my-secret6", Namespace: "default"
for: "./recreate.yaml": Operation cannot be fulfilled on secrets "my-secret6": the object has been modified; please apply your changes to the latest version and try again...

Summary

We worked out a webhook trigger on kubewatch. We then setup our Azure DevOps service principal to have secret access to AKV and stored a Kubeconfig so Azure DevOps could also interact with our kubernetes cluster. Lastly, we worked out a pipeline that could both store the individual keys from a secret as well as the kubernetes secret itself for disaster recovery and tested with a bash one-liner.

The goal was to take what we started in the last blog entry and get it over the line to a complete working demo. It shows that as valuable as things like Dapr are, sometimes the most straightforward approach of an existing helm based service and a pipelining tool like Azure DevOps Pipelines is more than enough to implement a working backup and DR system on secrets.

Kubewatch to sync secrets to AKV for DR

A plan

Working out the kubewatch payload

Webhooks in Azure DevOps

Verification

Saving to AKV

Testing

A DR Strategy

Testing

Summary

Feedback Forms to AzDO Work Items

Dapr K8s Events Bindings, Kubewatch and AzDO Webhook triggers