Let’s keep going with what we started last week.  This week I’ll show how to format secrets for various database engines, expose Vault with a public IP and load balancer and lastly how to connect multiple kubernetes clusters to an auto-unsealing Vault in AKS.

Getting started

You can look at last week's post to spin the cluster. Otherwise, let’s just log back in to our existing cluster:

$ az account set --subscription aabbccdd-aabb-ccdd-eeff-aabbccddeeff
$ az aks list -o table
Name        Location    ResourceGroup    KubernetesVersion    ProvisioningState    Fqdn
----------  ----------  ---------------  -------------------  -------------------  ---------------------------------------------------------------
ijvaultaks  centralus   ijvaultk8srg     1.17.11              Succeeded            ijvaultaks-ijvaultk8srg-70b42e-e2c32296.hcp.centralus.azmk8s.io

And a sanity that we are in the right kubernetes cluster:

$ kubectl get nodes
NAME                                STATUS   ROLES   AGE     VERSION
aks-nodepool1-84770334-vmss000000   Ready    agent   3d11h   v1.17.11
aks-nodepool1-84770334-vmss000001   Ready    agent   3d11h   v1.17.11
$ kubectl get deployments
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
vault-agent-injector   1/1     1            1           3d11h
webapp                 1/1     1            1           2d15h

Creating a DB User/Pass

In following their tutorials, let’s login and create a database secret and policy:

$ kubectl exec -it vault-0 -- /bin/sh
/ $ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.AghmoNp4pAQekX7ADcM3NID9
token_accessor       yzz0shnNSMnXIK1mQ7b4JogB
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Check that we haven't already set the database string

/ $ vault kv get internal/database/config
Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/internal/database/config
Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path "internal/database/config/"

Add the kv-v2 secrets engine

/ $ vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/

Let’s set a user and password secret:

/ $ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
Key              Value
---              -----
created_time     2020-10-05T11:22:49.6524387Z
deletion_time    n/a
destroyed        false
version          1

And now we should be able to get it back:

/ $ vault kv get internal/database/config
====== Metadata ======
Key              Value
---              -----
created_time     2020-10-05T11:22:49.6524387Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    db-secret-password
username    db-readonly-username

Next we double check on the kubernetes auth.  If you are following the last guide, this should be set and generate an error:

/ $ vault auth enable kubernetes
Error enabling kubernetes auth: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/auth/kubernetes
Code: 400. Errors:

* path is already in use at kubernetes/

If not set, enable then set the auth config (we’ll skip this since it’s already set):

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
/ $ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

Next we’ll create an internal-app policy that reads from those settings:

/ $ vault policy write internal-app - <<EOF
> path "internal/data/database/config" {
>   capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: internal-app
/ $

Lastly, create a role that leverages that policy

/ $ vault write auth/kubernetes/role/internal-app \
>     bound_service_account_names=internal-app \
>     bound_service_account_namespaces=default \
>     policies=internal-app \
>     ttl=24h
Success! Data written to: auth/kubernetes/role/internal-app

We’ll exit out of the pod and create the service account now. Verify it isn't there already.

$ kubectl get sa
NAME                   SECRETS   AGE
default                1         3d11h
vault                  1         3d11h
vault-agent-injector   1         3d11h

Then create the internal-app sa:

$ kubectl create sa internal-app
serviceaccount/internal-app created

Next let’s launch their sample org chart:

$ cat dep-orgchart.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: orgchart
spec:
  selector:
    matchLabels:
      app: orgchart
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: orgchart
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1
$ kubectl apply -f dep-orgchart.yaml
deployment.apps/orgchart created

We can check that it’s running:

$ kubectl get pods -l app=orgchart
NAME                        READY   STATUS    RESTARTS   AGE
orgchart-666df5cf59-82z56   1/1     Running   0          40s

Now let’s see if it has any secrets:

$ kubectl exec `kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}"` --container orgchart -- ls /vault/secrets
ls: /vault/secrets: No such file or directory
command terminated with exit code 1

None exist, so now we can use annotations to inject them into the pod:

$ cat patch-secrets.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

$ kubectl patch deployment orgchart --patch "$(cat patch-secrets.yaml)"
deployment.apps/orgchart patched

We can see they’ve been applied:

$ kubectl get deployments orgchart -o yaml | head -n 37 | tail -n 10
    type: RollingUpdate
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: internal/data/database/config
        vault.hashicorp.com/role: internal-app
      creationTimestamp: null
      labels:
        app: orgchart

Let’s check our pods again:

$ kubectl get pods -l app=orgchart
NAME                        READY   STATUS    RESTARTS   AGE
orgchart-7447d696df-zp9fd   2/2     Running   0          3m11s

We can tell from above, this is a new pod launched after the annotations were applied.  We can now see if we can get secrets and indeed we can:

$ kubectl exec `kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}"` --container orgchart -- ls /vault/secrets
database-config.txt
$ kubectl exec `kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}"` --container orgchart -- cat /vault/secrets/database-config.txt
data: map[password:db-secret-password username:db-readonly-username]
metadata: map[created_time:2020-10-05T11:22:49.6524387Z deletion_time: destroyed:false version:1]

Templates

But what if we need to format those secrets so our app can understand them.  We’ll deviate from the Hashi tutorial a bit to show some common patterns using sqlserver with java and dotnet:

$ cat patch-secrets.new.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-dotnetsql-dbconnection.txt: |
          {{- with secret "internal/data/database/config" -}}
          Server=sqlserver.ad.domain.com;Database=mysqldatabase;User Id={{ .Data.data.username }};Password={{ .Data.data.password }}
          {{- end -}}
        vault.hashicorp.com/agent-inject-javasql-dbconnection.txt: |
          {{- with secret "internal/data/database/config" -}}
          jdbc:sqlserver://sqlserver.ad.domain.com;databaseName=mysqldatabase;userName={{ .Data.data.username }};password={{ .Data.data.password }}
          {{- end -}}

Then patch:

$ kubectl patch deployment orgchart --patch "$(cat patch-secrets.new.yaml)"
deployment.apps/orgchart patched

Lastly, we can check that indeed, it created usable connection strings in our filesystem:

$ kubectl exec `kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}"` --container orgchart -- find /vault/secrets -type f -exec cat -n {} \; -print
     1  data: map[password:db-secret-password username:db-readonly-username]
     2  metadata: map[created_time:2020-10-05T11:22:49.6524387Z deletion_time: destroyed:false version:1]
/vault/secrets/database-config.txt
     1  jdbc:sqlserver://sqlserver.ad.domain.com;databaseName=mysqldatabase;userName=db-readonly-username;password=db-secret-password
/vault/secrets/database-java-config.txt
     1  Server=sqlserver.ad.domain.com;Database=mysqldatabase;User Id=db-readonly-username;Password=db-secret-password
/vault/secrets/database-dotnet-config.txt

Interactive Shell

How might that look on an interactive shell?

Launch an ubuntu pod:

$ kubectl run my-shell -i --tty --image ubuntu -- bash
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
root@my-shell-7d5c87b77c-bnfmz:/# ls /vault/secrets
ls: cannot access '/vault/secrets': No such file or directory
root@my-shell-7d5c87b77c-bnfmz:/# exit
exit
Session ended, resume using 'kubectl attach my-shell-7d5c87b77c-bnfmz -c my-shell -i -t' command when the pod is running

Then we should change the mount point that works for ubuntu (/tmp):

$ cat patch-secrets.new.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/secret-volume-path: "/tmp/"
        vault.hashicorp.com/agent-inject-secret-database-dotnet-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-database-dotnet-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          Server=sqlserver.ad.domain.com;Database=mysqldatabase;User Id={{ .Data.data.username }};Password={{ .Data.data.password }}
          {{- end -}}
        vault.hashicorp.com/agent-inject-secret-database-java-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-database-java-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          jdbc:sqlserver://sqlserver.ad.domain.com;databaseName=mysqldatabase;userName={{ .Data.data.username }};password={{ .Data.data.password }}
          {{- end -}}

And apply the deployment:

$ kubectl patch deployment my-shell --patch "$(cat patch-secrets.new.yaml)"
deployment.apps/my-shell patched

However, you will notice the pods get stuck..

$ kubectl get pods
NAME                                   READY   STATUS     RESTARTS   AGE
my-shell-6cf47859b7-jmvms              0/2     Init:0/1   0          110s
my-shell-7d5c87b77c-7d4wb              1/1     Running    0          3m5s
orgchart-b8ff4d77-krw8b                2/2     Running    0          16m
vault-0                                1/1     Running    0          2d14h
vault-agent-injector-bdbf7b844-kqbnp   1/1     Running    0          3d12h
webapp-66bd6d455d-6rnfm                1/1     Running    0          2d16h

That’s because Vault doesn't know who this pod is.. We need to set the service account the ubuntu container runs as in order for the vault injector to mount:

$ kubectl get deployments my-shell -o yaml > my-shell.yaml

Here i then added a section to set service accounts in the container block:

     serviceAccount: internal-app
     serviceAccountName: internal-app

The full yaml:

$ cat my-shell.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-database-config.txt: internal/data/database/config
    vault.hashicorp.com/role: internal-app
  creationTimestamp: "2020-10-05T12:01:45Z"
  generation: 4
  labels:
    run: my-shell
  name: my-shell
  namespace: default
  resourceVersion: "729218"
  selfLink: /apis/apps/v1/namespaces/default/deployments/my-shell
  uid: f0f3ce43-dca7-408b-aa48-2cae55c37071
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      run: my-shell
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-database-dotnet-config.txt: internal/data/database/config
        vault.hashicorp.com/agent-inject-secret-database-java-config.txt: internal/data/database/config
        vault.hashicorp.com/agent-inject-status: update
        vault.hashicorp.com/agent-inject-template-database-dotnet-config.txt: "{{-
          with secret \"internal/data/database/config\" -}} \nServer=sqlserver.ad.domain.com;Database=mysqldatabase;User
          Id={{ .Data.data.username }};Password={{ .Data.data.password }}\n{{- end
          -}}\n"
        vault.hashicorp.com/agent-inject-template-database-java-config.txt: |-
          {{- with secret "internal/data/database/config" -}}
          jdbc:sqlserver://sqlserver.ad.domain.com;databaseName=mysqldatabase;userName={{ .Data.data.username }};password={{ .Data.data.password }}
          {{- end -}}
        vault.hashicorp.com/role: internal-app
        vault.hashicorp.com/secret-volume-path: /tmp/
      creationTimestamp: null
      labels:
        run: my-shell
    spec:
      containers:
      - args:
        - bash
        image: ubuntu
        imagePullPolicy: Always
        name: my-shell
        resources: {}
        stdin: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        tty: true
      serviceAccount: internal-app
      serviceAccountName: internal-app
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-10-05T12:08:20Z"
    lastUpdateTime: "2020-10-05T12:08:20Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-10-05T12:01:45Z"
    lastUpdateTime: "2020-10-05T12:12:20Z"
    message: ReplicaSet "my-shell-84c788bb66" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 4
  readyReplicas: 1
  replicas: 2
  unavailableReplicas: 1
  updatedReplicas: 1

We can now see it launched:

$ kubectl apply -f my-shell.yaml
$ kubectl get pods | head -n2
NAME                                   READY   STATUS    RESTARTS   AGE
my-shell-7fdd4d5bc8-bttrv              2/2     Running   0          5m25s

And we can verify we can see the contents:

$ kubectl exec -it my-shell-7fdd4d5bc8-bttrv -- /bin/bash
Defaulting container name to my-shell.
Use 'kubectl describe pod/my-shell-7fdd4d5bc8-bttrv -n default' to see all of the containers in this pod.
root@my-shell-7fdd4d5bc8-bttrv:/# ls /tmp
database-dotnet-config.txt  database-java-config.txt
root@my-shell-7fdd4d5bc8-bttrv:/# cat /tmp/database-dotnet-config.txt
Server=sqlserver.ad.domain.com;Database=mysqldatabase;User Id=db-readonly-username;Password=db-secret-passwordroot@my-shell-7fdd4d5bc8-bttrv:/#

Exposing Vault on Public IP with LoadBalancer

You may wish to expose Vault to the outside world, or at least outside of your cluster.

In doing so, You’ll want to expose the service:

$ kubectl expose svc vault --port=8200 --target-port=8200 --name=vault2 --type=LoadBalance
r
service/vault2 exposed
$ kubectl get svc
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)             AGE
kubernetes                 ClusterIP      10.0.0.1       <none>           443/TCP             3d13h
vault                      ClusterIP      10.0.198.124   <none>           8200/TCP,8201/TCP   3d12h
vault-agent-injector-svc   ClusterIP      10.0.169.79    <none>           443/TCP             3d12h
vault-internal             ClusterIP      None           <none>           8200/TCP,8201/TCP   3d12h
vault2                     LoadBalancer   10.0.19.192    52.154.254.196   8200:30615/TCP      6s

However, this exposes vault globally.  Most of the time, this is a pretty bad idea.  So let’s restrict via Azure’s NSG applied to the VNet.

Log into the Azure port and find the NSG in the Resource Group for the cluster and delete the inbound “Internet” (everywhere) rule...

We can now reload the browser and see we cannot access that UI anymore:

Now I'll go to Inbound Security Rules and add a new one.  I’ll use whatismyip.com or similar to determine my local output.  And to be more open, i’ll open a range

I did update that source port to “*” since i was filtering on IP and destination already:

And now i can reach Vault again:


This means we could direct pods in an external kubernetes to use that cluster.

e.g.

$ cat <<EOF | kubectl apply -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: devwebapp
  labels:
    app: devwebapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devwebapp
  template:
    metadata:
      labels:
        app: devwebapp
    spec:
      serviceAccountName: internal-app
      containers:
      - name: app
        image: burtlo/devwebapp-ruby:k8s
        imagePullPolicy: Always
        env:
        - name: VAULT_ADDR
          value: "http://52.154.254.196:8200"
EOF

Or we could expose a service in an external cluster for much the same purpose:

$ cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Service
metadata:
  name: external-vault
  namespace: default
spec:
  ports:
  - protocol: TCP
    port: 8200
---
apiVersion: v1
kind: Endpoints
metadata:
  name: external-vault
subsets:
  - addresses:
      - ip: 52.154.254.196
    ports:
      - port: 8200
EOF

Lastly, we could install vault with helm in another cluster pointing at this vault:

helm install vault hashicorp/vault \
    --set "injector.externalVaultAddr=http://52.154.254.196:8200"

In fact.. let's do that next

Adding another external K8s

Next let’s try to join an outside cluster with the Vault Injector.

Let’s create a fresh AKS cluster:

$ az aks create --resource-group ijvaultk8srg --name ijvaultaks2 --location centralus --node-count 3 --enable-cluster-autoscaler --min-count 2 --max-count 4 --generate-ssh-keys --network-plugin azure --network-policy azure --service-principal $SP_ID --client-secret $SP_PASS

Get credentials

$ az aks list -o table
Name         Location    ResourceGroup    KubernetesVersion    ProvisioningState    Fqdn
-----------  ----------  ---------------  -------------------  -------------------  ---------------------------------------------------------------
ijvaultaks   centralus   ijvaultk8srg     1.17.11              Succeeded            ijvaultaks-ijvaultk8srg-70b42e-e2c32296.hcp.centralus.azmk8s.io
ijvaultaks2  centralus   ijvaultk8srg     1.17.11              Succeeded            ijvaultaks-ijvaultk8srg-70b42e-1ff86476.hcp.centralus.azmk8s.io
$ az aks get-credentials -n ijvaultaks2 -g ijvaultk8srg --admin
Merged "ijvaultaks2-admin" as current context in /home/builder/.kube/config

Double check we have the auth-delegator role (we’ll need it in a moment)

$ kubectl get clusterroles --all-namespaces | grep auth
system:auth-delegator                                                  12m

We want to install the following:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default

We can use inline:

$ cat <<EOF | kubectl create -f -
-
apiVe> ---
rsion: > apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: vault-auth
> ---
iVersio> apiVersion: v1
> kind: Secret
> metadata:
>   name: vault-auth
>   annotations:
>     kubernetes.io/service-account.name: vault-auth
> type: kubernetes.io/service-account-token
> ---
> apiVersion: rbac.authorization.k8s.io/v1beta1
> kind: ClusterRoleBinding
> metadata:
>   name: role-tokenreview-binding
> roleRef:
>   apiGroup: rbac.authorization.k8s.io
>   kind: ClusterRole
>   name: system:auth-delegator
> subjects:
>   - kind: ServiceAccount
>     name: vault-auth
>     namespace: default
> EOF
serviceaccount/vault-auth created
secret/vault-auth created
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

Next we need the token.  Since i had some difficulties in LKE with this, i’ll show you the values to avoid any confusion:

$ kubectl get secret vault-auth -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUV5VENDQXJHZ0F3SUJBZ0lRT3UxaTZjR1UxU2tac01sVHZjeGplREFOQmdrcWhraUc5dzBCQVFzRkFEQU4KTVFzd0NRWURWUVFERXdKallUQWdGdzB5TURFd01EWXhNVEU0TkRsYUdBOHlNRFV3TVRBd05qRXhNamcwT1ZvdwpEVEVMTUFrR0ExVUVBeE1DWTJFd2dnSWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUNEd0F3Z2dJS0FvSUNBUURWCjZqbUI3YWJ1VEZzR0VIUG1ZdHhEU1czbmVvUVVtVGRiRXczWGFHcmxlV280cjFmTDVSaW12WkJLL2MxelBKMTcKRGo4QkJ2RGNobGZHaEpEcVgxL0g1TVRmeWcwQUNKSnVlcy9uQ2tmU0o5eWRwb012c25zT3Z0VlJKZktsTDZWegp5UFE4SjVFdnhMY0RVRjg0ZDNzUXRCa3FiMzVMdWR2UENlQTFKam5jait3ejV1RXhTV1MyQk5US0xIV0I2YlB1CjFJMkJVQVhvbm5SZTVIczBoQ2ZFZ0pwZDgzUSsvd0lzS1AvbWNNaEJONUNsbWpmSk1Ib3YvWkhtUXBQcnY4TkMKd2dJTThqYlVPRWFpd3NqWXF0NDBTNDY1OGxmSnEya0tiMk9kSXNQclN4ZmdXeVh0TVh4c29wZHViRDBObW9KTgpNdWdUZHhWMkpxU1diRTR2YUJaMnluUzZycTh3NXduT2ZwSi9EZE9Rc1doa1R5TlluVkxzSHpjY0orWDQvRnZ1ClNWd01VL2NTYndKUFppSmwxeGE1OFk1UHA4ZHNlNnF3azluOWUvdWRRN3d0cDc0Qk9qQ2JaZzVXdnN1WGVWS1EKK0ZTaWNNR0pkWTN0c3k3dnN2ZEI1Q1JpTEJwR1Z1eENuR1l2Q1JPV2l5RjZpWmt1RW9tRjVYVTBTNWpwTk5tMgpiT0VsTGQ0aXBLMHVNbnVONldQVTMwTEY4cVFLR3hMOWdJSERaaXFnZlg3b1cvNGw2ZzNBdi85L3pIak5CekxFCjg2S3JPRUVmZzI2cXZEQjEwTUliMUJxQjgvSkhyRVBReGY1MSszaXZnSHlqdTJnclY5Y3pmbWp3Tlk0NTE4amUKbmdoY0duZGtLd2xqTm1oSi9INjlCSy92WmxGbWVHWGs3TjlIV1g1V0t3SURBUUFCb3lNd0lUQU9CZ05WSFE4QgpBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FnRUF6OTIxCnhiRTZCKytGZ1dDTGVpbldYQjg0dHdjNUNidG1mazIwWWwvWWRUWDlFMjFFSlNPckFBc3VFbDNTWHQxTEVFbFAKc3J1VzJsdXkzclltdENDa3lxZEFNSW9QYmh0dTBwa01jcGZhUWNNQU0vME9QVS9FWkhzRHZQbnlLWW1ZZTB2Ugo1Y3NHS3F6N3ZIM2lDZFBndUdQSkJyMmJtVHp2RElFYTBOZEpxSDhYVWluTFAwT2JvbGQvdnJtazlXWUxwQW9kCnoyVVhSNWtNV1BXNWI1WmNSR1gzT2twOU5EL2c0SzZPYWJVK1dOdVcrNWtRVVRKYlpsVmx2U1RlWVNSRUFIY2oKV0ZkYVBCaFdpVXRsZHVjb0tQSmhBNXNqcTJibmlhMnB2dW9YQVRISEozU2p4TE5aQ0FjZ1BZc2h6Nm8wVUNpYQp3aUwxQSt2MWFIL25DRm5UYlIydUJnZWl5Uzl1WGh4UzkyYVJoS09ibWkrK25JU1B3b2J4OUpIZWk2UWZralZVCkV3TVo1R1BkWG5mOHdpYUpEdFdRTEJIZkRvcUtDejJReDZIVDV3T3BxWFVZMis1ZEdPS3Z3T1hsSlQ1REU0bVIKSkpBUnNiMytodHk1L3A2OGlpck9MV1RPbTZYdjZ2SWJVZmNENlBvTGI4L2c5QWJTM0NPUzBKV1ltWmRPdzFQZworN2U5dCt4R0lZYTNtRVF6MklrYyt4MHFab1NRbllxYXVVaU1rd1dqSmdMRVhJdTJPOXhsaUJMaDduMzJ1emFjCmFlbEZmTlZMUjBmbGthbFduNVkvczhzSVl1L0lpRTFLWGcvSkw5WTNNS0lmZHplTThTT25maHU3MFg4MU9NZHEKeTZGWjZYZ2tISVhMWXZrNVJ2TWZRbFpxQ21zLzVjRmVWc3lGd1RvPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  namespace: ZGVmYXVsdA==
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklteFhZa1pSYzJkeVEzaGFPVFJJUW1kTU9YaG5jVUpWVmxsTU5VRXpOWEJCUkVOclRGWklVRFozUTFraWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJblpoZFd4MExXRjFkR2dpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pZG1GMWJIUXRZWFYwYUNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExuVnBaQ0k2SWpRME5URXlObVEzTFRrMllXUXRORGcwT0MxaE9XVmhMVEJrT0dGbFpUVTRNRGxrWXlJc0luTjFZaUk2SW5ONWMzUmxiVHB6WlhKMmFXTmxZV05qYjNWdWREcGtaV1poZFd4ME9uWmhkV3gwTFdGMWRHZ2lmUS5DbGE4NTNtTk50Y1hHN1RyTDB0ejRwS1Q1ekQyWEVnQ09YUFJyVHV5MFBjRmJaX3ZkM25vcjk2bkE0dF91X2pRYW4ySW1zYmduTVNRSmNFLTVpcmJ4b2Jtc3BodnYwWFdfcXMwZm5aLWlKQTVOd3NaMi1jWjN4bHBZdDQ1UlUtb3o4Yk4yTUNWdVZ6S0ZQTHhQZWV0M1gxZlQ1RUdkckpvd3QwendUZjNDdWVrU3ZPUVZYa2FaTXJIM0pBQjB1RURWWXFRb05aRlJpODhHYVl5RjdQZHhuVHY2TTFmeDNJRWlMc1FQM0h4OVFtSVBNOWFFWHZrQ1g1NklIbHpUdnprQ19WMl9UaEFmb01xODhWZXNGbGxqZVBtUElOem1lRkNEUDA3S2xEX3N2Q3RwZUxpcGdXMzc4RjZZUTF0OGpVekl4YU10NTBSZEZ1eWtrVUpjdHctcjN4QXFSeUdkblBxRG1vdldfX1preXJYemwtelBpSHhKcjhsSzdBNkVXNS1iS1RUTEJidi1hejNYOUZzZTNhTGtrakRUdVkyTVdlNWZZcEMyV3J6SFJ6dFMtaFNJX05jUnM1MXF1ajhWd1lsUXlWMDYyaXBaSXdHdFNheERQdW9EaXRzSHg3RkhyeXJPei1wQWVMbS1WN0ZaV3JjdGR3b0xaeXJfY3g5R2I1U3kzVHFvWmFBeEVJV2szR3Znd2g3M3cyZmc3ck5JdmZCOFhJYnhadl9UZlV6WEhIZGgyVFVkODBnVkRWT1NDa3EtakJFMFJCMDYtVDRPWEwxLWN1MFU4bDhXcXdmbDI1ZmVaWTczbjVPZWl4dzU2cTU2eEFxdTdDSlp2czhSVnA2aHhmWU1UN1piSzdjYUtCX3E3SFp1WEljS3pQTzQySWwxZ3YxeWZqcU00RQ==
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: vault-auth
    kubernetes.io/service-account.uid: 445126d7-96ad-4848-a9ea-0d8aee5809dc
  creationTimestamp: "2020-10-06T11:43:57Z"
  name: vault-auth
  namespace: default
  resourceVersion: "2913"
  selfLink: /api/v1/namespaces/default/secrets/vault-auth
  uid: 1b8dd7e1-7372-48ea-b1ae-522e422a1e6b
type: kubernetes.io/service-account-token

$ kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode
eyJhbGciOiJSUzI1NiIsImtpZCI6ImxXYkZRc2dyQ3haOTRIQmdMOXhncUJVVllMNUEzNXBBRENrTFZIUDZ3Q1kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ0NTEyNmQ3LTk2YWQtNDg0OC1hOWVhLTBkOGFlZTU4MDlkYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.Cla853mNNtcXG7TrL0tz4pKT5zD2XEgCOXPRrTuy0PcFbZ_vd3nor96nA4t_u_jQan2ImsbgnMSQJcE-5irbxobmsphvv0XW_qs0fnZ-iJA5NwsZ2-cZ3xlpYt45RU-oz8bN2MCVuVzKFPLxPeet3X1fT5EGdrJowt0zwTf3CuekSvOQVXkaZMrH3JAB0uEDVYqQoNZFRi88GaYyF7PdxnTv6M1fx3IEiLsQP3Hx9QmIPM9aEXvkCX56IHlzTvzkC_V2_ThAfoMq88VesFlljePmPINzmeFCDP07KlD_svCtpeLipgW378F6YQ1t8jUzIxaMt50RdFuykkUJctw-r3xAqRyGdnPqDmovW__ZkyrXzl-zPiHxJr8lK7A6EW5-bKTTLBbv-az3X9Fse3aLkkjDTuY2MWe5fYpC2WrzHRztS-hSI_NcRs51quj8VwYlQyV062ipZIwGtSaxDPuoDitsHx7FHryrOz-pAeLm-V7FZWrctdwoLZyr_cx9Gb5Sy3TqoZaAxEIWk3Gvgwh73w2fg7rNIvfB8XIbxZv_TfUzXHHdh2TUd80gVDVOSCkq-jBE0RB06-T4OXL1-cu0U8l8Wqwfl25feZY73n5Oeixw56q56xAqu7CJZvs8RVp6hxfYMT7ZbK7caKB_q7HZuXIcKzPO42Il1gv1yfjqM4E

(note, no newline at the end of that)

$ export TOKEN_REVEW_JWT=`kubectl get secret vault-auth -o go-template='{{ .dat
a.token }}' | base64 --decode`

$ export | grep JWT
declare -x TOKEN_REVEW_JWT="eyJhbGciOiJSUzI1NiIsImtpZCI6ImxXYkZRc2dyQ3haOTRIQmdMOXhncUJVVllMNUEzNXBBRENrTFZIUDZ3Q1kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ0NTEyNmQ3LTk2YWQtNDg0OC1hOWVhLTBkOGFlZTU4MDlkYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.Cla853mNNtcXG7TrL0tz4pKT5zD2XEgCOXPRrTuy0PcFbZ_vd3nor96nA4t_u_jQan2ImsbgnMSQJcE-5irbxobmsphvv0XW_qs0fnZ-iJA5NwsZ2-cZ3xlpYt45RU-oz8bN2MCVuVzKFPLxPeet3X1fT5EGdrJowt0zwTf3CuekSvOQVXkaZMrH3JAB0uEDVYqQoNZFRi88GaYyF7PdxnTv6M1fx3IEiLsQP3Hx9QmIPM9aEXvkCX56IHlzTvzkC_V2_ThAfoMq88VesFlljePmPINzmeFCDP07KlD_svCtpeLipgW378F6YQ1t8jUzIxaMt50RdFuykkUJctw-r3xAqRyGdnPqDmovW__ZkyrXzl-zPiHxJr8lK7A6EW5-bKTTLBbv-az3X9Fse3aLkkjDTuY2MWe5fYpC2WrzHRztS-hSI_NcRs51quj8VwYlQyV062ipZIwGtSaxDPuoDitsHx7FHryrOz-pAeLm-V7FZWrctdwoLZyr_cx9Gb5Sy3TqoZaAxEIWk3Gvgwh73w2fg7rNIvfB8XIbxZv_TfUzXHHdh2TUd80gVDVOSCkq-jBE0RB06-T4OXL1-cu0U8l8Wqwfl25feZY73n5Oeixw56q56xAqu7CJZvs8RVp6hxfYMT7ZbK7caKB_q7HZuXIcKzPO42Il1gv1yfjqM4E"

$ export KUBE_CA_CERT=`kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode`

And you’ll see this is the formatted multi-line version of the ca cert:

$ export | head -n 34 | tail -n 29
declare -x KUBE_CA_CERT="-----BEGIN CERTIFICATE-----
MIIEyTCCArGgAwIBAgIQOu1i6cGU1SkZsMlTvcxjeDANBgkqhkiG9w0BAQsFADAN
MQswCQYDVQQDEwJjYTAgFw0yMDEwMDYxMTE4NDlaGA8yMDUwMTAwNjExMjg0OVow
DTELMAkGA1UEAxMCY2EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDV
6jmB7abuTFsGEHPmYtxDSW3neoQUmTdbEw3XaGrleWo4r1fL5RimvZBK/c1zPJ17
Dj8BBvDchlfGhJDqX1/H5MTfyg0ACJJues/nCkfSJ9ydpoMvsnsOvtVRJfKlL6Vz
yPQ8J5EvxLcDUF84d3sQtBkqb35LudvPCeA1Jjncj+wz5uExSWS2BNTKLHWB6bPu
1I2BUAXonnRe5Hs0hCfEgJpd83Q+/wIsKP/mcMhBN5ClmjfJMHov/ZHmQpPrv8NC
wgIM8jbUOEaiwsjYqt40S4658lfJq2kKb2OdIsPrSxfgWyXtMXxsopdubD0NmoJN
MugTdxV2JqSWbE4vaBZ2ynS6rq8w5wnOfpJ/DdOQsWhkTyNYnVLsHzccJ+X4/Fvu
SVwMU/cSbwJPZiJl1xa58Y5Pp8dse6qwk9n9e/udQ7wtp74BOjCbZg5WvsuXeVKQ
+FSicMGJdY3tsy7vsvdB5CRiLBpGVuxCnGYvCROWiyF6iZkuEomF5XU0S5jpNNm2
bOElLd4ipK0uMnuN6WPU30LF8qQKGxL9gIHDZiqgfX7oW/4l6g3Av/9/zHjNBzLE
86KrOEEfg26qvDB10MIb1BqB8/JHrEPQxf51+3ivgHyju2grV9czfmjwNY4518je
nghcGndkKwljNmhJ/H69BK/vZlFmeGXk7N9HWX5WKwIDAQABoyMwITAOBgNVHQ8B
Af8EBAMCAqQwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAgEAz921
xbE6B++FgWCLeinWXB84twc5Cbtmfk20Yl/YdTX9E21EJSOrAAsuEl3SXt1LEElP
sruW2luy3rYmtCCkyqdAMIoPbhtu0pkMcpfaQcMAM/0OPU/EZHsDvPnyKYmYe0vR
5csGKqz7vH3iCdPguGPJBr2bmTzvDIEa0NdJqH8XUinLP0Obold/vrmk9WYLpAod
z2UXR5kMWPW5b5ZcRGX3Okp9ND/g4K6OabU+WNuW+5kQUTJbZlVlvSTeYSREAHcj
WFdaPBhWiUtlducoKPJhA5sjq2bnia2pvuoXATHHJ3SjxLNZCAcgPYshz6o0UCia
wiL1A+v1aH/nCFnTbR2uBgeiyS9uXhxS92aRhKObmi++nISPwobx9JHei6QfkjVU
EwMZ5GPdXnf8wiaJDtWQLBHfDoqKCz2Qx6HT5wOpqXUY2+5dGOKvwOXlJT5DE4mR
JJARsb3+hty5/p68iirOLWTOm6Xv6vIbUfcD6PoLb8/g9AbS3COS0JWYmZdOw1Pg
+7e9t+xGIYa3mEQz2Ikc+x0qZoSQnYqauUiMkwWjJgLEXIu2O9xliBLh7n32uzac
aelFfNVLR0flkalWn5Y/s8sIYu/IiE1KXg/JL9Y3MKIfdzeM8SOnfhu70X81OMdq
y6FZ6XgkHIXLYvk5RvMfQlZqCms/5cFeVsyFwTo=
-----END CERTIFICATE-----"
declare -x LANG="C.UTF-8"

Lastly, set our host:

$ export KUBE_HOST=`kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}'`

$ export | grep KUBE_HOST
declare -x KUBE_HOST="https://ijvaultaks-ijvaultk8srg-70b42e-1ff86476.hcp.centralus.azmk8s.io:443"

We need to actually set these on the vault server…

I’ll head back to the first cluster… you can use az aks get-credentials but often i keep a local copy of my config to make switching easier...

$ cp ~/.kube/vse-vault1 ~/.kube/config

Go to the vault server and lets setup the values there. note, i did a "tr" to get rid of a newline that vi leaves in there.

$ kubectl exec -it vault-0 -- /bin/sh
/ $ vim /tmp/KUBE_CA_CERT
/bin/sh: vim: not found
/ $ vi /tmp/KUBE_CA_CERT
/ $ vi /tmp/TOKEN_REVIEW_JWT
/ $ cat /tmp/TOKEN_REVIEW_JWT | tr -d '\n' > /tmp/TOKEN_REVIEW_JWT2
/ $ diff /tmp/TOKEN_REVIEW_JWT /tmp/TOKEN_REVIEW_JWT2
--- /tmp/TOKEN_REVIEW_JWT
+++ /tmp/TOKEN_REVIEW_JWT2
@@ -1 +1 @@
-eyJhbGciOiJSUzI1NiIsImtpZCI6ImxXYkZRc2dyQ3haOTRIQmdMOXhncUJVVllMNUEzNXBBRENrTFZIUDZ3Q1kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ0NTEyNmQ3LTk2YWQtNDg0OC1hOWVhLTBkOGFlZTU4MDlkYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.Cla853mNNtcXG7TrL0tz4pKT5zD2XEgCOXPRrTuy0PcFbZ_vd3nor96nA4t_u_jQan2ImsbgnMSQJcE-5irbxobmsphvv0XW_qs0fnZ-iJA5NwsZ2-cZ3xlpYt45RU-oz8bN2MCVuVzKFPLxPeet3X1fT5EGdrJowt0zwTf3CuekSvOQVXkaZMrH3JAB0uEDVYqQoNZFRi88GaYyF7PdxnTv6M1fx3IEiLsQP3Hx9QmIPM9aEXvkCX56IHlzTvzkC_V2_ThAfoMq88VesFlljePmPINzmeFCDP07KlD_svCtpeLipgW378F6YQ1t8jUzIxaMt50RdFuykkUJctw-r3xAqRyGdnPqDmovW__ZkyrXzl-zPiHxJr8lK7A6EW5-bKTTLBbv-az3X9Fse3aLkkjDTuY2MWe5fYpC2WrzHRztS-hSI_NcRs51quj8VwYlQyV062ipZIwGtSaxDPuoDitsHx7FHryrOz-pAeLm-V7FZWrctdwoLZyr_cx9Gb5Sy3TqoZaAxEIWk3Gvgwh73w2fg7rNIvfB8XIbxZv_TfUzXHHdh2TUd80gVDVOSCkq-jBE0RB06-T4OXL1-cu0U8l8Wqwfl25feZY73n5Oeixw56q56xAqu7CJZvs8RVp6hxfYMT7ZbK7caKB_q7HZuXIcKzPO42Il1gv1yfjqM4E
+eyJhbGciOiJSUzI1NiIsImtpZCI6ImxXYkZRc2dyQ3haOTRIQmdMOXhncUJVVllMNUEzNXBBRENrTFZIUDZ3Q1kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ0NTEyNmQ3LTk2YWQtNDg0OC1hOWVhLTBkOGFlZTU4MDlkYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.Cla853mNNtcXG7TrL0tz4pKT5zD2XEgCOXPRrTuy0PcFbZ_vd3nor96nA4t_u_jQan2ImsbgnMSQJcE-5irbxobmsphvv0XW_qs0fnZ-iJA5NwsZ2-cZ3xlpYt45RU-oz8bN2MCVuVzKFPLxPeet3X1fT5EGdrJowt0zwTf3CuekSvOQVXkaZMrH3JAB0uEDVYqQoNZFRi88GaYyF7PdxnTv6M1fx3IEiLsQP3Hx9QmIPM9aEXvkCX56IHlzTvzkC_V2_ThAfoMq88VesFlljePmPINzmeFCDP07KlD_svCtpeLipgW378F6YQ1t8jUzIxaMt50RdFuykkUJctw-r3xAqRyGdnPqDmovW__ZkyrXzl-zPiHxJr8lK7A6EW5-bKTTLBbv-az3X9Fse3aLkkjDTuY2MWe5fYpC2WrzHRztS-hSI_NcRs51quj8VwYlQyV062ipZIwGtSaxDPuoDitsHx7FHryrOz-pAeLm-V7FZWrctdwoLZyr_cx9Gb5Sy3TqoZaAxEIWk3Gvgwh73w2fg7rNIvfB8XIbxZv_TfUzXHHdh2TUd80gVDVOSCkq-jBE0RB06-T4OXL1-cu0U8l8Wqwfl25feZY73n5Oeixw56q56xAqu7CJZvs8RVp6hxfYMT7ZbK7caKB_q7HZuXIcKzPO42Il1gv1yfjqM4E
\ No newline at end of file

Set the env vars here

/ $ export KUBE_HOST=https://ijvaultaks-ijvaultk8srg-70b42e-1ff86476.hcp.centralus.azmk8s.io:443
/ $ export TOKEN_REVIEW_JWT=`cat /tmp/TOKEN_REVIEW_JWT2`
/ $ export KUBE_CA_CERT=`cat /tmp/KUBE_CA_CERT`

Create a path and launch the auth. This is necessary because we already have a kubernetes auth in use by the cluster that hosts Vault.

/ $ vault auth enable --path="aks2" kubernetes
Success! Enabled kubernetes auth method at: aks2/
/ $ vault write auth/aks2/config token_reviewer_jwt="$TOKEN_REVIEW_JWT" kubernetes_host="$KUBE_HOST" kubernetes_ca_cert="$KUBE_CA_CERT"

We next will add a policy

/ $ vault policy write devwebapp - <<EOF
> path "secret/data/devwebapp/config" {
>   capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: devwebapp

We need to set the secret next…

/ $ vault kv put secret/devwebapp/config username='howdy' password='salasanaapina'
Key              Value
---              -----
created_time     2020-10-06T12:04:48.71500306Z
deletion_time    n/a
destroyed        false
version          1
/ $  vault read -format json secret/data/devwebapp/config
{
  "request_id": "ebae5365-646c-1a66-dc6c-267a3ad7afea",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "password": "salasanaapina",
      "username": "howdy"
    },
    "metadata": {
      "created_time": "2020-10-06T12:04:48.71500306Z",
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "warnings": null
}

We can write a role to that auth endpoint;

/ $ vault write auth/aks2/role/devweb-app \
> bound_service_account_names=internal-app \
> bound_service_account_namespaces=default \
> policies=devwebapp \
> ttl=24h
Success! Data written to: auth/aks2/role/devweb-app

Now we head back to our AKS2 cluster:

$ az aks list -o table
Name         Location    ResourceGroup    KubernetesVersion    ProvisioningState    Fqdn
-----------  ----------  ---------------  -------------------  -------------------  ---------------------------------------------------------------
ijvaultaks   centralus   ijvaultk8srg     1.17.11              Succeeded            ijvaultaks-ijvaultk8srg-70b42e-e2c32296.hcp.centralus.azmk8s.io
ijvaultaks2  centralus   ijvaultk8srg     1.17.11              Succeeded            ijvaultaks-ijvaultk8srg-70b42e-1ff86476.hcp.centralus.azmk8s.io
$ rm -f ~/.kube/config && az aks get-credentials -n ijvaultaks2 -g ijvaultk8srg --admin
Merged "ijvaultaks2-admin" as current context in /home/builder/.kube/config

Install Vault pointing at our external IP for Vault in the other cluster…

$ helm install vault hashicorp/vault --set "injector.externalVaultAddr=http://52.154.254.196:8200"
NAME: vault
LAST DEPLOYED: Tue Oct  6 07:09:18 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get vault

Next, as before, we can make an internal-app service account and launch a deployment.

$ cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: internal-app
> EOF
serviceaccount/internal-app created
$ vi devwebapp.yaml
$ cat devwebapp.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: devwebapp
  labels:
    app: devwebapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devwebapp
  template:
    metadata:
      labels:
        app: devwebapp
    spec:
      serviceAccountName: internal-app
      containers:
      - name: app
        image: burtlo/devwebapp-ruby:k8s
        imagePullPolicy: Always
        env:
        - name: VAULT_ADDR
          value: "http://52.154.254.196:8200"
$ kubectl apply -f devwebapp.yaml
deployment.apps/devwebapp created

We can patch the deployment for the annotations we need.

$ cat patch-secrets.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "devweb-app"
        vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"
$ kubectl patch deployment devwebapp --patch "$(cat patch-secrets.yaml)"
deployment.apps/devwebapp patched

Again, we get errors.. This time it's about the role.. Why is it stuck to /auth/kubernetes?

$ kubectl logs devwebapp-b7df56946-2p4jj vault-agent-init
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.5.2
2020-10-06T12:16:35.549Z [INFO]  sink.file: creating file sink
2020-10-06T12:16:35.549Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2020-10-06T12:16:35.549Z [INFO]  auth.handler: starting auth handler
2020-10-06T12:16:35.549Z [INFO]  auth.handler: authenticating
             Version Sha: 685fdfa60d607bca069c09d2d52b6958a7a2febd

2020-10-06T12:16:35.550Z [INFO]  template.server: starting template server
2020/10/06 12:16:35.550320 [INFO] (runner) creating new runner (dry: false, once: false)
2020-10-06T12:16:35.550Z [INFO]  sink.server: starting sink server
2020/10/06 12:16:35.550903 [INFO] (runner) creating watcher
2020-10-06T12:16:35.553Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://52.154.254.196:8200/v1/auth/kubernetes/login
Code: 400. Errors:

* invalid role name "devweb-app"" backoff=2.084195088
2020-10-06T12:16:37.637Z [INFO]  auth.handler: authenticating
2020-10-06T12:16:37.641Z [ERROR] auth.handler: error authenticating: error="Error making API request.

And here is where I found the issue.  There are a lot of config variables one can use. In looking through the annotations, I found the problem was the default auth is “auth/kubernetes” which is fine in a single cluster, but not so hot when we have multiple clusters.

I changed to add an “auth-path” block:

$ cat patch-secrets.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/auth-path: "auth/aks2"
        vault.hashicorp.com/role: "devweb-app"
        vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"

Patched

$ kubectl patch deployment devwebapp --patch "$(cat patch-secrets.yaml)"
deployment.apps/devwebapp patched

We can now see it running:

$ kubectl get pods
NAME                                   READY   STATUS        RESTARTS   AGE
devwebapp-6b9bbcf5b6-4gxjs             0/1     Terminating   0          8m25s
devwebapp-867d67856c-grw7k             2/2     Running       0          6s
devwebapp-b7df56946-2p4jj              0/2     Terminating   0          5m2s
vault-agent-injector-8697d97cf-q5nsl   1/1     Running       0          12m

And if we check the app container in the running pod, we can see the credentials were successfully brought over:

$ kubectl exec devwebapp-867d67856c-grw7k -c app -- cat /vault/secrets/credentials.txt
data: map[password:salasanaapina username:howdy]
metadata: map[created_time:2020-10-06T12:04:48.71500306Z deletion_time: destroyed:false version:1]

Cleanup

You can easily cleanup your clusters with the command line.

$ az aks delete --resource-group ijvaultk8srg --name ijvaultaks2
Are you sure you want to perform this operation? (y/n): y
 - Running ..

Storage of Secrets

In discussing this with a colleague it was brought up "where does vault actually store the secrets"?

Vault, in this case, uses a Persistant Volume Claim:

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-vault-0   Bound    pvc-0adbb0ad-73fa-422a-a079-50c06b1746d8   10Gi       RWO            default        4d19h

This will leverage the default storage class of the cluster (which for AKS, by default, is Azure Disk):

You can override this in AKS:

$ kubectl patch storageclass azurefile-premium -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/azurefile-premium patched
$ kubectl patch storageclass default -p '{"metadata": {"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"false", "kubectl.kubernetes.io/last-applied-configuration": "{\"allowVolumeExpansion\":true,\"apiVersion\":\"storage.k8s.io/v1beta1\",\"kind\":\"StorageClass\",\"metadata\":{\"annotations\":{\"storageclass.beta.kubernetes.io/is-default-class\":\"false\"},\"labels\":{\"kubernetes.io/cluster-service\":\"true\"},\"name\":\"default\"},\"parameters\":{\"cachingmode\":\"ReadOnly\",\"kind\":\"Managed\",\"storageaccounttype\":\"StandardSSD_LRS\"},\"provisioner\":\"kubernetes.io/azure-disk\"}"}}}'
storageclass.storage.k8s.io/default patched

$ kubectl get sc
NAME                          PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
azurefile                     kubernetes.io/azure-file   Delete          Immediate           true                   4d19h
azurefile-premium (default)   kubernetes.io/azure-file   Delete          Immediate           true                   4d19h
default                       kubernetes.io/azure-disk   Delete          Immediate           true                   4d19h
managed-premium               kubernetes.io/azure-disk   Delete          Immediate           true                   4d19h

Because we used Auto-Unseal, instead of shamir with unlock keys in process, Vault will leaverage it's vault key in AKV to handle unsealing after reboot.

Summary

In this post we explored Vault in Kubernetes a little more.  We created a database user and password and exposed it to a deployment.  We leveraged the “template” feature to format the string for a variety of common database engines and showed how to mount secrets on an interactive ubuntu shell.  Lastly, we showed how to expose Vault with a public IP, restrict with an NSG and lastly connected a second kubernetes cluster to our primary vault agent and after some debugging, figured out how to launch a deployment that could retrieve from an external containerized vault server.

This shows us the power of the Open Source Vault agent and more ways we can use it in a Kubernetes environment. Ideally Vault uses Consul as it’s mesh network and in a proper commercial instance, we would not expose Vault directly (instead relying on a Consul cluster to bridge clouds).  However, this shows us how we can get started using an auto-unsealing containerized Hashi Vault in Kubernetes.