Vault on Kubernetes: Getting Started

Published: Oct 3, 2020 by Isaac Johnson

Hashi Vault has long been a leader in secret management and has had a Kubernetes Helm Chart now for a year. I’ve been meaning to get back to Vault for some time and this week I took the time to take a tour of Vault on Azure Kubernetes Service.

Getting Started

First, let’s spin up the basics in Azure. We’ve covered this enough time I’ll just abbreviate the steps:

$ az group create --name ijvaultk8srg --location centralus
$ az ad sp create-for-rbac -n ijvaultk8ssp --skip-assignment --output json > my_sp.json
$ export SP_PASS=`cat my_sp.json | jq -r .password`
$ export SP_ID=`cat my_sp.json | jq -r .appId`
$ az aks create --resource-group ijvaultk8srg --name ijvaultaks --location centralus --node-count 3 --enable-cluster-autoscaler --min-count 2 --max-count 4 --generate-ssh-keys --network-plugin azure --network-policy azure --service-principal $SP_ID --client-secret $SP_PASS

Verification

We can verify our config by checking the node pool.

$ az aks get-credentials -n ijvaultaks -g ijvaultk8srg --admin
Merged "ijvaultaks-admin" as current context in /home/builder/.kube/config

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-84770334-vmss000000 Ready agent 4m10s v1.17.11
aks-nodepool1-84770334-vmss000001 Ready agent 4m9s v1.17.11
aks-nodepool1-84770334-vmss000002 Ready agent 4m10s v1.17.11

Installing Hashi Vault

We need to add the Helm repo and install the chart

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

$ helm install vault hashicorp/vault
NAME: vault
LAST DEPLOYED: Thu Oct 1 18:42:19 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get vault

If we get pods, we’ll see vault is up, but not “ready”

$ kubectl get pods
NAMESPACE NAME READY STATUS RESTARTS AGE
default vault-0 0/1 Running 0 5m13s
default vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 62s

If we get the vault status, we’ll see it’s presently sealed:

$ kubectl exec vault-0 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version n/a
HA Enabled false
command terminated with exit code 2

Next we can to go to the UI:

$ kubectl port-forward vault-0 8200:8200
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200
Handling connection for 8200
Handling connection for 8200
Handling connection for 8200
Handling connection for 8200
Handling connection for 8200

We will use 3 shares with 2 shares minimum and initialize. You can choose to download the keys at this point:

builder@DESKTOP-2SQ9NQM:/mnt/c/Users/isaac/Downloads$ cat vault-cluster-vault-2020-10-01T23_52_23.801Z.json
{
  "keys": [
    "c725ad463cfab00fed28284c7bcb60be4ba4889f003da744e6d7d4eceb8a203918",
    "4b9082a3039a6a56c73b1f3b91a47c39bdab9d215cdfa7ee42b108dad10b1c1711",
    "31c9ec8ad33393dcae71907ab1043fe8dea224d3d1a0a70000eb02f0062938dceb"
  ],
  "keys_base64": [
    "xyWtRjz6sA/tKChMe8tgvkukiJ8APadE5tfU7OuKIDkY",
    "S5CCowOaalbHOx87kaR8Ob2rnSFc36fuQrEI2tELHBcR",
    "McnsitMzk9yucZB6sQQ/6N6iJNPRoKcAAOsC8AYpONzr"
  ],
  "root_token": "s.AghmoNp4pAQekX7ADcM3NID9"

We can also do this on the command line as well:

$ kubectl exec vault-0 -- vault operator init -key-shares=3 -key-threshold=2 -format=json > cluster-keys.json
$ VAULT_UNSEAL_KEY1=$(cat vault-cluster-vault-2020-10-01T23_52_23.801Z.json | jq -r '.keys_base64[0]')
$ VAULT_UNSEAL_KEY2=$(cat vault-cluster-vault-2020-10-01T23_52_23.801Z.json | jq -r '.keys_base64[1]')
$ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY1
$ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY2

Once we unseal with two tokens, we can login with the root token:

Let’s login on the pod itself and enable secrets engine:

$ kubectl exec -it vault-0 -- /bin/sh
/ $ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token s.AghmoNp4pAQekX7ADcM3NID9
token_accessor yzz0shnNSMnXIK1mQ7b4JogB
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
/ $ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

Let’s set a key and then retrieve it:

/ $ vault kv put secret/webapp/config username="myUser" password="myPassword"
Key Value
--- -----
created_time 2020-10-02T00:59:57.502286049Z
deletion_time n/a
destroyed false
version 1
/ $ vault kv get secret/webapp/config
====== Metadata ======
Key Value
--- -----
created_time 2020-10-02T00:59:57.502286049Z
deletion_time n/a
destroyed false
version 1

====== Data ======
Key Value
--- -----
password myPassword
username myUser

We can also enable kubernetes pods to use vault at this time as well

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

/ $ echo $KUBERNETES_PORT_443_TCP_ADDR
10.0.0.1
/ $ vault write auth/kubernetes/config \
> token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
> kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
> kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

Set a read policy

/ $ vault policy write webapp - <<EOF
> path "secret/data/webapp/config" {
> capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: webapp

Applying that policy to a role:

/ $ vault write auth/kubernetes/role/webapp \
> bound_service_account_names=vault \
> bound_service_account_namespaces=default \
> policies=webapp \
> ttl=24h
Success! Data written to: auth/kubernetes/role/webapp

We can now use a webapp to test. We can use the one out of the Hashi guide

$ cat deployment-01-webapp.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      serviceAccountName: vault
      containers:
      - name: app
        image: burtlo/exampleapp-ruby:k8s
        imagePullPolicy: Always
        env:
        - name: VAULT_ADDR
          value: "http://vault:8200"
        - name: JWT_PATH
          value: "/var/run/secrets/kubernetes.io/serviceaccount/token"
        - name: SERVICE_PORT
          value: "8080"
$ kubectl apply -f deployment-01-webapp.yml
deployment.apps/webapp created

Let’s test

$ kubectl port-forward `kubectl get pod -l app=webapp -o jsonpath="{.items[0].metadata.name}"` 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080
Handling connection for 8080

And if we change it

$ kubectl exec -it vault-0 -- /bin/sh
/ $ vault kv put secret/webapp/config username="static-user" password="changed"
Key Value
--- -----
created_time 2020-10-02T20:09:22.22298837Z
deletion_time n/a
destroyed false
version 2
/ $

we can use curl just as easily to see it reflected

$ curl http://localhost:8080
{"password"=>"changed", "username"=>"static-user"}

And this is a fine solution for a basic test, however if our Vault pod goes down, it will come back up sealed.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 20h
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 20h
webapp-66bd6d455d-6rnfm 1/1 Running 0 7m43s

$ kubectl delete pod vault-0
pod "vault-0" deleted

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 ContainerCreating 0 6s
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 20h
webapp-66bd6d455d-6rnfm 1/1 Running 0 8m7s

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 107s
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 20h
webapp-66bd6d455d-6rnfm 1/1 Running 0 9m48s

$ kubectl logs webapp-66bd6d455d-6rnfm
[2020-10-02 20:03:29] INFO WEBrick 1.4.2
[2020-10-02 20:03:29] INFO ruby 2.6.2 (2019-03-13) [x86_64-linux]
[2020-10-02 20:03:29] INFO WEBrick::HTTPServer#start: pid=1 port=8080
2020-10-02 20:13:03 - NoMethodError - undefined method `[]' for nil:NilClass:
        /app/lib/service.rb:51:in `block in <class:ExampleApp>'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1635:in `call'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1635:in `block in compile!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:987:in `block (3 levels) in route!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1006:in `route_eval'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:987:in `block (2 levels) in route!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1035:in `block in process_route'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1033:in `catch'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1033:in `process_route'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:985:in `block in route!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:984:in `each'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:984:in `route!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1097:in `block in dispatch!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1071:in `block in invoke'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1071:in `catch'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1071:in `invoke'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1094:in `dispatch!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:919:in `block in call!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1071:in `block in invoke'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1071:in `catch'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1071:in `invoke'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:919:in `call!'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:908:in `call'
        /usr/local/bundle/gems/rack-protection-2.0.7/lib/rack/protection/xss_header.rb:18:in `call'
        /usr/local/bundle/gems/rack-protection-2.0.7/lib/rack/protection/path_traversal.rb:16:in `call'
        /usr/local/bundle/gems/rack-protection-2.0.7/lib/rack/protection/json_csrf.rb:26:in `call'
        /usr/local/bundle/gems/rack-protection-2.0.7/lib/rack/protection/base.rb:50:in `call'
        /usr/local/bundle/gems/rack-protection-2.0.7/lib/rack/protection/base.rb:50:in `call'
        /usr/local/bundle/gems/rack-protection-2.0.7/lib/rack/protection/frame_options.rb:31:in `call'
        /usr/local/bundle/gems/rack-2.0.7/lib/rack/null_logger.rb:9:in `call'
        /usr/local/bundle/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:194:in `call'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1950:in `call'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1502:in `block in call'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1729:in `synchronize'
        /usr/local/bundle/gems/sinatra-2.0.7/lib/sinatra/base.rb:1502:in `call'
        /usr/local/bundle/gems/rack-2.0.7/lib/rack/handler/webrick.rb:86:in `service'
        /usr/local/lib/ruby/2.6.0/webrick/httpserver.rb:140:in `service'
        /usr/local/lib/ruby/2.6.0/webrick/httpserver.rb:96:in `run'
        /usr/local/lib/ruby/2.6.0/webrick/server.rb:307:in `block in start_thread'

If we follow the instructions from above, we can unseal and make the apps usable again:

$ VAULT_UNSEAL_KEY1=$(cat vault-cluster-vault-2020-10-01T23_52_23.801Z.json | jq -r '.keys_base64[0]')
$ VAULT_UNSEAL_KEY2=$(cat vault-cluster-vault-2020-10-01T23_52_23.801Z.json | jq -r '.keys_base64[1]')

$ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY1
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 3
Threshold 2
Unseal Progress 1/2
Unseal Nonce 5aba5b42-dde9-a5b9-022d-3fb753e103ad
Version 1.5.2
HA Enabled false
$ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY2
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 3
Threshold 2
Version 1.5.2
Cluster Name vault-cluster-c20f2dc8
Cluster ID 001a2736-d40e-6bd9-8ca6-b3a60afd4e5c
HA Enabled false

And we can see the pods

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 6m58s
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 20h
webapp-66bd6d455d-6rnfm 1/1 Running 0 14m

Setting HA

Let’s create an AKV backend.

First we need an AKV to use

$ az keyvault create -n ijvaultakv -g ijvaultk8srg
{
  "id": "/subscriptions/70b42e6a-6faf-4fed-bcec-9f3995b1aca8/resourceGroups/ijvaultk8srg/providers/Microsoft.KeyVault/vaults/ijvaultakv",
  "location": "centralus",
  "name": "ijvaultakv",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "170d4247-9bca-4e26-82f9-919c7ec00101",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "d73a39db-6eda-495d-8000-7579f56d68b7"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableSoftDelete": true,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": null,
    "networkAcls": null,
    "privateEndpointConnections": null,
    "provisioningState": "Succeeded",
    "sku": {
      "name": "standard"
    },
    "tenantId": "d73a39db-6eda-495d-8000-7579f56d68b7",
    "vaultUri": "https://ijvaultakv.vault.azure.net/"
  },
  "resourceGroup": "ijvaultk8srg",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

We can now create a “seal” block in the config for auto-unseal:

    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "file" {
        path = "/vault/data"
      }
      seal "azurekeyvault" {
        client_id = "4b3095be-8939-4e49-9a43-81fe1580b21c"
        tenant_id = "aabbccdd-eeff-1122-3344-5566778899aa"
        client_secret = "aabbccdd-d4aa-4bef-8598-5566778899aa"
        vault_name = "ijvaultakv"
        key_name = "vaultkey"
      }

We then need to make a key… that word is so overloaded.. I kept thinking that we needed to create a “secret” with the shamir key value. But no, what it needs is a real 2048 RSA key in the keyvault made.

Let’s check the logs

$ kubectl logs vault-0
==> Vault server configuration:

       Azure Environment: AzurePublicCloud
          Azure Key Name: vaultkey
        Azure Vault Name: ijvaultakv
             Api Address: http://10.240.0.19:8200
                     Cgo: disabled
         Cluster Address: https://vault-0.vault-internal:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: file
                 Version: Vault v1.5.2
             Version Sha: 685fdfa60d607bca069c09d2d52b6958a7a2febd

==> Vault server started! Log data will stream in below:

2020-10-02T21:43:50.057Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2020-10-02T21:43:50.349Z [WARN] core: entering seal migration mode; Vault will not automatically unseal even if using an autoseal: from_barrier_type=shamir to_barrier_type=azurekeyvault

Another issue i encountered was that the default key permissions do not include unwrap:

/ $ vault operator unseal -migrate
Unseal Key (will be hidden):
Error unsealing: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/unseal
Code: 500. Errors:

* error setting new recovery key information: failed to encrypt keys for storage: keyvault.BaseClient#WrapKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The user, group or application 'appid=4b3095be-8939-4e49-9a43-81fe1580b21c;oid=83b5ef3a-f498-491d-8b89-973321cbe480;iss=https://sts.windows.net/aabbccdd-eeff-1122-3344-5566778899aa/' does not have keys wrapKey permission on key vault 'ijvaultakv;location=centralus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287" InnerError={"code":"ForbiddenByPolicy"}

So add the Cryptographic Operations

I added the rest of the cryto ops

I then relaunched the migrate

/ $ vault operator unseal -migrate
Unseal Key (will be hidden):
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed true
Total Recovery Shares 3
Threshold 2
Unseal Progress 1/2
Unseal Nonce 8257a76d-c65e-dd8a-0e1e-7818b53d9cd8
Seal Migration in Progress true
Version 1.5.2
HA Enabled false
/ $ vault operator unseal -migrate
Unseal Key (will be hidden):
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 3
Threshold 2
Version 1.5.2
Cluster Name vault-cluster-c20f2dc8
Cluster ID 001a2736-d40e-6bd9-8ca6-b3a60afd4e5c
HA Enabled false

We can now see it’s unsealed:

/ $ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 3
Threshold 2
Version 1.5.2
Cluster Name vault-cluster-c20f2dc8
Cluster ID 001a2736-d40e-6bd9-8ca6-b3a60afd4e5c
HA Enabled false

Let’s rotate the pod to make sure it works….

$ kubectl delete pod vault-0
pod "vault-0" deleted
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 ContainerCreating 0 3s
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 22h
webapp-66bd6d455d-6rnfm 1/1 Running 0 114m
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 ContainerCreating 0 19s
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 22h
webapp-66bd6d455d-6rnfm 1/1 Running 0 114m
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 ContainerCreating 0 34s
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 22h
webapp-66bd6d455d-6rnfm 1/1 Running 0 114m
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 45s
vault-agent-injector-bdbf7b844-kqbnp 1/1 Running 0 22h
webapp-66bd6d455d-6rnfm 1/1 Running 0 115m

And indeed as we see about, it auto unsealed.

Summary

Hashi Vault has always been a great a tool for secrets management but it’s native Kubernetes support makes it that much more accessible in a containerized topology. By using Vault as a service within a namespace we can leverage a robust cloud agnostic solution for our secrets. When we add auto-unseal (as we saw above) and HA via Consul we can actually have a solid solution that spans clouds.