Hashi Vault on K8s with MFA

Published: Feb 21, 2023 by Isaac Johnson

Hashi Vault is a robust secrets service that we can use many different ways; as a SaaS offering (HCP Vault), marketplace option in our cloud provider or host it ourselves. Today we’ll setup Vault on Kubernetes with AKV auto-unseal. Then we’ll configure user/pass authentication with MFA using Duo.

Getting an SP Ready

We’re going to jump right in to Hashi Vault HA with AKV backed auto-unsealing. To do this, we need to create a Service Principal.

The common way is something like this

$ az group create --name ijvaultk8srg --location centralus
$ az ad sp create-for-rbac -n ijvaultk8ssp --skip-assignment --output json > my_sp.json
$ export SP_PASS=`cat my_sp.json | jq -r .password`
$ export SP_ID=`cat my_sp.json | jq -r .appId`

However, I’m going to use an existing SP this time that just needs some fresh certs.

Below we see an older SP that has the right name, but has well expired creds:

/content/images/2023/02/vault-01.png

I’ll click “+ Add Client Secret” and give it a name and duration (up to 3y)

/content/images/2023/02/vault-02.png

That will give us a new value we can copy

/content/images/2023/02/vault-03.png

Next, we’ll need a KeyVault to use. As before, we could use the command line

$ az keyvault create -n ijvaultakv -g ijvaultk8srg

However, I’ll use the UI.

/content/images/2023/02/vault-04.png

I’ll create a new resource group and give this AKV a unique name

/content/images/2023/02/vault-05.png

Then click “Create” to create it

/content/images/2023/02/vault-06.png

Our last step is then to grant the SP we used at the start access to Secrets in AKV.

We’ll go to “Access Policies” and click create:

/content/images/2023/02/vault-07.png

I’ll chose “Key, Secret, & Certificate Management” for the template.

/content/images/2023/02/vault-08.png

In the next step, I’ll select the principal we updated.

/content/images/2023/02/vault-09.png

Lastly, I’ll review and create.

/content/images/2023/02/vault-10.png

One other area we will need is Cryptographic Operations. I went and clicked the SP under Access Policies and chose Edit to add them after the fact.

/content/images/2023/02/vault-18.png

(Make sure to hit save to make the changes live)

We’ll end up needing the client_id and client_secret, which comes from the SP credential.

If you used an Enterprise Application, you’ll find the client_id as the “Application ID” in Overview:

/content/images/2023/02/vault-11.png

If you used a standard App Registration, you’ll see the Application ID and Directory (tenant) under a standard “App Registration” in Overview:

/content/images/2023/02/vault-12.png

Key Vault

Let’s now setup Hashi Vault in our Cluster.

I’ll make sure I’m logged into the primary/infrastructure cluster:

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectx mac77
Switched to context "mac77".
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get nodes
NAME                          STATUS   ROLES                  AGE    VERSION
hp-hp-elitebook-850-g2        Ready    <none>                 209d   v1.23.9+k3s1
isaac-macbookair              Ready    control-plane,master   209d   v1.23.9+k3s1
builder-hp-elitebook-850-g2   Ready    <none>                 201d   v1.23.9+k3s1
builder-hp-elitebook-850-g1   Ready    <none>                 209d   v1.23.9+k3s1

Note: If you don’t have an on-prem cluster, I have created a series of articles on setting up k3s, from “Getting Started with K3S”, a more recent K3s rebuild, and more

Let’s now add the Hashi helm chart and update:

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" already exists with the same configuration, skipping
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "freshbrewed" chart repository
...Successfully got an update from the "myharbor" chart repository
...Successfully got an update from the "confluentinc" chart repository
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "dapr" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "uptime-kuma" chart repository
...Successfully got an update from the "sonarqube" chart repository
...Successfully got an update from the "novum-rgi-helm" chart repository
...Successfully got an update from the "adwerx" chart repository
...Successfully got an update from the "rhcharts" chart repository
...Successfully got an update from the "longhorn" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "kubecost" chart repository
...Successfully got an update from the "epsagon" chart repository
...Successfully got an update from the "lifen-charts" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "sumologic" chart repository
...Successfully got an update from the "open-telemetry" chart repository
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "elastic" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "argo-cd" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "crossplane-stable" chart repository
...Successfully got an update from the "newrelic" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "bitnami" chart repository
...Successfully got an update from the "rancher-latest" chart repository
Update Complete. ⎈Happy Helming!⎈

Then we can install it with helm.

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm install vault hashicorp/vault
NAME: vault
LAST DEPLOYED: Mon Feb 20 06:43:14 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

I had expected to see a vault-0 ui pod when launched (as I had before)

However, there is only the injector:

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get pod --all-namespaces | grep vault
default                 vault-agent-injector-54bdddbb94-qrdb5                      1/1     Running            0                  3m42s

I pulled the helm chart all values and noted that now the UI is, by default, disabled

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm get values --all vault
COMPUTED VALUES:
csi:
  daemonSet:
    annotations: {}
    extraLabels: {}
    kubeletRootDir: /var/lib/kubelet
    providersDir: /etc/kubernetes/secrets-store-csi-providers
    securityContext:
      container: {}
      pod: {}
    updateStrategy:
      maxUnavailable: ""
      type: RollingUpdate
  debug: false
  enabled: false
  extraArgs: []
  image:
    pullPolicy: IfNotPresent
    repository: hashicorp/vault-csi-provider
    tag: 1.2.1
  livenessProbe:
    failureThreshold: 2
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 3
  pod:
    annotations: {}
    extraLabels: {}
    tolerations: []
  priorityClassName: ""
  readinessProbe:
    failureThreshold: 2
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 3
  resources: {}
  serviceAccount:
    annotations: {}
    extraLabels: {}
  volumeMounts: null
  volumes: null
global:
  enabled: true
  externalVaultAddr: ""
  imagePullSecrets: []
  openshift: false
  psp:
    annotations: |
      seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default,runtime/default
      apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
      seccomp.security.alpha.kubernetes.io/defaultProfileName:  runtime/default
      apparmor.security.beta.kubernetes.io/defaultProfileName:  runtime/default
    enable: false
  serverTelemetry:
    prometheusOperator: false
  tlsDisable: true
injector:
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: {{ template "vault.name" . }}-agent-injector
              app.kubernetes.io/instance: "{{ .Release.Name }}"
              component: webhook
          topologyKey: kubernetes.io/hostname
  agentDefaults:
    cpuLimit: 500m
    cpuRequest: 250m
    memLimit: 128Mi
    memRequest: 64Mi
    template: map
    templateConfig:
      exitOnRetryFailure: true
      staticSecretRenderInterval: ""
  agentImage:
    repository: hashicorp/vault
    tag: 1.12.1
  annotations: {}
  authPath: auth/kubernetes
  certs:
    caBundle: ""
    certName: tls.crt
    keyName: tls.key
    secretName: null
  enabled: '-'
  externalVaultAddr: ""
  extraEnvironmentVars: {}
  extraLabels: {}
  failurePolicy: Ignore
  hostNetwork: false
  image:
    pullPolicy: IfNotPresent
    repository: hashicorp/vault-k8s
    tag: 1.1.0
  leaderElector:
    enabled: true
  logFormat: standard
  logLevel: info
  metrics:
    enabled: false
  namespaceSelector: {}
  nodeSelector: {}
  objectSelector: {}
  podDisruptionBudget: {}
  port: 8080
  priorityClassName: ""
  replicas: 1
  resources: {}
  revokeOnShutdown: false
  securityContext:
    container: {}
    pod: {}
  service:
    annotations: {}
  serviceAccount:
    annotations: {}
  strategy: {}
  tolerations: []
  topologySpreadConstraints: []
  webhook:
    annotations: {}
    failurePolicy: Ignore
    matchPolicy: Exact
    namespaceSelector: {}
    objectSelector: |
      matchExpressions:
      - key: app.kubernetes.io/name
        operator: NotIn
        values:
        - {{ template "vault.name" . }}-agent-injector
    timeoutSeconds: 30
  webhookAnnotations: {}
server:
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: {{ template "vault.name" . }}
              app.kubernetes.io/instance: "{{ .Release.Name }}"
              component: server
          topologyKey: kubernetes.io/hostname
  annotations: {}
  auditStorage:
    accessMode: ReadWriteOnce
    annotations: {}
    enabled: false
    mountPath: /vault/audit
    size: 10Gi
    storageClass: null
  authDelegator:
    enabled: true
  dataStorage:
    accessMode: ReadWriteOnce
    annotations: {}
    enabled: true
    mountPath: /vault/data
    size: 10Gi
    storageClass: null
  dev:
    devRootToken: root
    enabled: false
  enabled: '-'
  enterpriseLicense:
    secretKey: license
    secretName: ""
  extraArgs: ""
  extraContainers: null
  extraEnvironmentVars: {}
  extraInitContainers: null
  extraLabels: {}
  extraSecretEnvironmentVars: []
  extraVolumes: []
  ha:
    apiAddr: null
    clusterAddr: null
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }

      service_registration "kubernetes" {}

      # Example configuration for using auto-unseal, using Google Cloud KMS. The
      # GKMS keys must already exist, and the cluster must have a service account
      # that is authorized to access GCP KMS.
      #seal "gcpckms" {
      #   project     = "vault-helm-dev-246514"
      #   region      = "global"
      #   key_ring    = "vault-helm-unseal-kr"
      #   crypto_key  = "vault-helm-unseal-key"
      #}

      # Example configuration for enabling Prometheus metrics.
      # If you are using Prometheus Operator you can enable a ServiceMonitor resource below.
      # You may wish to enable unauthenticated metrics in the listener block above.
      #telemetry {
      #  prometheus_retention_time = "30s",
      #  disable_hostname = true
      #}
    disruptionBudget:
      enabled: true
      maxUnavailable: null
    enabled: false
    raft:
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          # Enable unauthenticated metrics access (necessary for Prometheus Operator)
          #telemetry {
          #  unauthenticated_metrics_access = "true"
          #}
        }

        storage "raft" {
          path = "/vault/data"
        }

        service_registration "kubernetes" {}
      enabled: false
      setNodeId: false
    replicas: 3
  hostNetwork: false
  image:
    pullPolicy: IfNotPresent
    repository: hashicorp/vault
    tag: 1.12.1
  ingress:
    activeService: true
    annotations: {}
    enabled: false
    extraPaths: []
    hosts:
    - host: chart-example.local
      paths: []
    ingressClassName: ""
    labels: {}
    pathType: Prefix
    tls: []
  livenessProbe:
    enabled: false
    failureThreshold: 2
    initialDelaySeconds: 60
    path: /v1/sys/health?standbyok=true
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 3
  logFormat: ""
  logLevel: ""
  networkPolicy:
    egress: []
    enabled: false
  nodeSelector: {}
  postStart: []
  preStopSleepSeconds: 5
  priorityClassName: ""
  readinessProbe:
    enabled: true
    failureThreshold: 2
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 3
  resources: {}
  route:
    activeService: true
    annotations: {}
    enabled: false
    host: chart-example.local
    labels: {}
    tls:
      termination: passthrough
  service:
    active:
      enabled: true
    annotations: {}
    enabled: true
    externalTrafficPolicy: Cluster
    instanceSelector:
      enabled: true
    port: 8200
    publishNotReadyAddresses: true
    standby:
      enabled: true
    targetPort: 8200
  serviceAccount:
    annotations: {}
    create: true
    extraLabels: {}
    name: ""
    serviceDiscovery:
      enabled: true
  shareProcessNamespace: false
  standalone:
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        # Enable unauthenticated metrics access (necessary for Prometheus Operator)
        #telemetry {
        #  unauthenticated_metrics_access = "true"
        #}
      }
      storage "file" {
        path = "/vault/data"
      }

      # Example configuration for using auto-unseal, using Google Cloud KMS. The
      # GKMS keys must already exist, and the cluster must have a service account
      # that is authorized to access GCP KMS.
      #seal "gcpckms" {
      #   project     = "vault-helm-dev"
      #   region      = "global"
      #   key_ring    = "vault-helm-unseal-kr"
      #   crypto_key  = "vault-helm-unseal-key"
      #}

      # Example configuration for enabling Prometheus metrics in your config.
      #telemetry {
      #  prometheus_retention_time = "30s",
      #  disable_hostname = true
      #}
    enabled: '-'
  statefulSet:
    annotations: {}
    securityContext:
      container: {}
      pod: {}
  terminationGracePeriodSeconds: 10
  tolerations: []
  topologySpreadConstraints: []
  updateStrategyType: OnDelete
  volumeMounts: null
  volumes: null
serverTelemetry:
  prometheusRules:
    enabled: false
    rules: {}
    selectors: {}
  serviceMonitor:
    enabled: false
    interval: 30s
    scrapeTimeout: 10s
    selectors: {}
ui:
  activeVaultPodOnly: false
  annotations: {}
  enabled: false
  externalPort: 8200
  externalTrafficPolicy: Cluster
  publishNotReadyAddresses: true
  serviceNodePort: null
  serviceType: ClusterIP
  targetPort: 8200

I set the current values asside

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm get values -o yaml --all vault > currentValues.yaml
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm get values -o yaml --all vault > currentValues.yaml.bak

Then I edited it. I noted that Standalone mode was errantly set to “-“ instead of true which is why I likely didn’t get the UI. I also enabled the UI in the UI block. (foreshadowing… this isn’t the issue)

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ diff -c currentValues.yaml currentValues.yaml.bak
*** currentValues.yaml  2023-02-20 06:54:36.402358212 -0600
--- currentValues.yaml.bak      2023-02-20 06:48:55.298980129 -0600
***************
*** 337,343 ****
        #  prometheus_retention_time = "30s",
        #  disable_hostname = true
        #}
!     enabled: true
    statefulSet:
      annotations: {}
      securityContext:
--- 337,343 ----
        #  prometheus_retention_time = "30s",
        #  disable_hostname = true
        #}
!     enabled: '-'
    statefulSet:
      annotations: {}
      securityContext:
***************
*** 362,368 ****
  ui:
    activeVaultPodOnly: false
    annotations: {}
!   enabled: true
    externalPort: 8200
    externalTrafficPolicy: Cluster
    publishNotReadyAddresses: true
--- 362,368 ----
  ui:
    activeVaultPodOnly: false
    annotations: {}
!   enabled: false
    externalPort: 8200
    externalTrafficPolicy: Cluster
    publishNotReadyAddresses: true

I’ll update the installed chart to use the values we set:

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm upgrade vault -f currentValues.yaml hashicorp/vault
Release "vault" has been upgraded. Happy Helming!
NAME: vault
LAST DEPLOYED: Mon Feb 20 06:56:30 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

I realized my error after reviewing manifest. While the injectors are still normal pods

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get pods --all-namespaces | grep vault
default                 vault-agent-injector-54bdddbb94-gc5wl                      1/1     Running            0                  83s

The vault (main) is actually a statefulset. (Nope… this isn’t the problem either…)

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get statefulset vault
NAME    READY   AGE
vault   0/1     2m36s

moreover, the reason my SS is failing is because it cannot create a PVC (Here we go… this is my goof)

Events:
  Type     Reason        Age                     From                    Message
  ----     ------        ----                    ----                    -------
  Warning  FailedCreate  4m13s (x12 over 4m23s)  statefulset-controller  create Pod vault-0 in StatefulSet vault failed error: failed to create PVC data-vault-0: persistentvolumeclaims "data-vault-0" is forbidden: Internal error occurred: 2 default StorageClasses were found
  Warning  FailedCreate  4m2s (x13 over 4m23s)   statefulset-controller  create Claim data-vault-0 for Pod vault-0 in StatefulSet vault failed error: persistentvolumeclaims "data-vault-0" is forbidden: Internal error occurred: 2 default StorageClasses were found

And indeed, I see the oops on my part - two default StorageClasses

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get storageclass
NAME                            PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs                             cluster.local/nfs-server-provisioner-1658802767   Delete          Immediate              true
      209d
managed-nfs-storage (default)   fuseim.pri/ifs                                    Delete          Immediate              true
      208d
local-path (default)            rancher.io/local-path                             Delete          WaitForFirstConsumer   false
      209d

There are two places in the values that set storage class. When set to ‘null’ they drop to “default” which is in error at the moment. I’ll set these directly to managed-nfs

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ diff currentValues.yaml currentValues.yaml.bak
152c152
<     storageClass: managed-nfs-storage
---
>     storageClass: null
161c161
<     storageClass: managed-nfs-storage
---
>     storageClass: null
340c340
<     enabled: true
---
>     enabled: '-'
365c365
<   enabled: true
---
>   enabled: false

That fixed my issue

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl describe statefulset vault | tail -n3
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  17s   statefulset-controller  create Claim data-vault-0 Pod vault-0 in StatefulSet vault success
  Normal  SuccessfulCreate  17s   statefulset-controller  create Pod vault-0 in StatefulSet vault successful

Quick Note: How to fix doubled default Storage Classes

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get sc
NAME                            PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs                             cluster.local/nfs-server-provisioner-1658802767   Delete          Immediate              true                   209d
managed-nfs-storage (default)   fuseim.pri/ifs                                    Delete          Immediate              true                   208d
local-path (default)            rancher.io/local-path                             Delete          WaitForFirstConsumer   false                  209d
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
storageclass.storage.k8s.io/local-path patched
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get sc
NAME                            PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs                             cluster.local/nfs-server-provisioner-1658802767   Delete          Immediate              true                   209d
managed-nfs-storage (default)   fuseim.pri/ifs                                    Delete          Immediate              true                   208d
local-path                      rancher.io/local-path                             Delete          WaitForFirstConsumer   false                  209d

Now we see both pods

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get pods | grep vault
vault-agent-injector-54bdddbb94-d9znc                    1/1     Running   0                  2m6s
vault-0                                                  0/1     Running   0                  2m6s

Setting up first time

Let’s port-forward to the ‘vault-0’

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl port-forward vault-0 8200:8200
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200

I’ll set 3 shares with 2 minimum

/content/images/2023/02/vault-13.png

I’ll then get a master init root token and 3 keys. We’ll download the JSON (Click “Download Keys”)

/content/images/2023/02/vault-14.png

We will “Continue to Unseal” which will use those keys.

/content/images/2023/02/vault-15.png

Now we can sign in using the root token

/content/images/2023/02/vault-16.png

At this point you should have a running functional, albeit empty, Hashi Vault instance running in your cluster

/content/images/2023/02/vault-17.png

Setting up Auto Unseal

In our currentValues.yaml (or whatever you named your Hashi Helm Chart values file), we see the commented out block for GCP KMS unseal

/content/images/2023/02/vault-19.png

Which will be similar to

--- 329,352 ----
        #   project     = "vault-helm-dev"
        #   region      = "global"
        #   key_ring    = "vault-helm-unseal-kr"
        #   crypto_key  = "vault-helm-unseal-key"
        #}
+       seal "azurekeyvault" {
+          client_id         = "1bb755be-asdf-asdf-asdf-asdfasdf2ad8"
+          tenant_id         = "28c575f6-asdf-asdf-asdf-asdfasdfeb4a"
+          client_secret     = "hasdfasfasfasdfasdfasdfasdfasdfasdfa-"
+          vault_name        = "idjhomelabakv"
+          key_name          = "homevaultunseal"
+       }

        # Example configuration for enabling Prometheus metrics in your config.
        #telemetry {
        #  prometheus_retention_time = "30s",
        #  disable_hostname = true
        #}

I’ll now “upgrade” the chart to use the values

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm upgrade vault -f currentValues.yaml hashicorp/vault
Release "vault" has been upgraded. Happy Helming!
NAME: vault
LAST DEPLOYED: Mon Feb 20 07:23:57 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

In testing this I grew confused. Rotating gave a failure:

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl logs vault-0
2023-02-20T22:39:41.009Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
Error parsing Seal configuration: error fetching Azure Key Vault wrapper key information: keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="KeyNotFound" Message="A key with (name/id) homevaultunseal was not found in this key vault. If you recently deleted this key you may be able to recover it using the correct recovery command. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125182"

It was then I realized that my habit was to always create “Secret”s not “Key”s

/content/images/2023/02/vault-20.png

To be clear; what I needed to do was go to Keys and make a new one.

/content/images/2023/02/vault-21.png

The key needs to be of type RSA and 2048 bits:

/content/images/2023/02/vault-22.png

Now we can pod rotate vault and see it picks up the new key from AKV:

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl delete pod vault-0
pod "vault-0" deleted
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl logs vault-0
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl logs vault-0
==> Vault server configuration:

             Api Address: http://10.42.1.81:8200
                     Cgo: disabled
         Cluster Address: https://vault-0.vault-internal:8201
              Go Version: go1.19.2
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: file
                 Version: Vault v1.12.1, built 2022-10-27T12:32:05Z
             Version Sha: e34f8a14fb7a88af4640b09f3ddbb5646b946d9c

==> Vault server started! Log data will stream in below:

2023-02-20T22:47:17.231Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2023-02-20T22:47:18.062Z [WARN]  core: entering seal migration mode; Vault will not automatically unseal even if using an autoseal: from_barrier_type=shamir to_barrier_type=azurekeyvault
2023-02-20T22:47:18.062Z [INFO]  core: Initializing version history cache for core

I’ll unseal it just this one time

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl exec -it vault-0 -- /bin/sh
/ $ vault operator unseal -migrate
Unseal Key (will be hidden):
Key                           Value
---                           -----
Recovery Seal Type            shamir
Initialized                   true
Sealed                        true
Total Recovery Shares         3
Threshold                     2
Unseal Progress               1/2
Unseal Nonce                  4ad8785a-8236-661f-8225-fa2bd3d74153
Seal Migration in Progress    true
Version                       1.12.1
Build Date                    2022-10-27T12:32:05Z
Storage Type                  file
HA Enabled                    false
/ $ vault operator unseal -migrate
Unseal Key (will be hidden):
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    3
Threshold                2
Version                  1.12.1
Build Date               2022-10-27T12:32:05Z
Storage Type             file
Cluster Name             vault-cluster-2452ca83
Cluster ID               63768fda-1b32-790a-0528-735bf9528902
HA Enabled               false
/ $ vault operator unseal -migrate
Unseal Key (will be hidden):
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    3
Threshold                2
Version                  1.12.1
Build Date               2022-10-27T12:32:05Z
Storage Type             file
Cluster Name             vault-cluster-2452ca83
Cluster ID               63768fda-1b32-790a-0528-735bf9528902
HA Enabled               false

I’ll login just to double check it’s active and unlocked.

/content/images/2023/02/vault-23.png

Let’s watch auto-unseal in action as I rotate the pod and show it comes back unsealed (using AKV)

So far everything we’ve done, we’ve done before.

Username / Pass

Let’s go to Auth Methods in Access and “Enable new method”

/content/images/2023/02/vault-34.png

We’ll select Username and Password and click “Next”

/content/images/2023/02/vault-35.png

Under “Auth Methods”, let’s now click the newly created “userpass”, then “Create use +”

/content/images/2023/02/vault-37.png

To keep it simple, I’ll create a user with my first name.

/content/images/2023/02/vault-38.png

And save

/content/images/2023/02/vault-39.png

MFA

As far as built-in MFA options, we have 4, at the time of this writing, from which we can choose:

/content/images/2023/02/vault-24.png

TOTP is a more generic option that, with some work, you can manage to integrate with an authenticator app.

The other three options are commercial. Since I’m cheap, let’s talk just about price - assume all three could likely do the job:

ame Description Per user Minimum Spend Free Trial Free Tier
Okta Most common, bought up Auth0 US$3/user/month Minium $1500/annual contract Yes No
Ping Plus option for MFA - US$40,000 Annually Yes No
Duo Owned by Cisco now US$3/user/month - Yes Yes

Signing up for Duo

I’ll go to Duo and sign-up for a free trial. However, we will be within the free tier.

/content/images/2023/02/vault-25.png

I’ll put in some details and “Start My Trial”.

/content/images/2023/02/vault-26.png

I’ll verify the email and click “Get started”.

/content/images/2023/02/vault-27.png

Once I setup the mobile app and finished the process, I get the notice that we are complete.

/content/images/2023/02/vault-28.png

I then logged into the Admin panel.

/content/images/2023/02/vault-29.png

Back in Hashi Vault, I’ll set Duo for MFA and click “Next”

/content/images/2023/02/vault-30.png

In Duo Admin panel, I’ll need to get a Secret Key, Integration Key and API Hostname. We can get these by going to “Protect and Application” and clicking “Protect” on an “Auth API” application

/content/images/2023/02/vault-31.png

We’ll copy those values:

/content/images/2023/02/vault-32.png

And I’ll give it a name:

/content/images/2023/02/vault-33.png

Duo User

I then need to create a User. We will make our users match what we created in Vault earlier.

I’ll select “Add User”

/content/images/2023/02/vault-43.png

Give it the same username as before

/content/images/2023/02/vault-44.png

Then give the user, at the very least, an email address

/content/images/2023/02/vault-45.png

I’ll click “Send Enrollment Email”

/content/images/2023/02/vault-46.png

I followed the setup and now have an “Admin” Duo profile and a “User” profile.

(Note: it took a few hours for this user to become live when I tested. It could be because I had a brand new account)

Back in Vault#

I’ll go to Access and select “Multi-factor authentication”, then click “New MFA method +”

/content/images/2023/02/vault-57.png

Then click “Next”

/content/images/2023/02/vault-58.png

I’ll add the Secret key, Integration key and API Hostname.

Lastly, I have to set the Authentication mount to the userpass one we created earlier in the “Enforcement” section.

/content/images/2023/02/vault-40.png

We can now see it is saved.

/content/images/2023/02/vault-41.png

Lastly, I’ll sign out to test.

/content/images/2023/02/vault-42.png

Then when I go to signin, I’ll chose Username this time.

/content/images/2023/02/vault-47.png

After enrollment completed, I was able to see the Duo prompt.

/content/images/2023/02/vault-48.png

I continued to get denied on Duo passcode.

/content/images/2023/02/vault-49.png

I enabled Passcodes in the global policy.

/content/images/2023/02/vault-50.png

One last nuance; the item passed will be the entity name not the User’s ID.

If we go to “Entities”, We can see one was created for our User.

/content/images/2023/02/vault-51.png

I know this because if I click on the entity and go to “Aliases”

/content/images/2023/02/vault-52.png

I can see it’s me:

/content/images/2023/02/vault-53.png

Now we can see the full MFA flow in action.

This works just as well with the CLI.

/content/images/2023/02/vault-55.png

Again, we can see that in action as well:

Ingress

If I wish, I can expose my Vault service externally.

First, I would create an R53 record and apply

$ cat r53-vault.json
{
    "Comment": "CREATE vault fb.s A record ",
    "Changes": [
      {
        "Action": "CREATE",
        "ResourceRecordSet": {
          "Name": "vault.freshbrewed.science",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [
            {
              "Value": "73.242.50.46"
            }
          ]
        }
      }
    ]
  }
$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-vault.json
{
    "ChangeInfo": {
        "Id": "/change/C0527907RMA9WQR6QGNU",
        "Status": "PENDING",
        "SubmittedAt": "2023-02-21T12:28:34.533Z",
        "Comment": "CREATE vault fb.s A record "
    }
}

Then an ingress that points to the Vault service and apply that as well

$ cat ingress-vault.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/name: vault
  name: vault
  namespace: default
spec:
  rules:
  - host: vault.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: vault
            port:
              number: 8200
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - vault.freshbrewed.science
    secretName: vault-tls
$ kubectl apply -f ingress-vault.yml
ingress.networking.k8s.io/vault created

This is now live, but it’s a demo so no real data will be there.

/content/images/2023/02/vault-56.png

We should be able to restrict by CIDR using the denylist and whitelist annotations as documented here. (e.g. nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.1.0/24). However, in my testing, I did not find they work.

I tried both forms of the annotation and including my local network and external, and it still let me through.

$ cat ingress-vault.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/denylist-source-range: "192.168.0.0/16, 73.242.50.46/32"
    ingress.kubernetes.io/denylist-source-range: "192.168.0.0/16, 73.242.50.46/32"
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/name: vault
  name: vault
  namespace: default
spec:
  rules:
  - host: vault.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: vault
            port:
              number: 8200
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - vault.freshbrewed.science
    secretName: vault-tls

Summary

Today we setup Hashi Vault in Kubernetes. To make it more durable, we set up an Azure AKV backend for Auto-unsealing and tested that it worked over a scaling event. We then setup User/Pass authentication with a named users. To make that more secure, we enabled MFA through Duo Security.

Lastly, we exposed the service externally via NGinx with TLS and attempted to restrict with CIDR whitelisting. The idea is that we could use the primary Vault cluster with remote workloads such as another Kubernetes cluster (we covered this using LKE with AKS here).

Hashi Vault MFA DUO K3s

Have something to add? Feedback? Try our new forums

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes