AKS and Stash: k8s native backups

Appscode Stash is slick k8s native suite to backup a variety of kubernetes objects including deployments and more. Let's dig in on AKS to see how we can use stash for a real application.

Creating a cluster to test:

By now this should be familiar.

First, login and create a resource group to contain our work:

$ az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all the subscriptions to which you have access...
[
  {
    "cloudName": "AzureCloud",
    "id": "8afad3d5-6093-4d1e-81e5-4f148a8c4621",
    "isDefault": true,
    "name": "My-Azure-Place",
    "state": "Enabled",
    "tenantId": "5cb73318-f7c7-4633-a7da-0e4b18115d29",
    "user": {
      "name": "Isaac.Johnson@cyberdyn.com",
      "type": "user"
    }
  }
]

$ az group create --location eastus --name idjakstest01rg
{
  "id": "/subscriptions/8afad3d5-6093-4d1e-81e5-4f148a8c4621/resourceGroups/idjakstest01rg",
  "location": "eastus",
  "managedBy": null,
  "name": "idjakstest01rg",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": {
    "Exception": "no",
    "Owner": "missing",
    "StopResources": "yes"
  },
  "type": null
}

Next, create the cluster:

$ az aks create --resource-group idjakstest01rg --name idj-aks-stash --kubernetes-version 1.12.6 --node-count 1 --enable-vmss --enable-cluster-autoscaler --min-count 1 --max-count 3 --generate-ssh-keys
The behavior of this command has been altered by the following extension: aks-preview
{
  "aadProfile": null,
  "addonProfiles": null,
  "agentPoolProfiles": [
    {
      "count": 1,
      "enableAutoScaling": true,
      "maxCount": 3,
      "maxPods": 110,
      "minCount": 1,
      "name": "nodepool1",
      "osDiskSizeGb": 100,
      "osType": "Linux",
      "type": "VirtualMachineScaleSets",
      "vmSize": "Standard_DS2_v2",
      "vnetSubnetId": null
    }
  ],
  "apiServerAuthorizedIpRanges": null,
  "dnsPrefix": "idj-aks-st-idjakstest01rg-fefd67",
  "enableRbac": true,
  "fqdn": "idj-aks-st-idjakstest01rg-fefd67-b5af97af.hcp.eastus.azmk8s.io",
  "id": "/subscriptions/8afad3d5-6093-4d1e-81e5-4f148a8c4621/resourcegroups/idjakstest01rg/providers/Microsoft.ContainerService/managedClusters/idj-aks-stash",
  "kubernetesVersion": "1.12.6",
  "linuxProfile": {
    "adminUsername": "azureuser",
    "ssh": {
      "publicKeys": [
        {
          "keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8kZzEtk7J7Mvv4hJIE1jcQ0q6h41g5hUwPtOUPjNWPIKm4djmy4+C4+Gtsxxh5jUFooAbwl+DubFZogbU1Q5aLOGKSsD/K4XimTyOhr90DO47naCnaSS0Rg0XyZlvQsHKwcXGuGOleCMhB2gQ70QAK4X/N1dvGfqCDdKBbTORKQyz0WHWo7YGA6YAgtvzn1C5W0l7cT0AXgOfFEAGF31nqqTuRVBbBmosq1qhXJlVt+PO32MqmxZv44ZuCP1jWjyTz1rbQ1OLHCxP/+eDIlpOlkYop4XgwiHHMRn/rxHFTKOAxtFOccFw9KEnDM0j0M5FRBj5qU1BCa/6jhnu7LIz"
        }
      ]
    }
  },
  "location": "eastus",
  "name": "idj-aks-stash",
  "networkProfile": {
    "dnsServiceIp": "10.0.0.10",
    "dockerBridgeCidr": "172.17.0.1/16",
    "networkPlugin": "kubenet",
    "networkPolicy": null,
    "podCidr": "10.244.0.0/16",
    "serviceCidr": "10.0.0.0/16"
  },
  "nodeResourceGroup": "MC_idjakstest01rg_idj-aks-stash_eastus",
  "provisioningState": "Succeeded",
  "resourceGroup": "idjakstest01rg",
  "servicePrincipalProfile": {
    "clientId": "05ed2bfd-6340-4bae-85b4-f8e9a0ff5c76",
    "secret": null
  },
  "tags": null,
  "type": "Microsoft.ContainerService/ManagedClusters"
}

and setup up tiller (for helm)

$ az aks get-credentials --name idj-aks-stash --resource-group idjakstest01rg
Merged "idj-aks-stash" as current context in /Users/isaac.johnson/.kube/config

$ cat rbac-config.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

$ kubectl create -f rbac-config.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created

$ helm init --service-account tiller --upgrade --history-max 200
$HELM_HOME has been configured at /Users/isaac.johnson/.helm.

Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!

Next, let's add Appscode helm repos and install Stash:

$ helm repo add appscode https://charts.appscode.com/stable/
"appscode" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "ibm-repo" chart repository
...Successfully got an update from the "akomljen-charts" chart repository
...Successfully got an update from the "appscode" chart repository
...Successfully got an update from the "incubatorold" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 


$ helm install appscode/stash --name stash-operator --namespace kube-system
NAME:   stash-operator
LAST DEPLOYED: Tue Apr 30 21:22:47 2019
NAMESPACE: kube-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME            AGE
stash-operator  2s

==> v1/ClusterRoleBinding
NAME                                     AGE
stash-operator                           2s
stash-operator-apiserver-auth-delegator  2s

==> v1/Deployment
NAME            READY  UP-TO-DATE  AVAILABLE  AGE
stash-operator  0/1    1           0          2s

==> v1/Pod(related)
NAME                             READY  STATUS             RESTARTS  AGE
stash-operator-5f7bb8f4d5-rkm9j  0/2    ContainerCreating  0         2s

==> v1/RoleBinding
NAME                                                             AGE
stash-operator-apiserver-extension-server-authentication-reader  2s

==> v1/Secret
NAME                           TYPE    DATA  AGE
stash-operator-apiserver-cert  Opaque  2     2s

==> v1/Service
NAME            TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)            AGE
stash-operator  ClusterIP  10.0.76.9   <none>       443/TCP,56789/TCP  2s

==> v1/ServiceAccount
NAME            SECRETS  AGE
stash-operator  1        2s

==> v1beta1/APIService
NAME                                      AGE
v1alpha1.admission.stash.appscode.com     2s
v1alpha1.repositories.stash.appscode.com  2s


NOTES:
To verify that Stash has started, run:

  kubectl --namespace=kube-system get deployments -l "release=stash-operator, app=stash"

Lastly, verify it started:

$ kubectl --namespace=kube-system get deployments -l "release=stash-operator, app=stash"
NAME             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
stash-operator   1         1         1            1           102s

Using Stash

Let's install our favourite demo app, Sonarqube:

$ helm install stable/sonarqube
NAME:   dusty-butterfly
LAST DEPLOYED: Tue Apr 30 22:07:52 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                                       DATA  AGE
dusty-butterfly-sonarqube-config           0     1s
dusty-butterfly-sonarqube-copy-plugins     1     1s
dusty-butterfly-sonarqube-install-plugins  1     1s
dusty-butterfly-sonarqube-tests            1     1s

==> v1/PersistentVolumeClaim
NAME                        STATUS   VOLUME   CAPACITY  ACCESS MODES  STORAGECLASS  AGE
dusty-butterfly-postgresql  Pending  default  1s

==> v1/Pod(related)
NAME                                        READY  STATUS             RESTARTS  AGE
dusty-butterfly-postgresql-865f459b5-jhn82  0/1    Pending            0         0s
dusty-butterfly-sonarqube-756c789969-fh2bg  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                        TYPE    DATA  AGE
dusty-butterfly-postgresql  Opaque  1     1s

==> v1/Service
NAME                        TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)         AGE
dusty-butterfly-postgresql  ClusterIP     10.0.113.110  <none>       5432/TCP        0s
dusty-butterfly-sonarqube   LoadBalancer  10.0.48.93    <pending>    9000:31335/TCP  0s

==> v1beta1/Deployment
NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
dusty-butterfly-postgresql  0/1    1           0          0s
dusty-butterfly-sonarqube   0/1    1           0          0s


NOTES:
1. Get the application URL by running these commands:
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get svc -w dusty-butterfly-sonarqube'
  export SERVICE_IP=$(kubectl get svc --namespace default dusty-butterfly-sonarqube -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
  echo http://$SERVICE_IP:9000

Pro-tip: say you cleaned up a bunch of k8s resource groups and service principles and now everything is exploding - it could that AKS is trying to use an old service principle (not that it happened to me, of course).  You can create a new service principal in Azure then apply it all the nodes in the cluster pretty easily:

$ az aks update-credentials --resource-group myResourceGroup idjakstest01rg --name idj-aks-stash --reset-service-principal --client-secret UDJuiejdJDUejDJJDuieUDJDJiJDJDJ+didjdjDi= --service-principal f1815adc-eeee-dddd-aaaa-ffffffffff

The SonarQube instance gets a public IP so we login and immediately change the admin password.

Then let’s follow some instructions and scan something:

fortran-cli isaac.johnson$ sonar-scanner   -Dsonar.projectKey=SAMPLE   -Dsonar.sources=F90   -Dsonar.host.url=http://168.61.36.201:9000   -Dsonar.login=4cdf844fe382f8d9003c28c92b36c4546b33d079
INFO: Scanner configuration file: /Users/isaac.johnson/Downloads/sonar-scanner-3.3.0.1492-macosx/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Mac OS X 10.14.4 x86_64
INFO: User cache: /Users/isaac.johnson/.sonar/cache
INFO: SonarQube server 7.7.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=176ms
INFO: Server id: 2090D6C0-AWpxX2_MBdGJAq0bH5y-
INFO: User cache: /Users/isaac.johnson/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=85ms
INFO: Load/download plugins (done) | time=91ms
INFO: Process project properties
INFO: Project key: SAMPLE
INFO: Base dir: /Users/isaac.johnson/Workspaces/fortran-cli
INFO: Working dir: /Users/isaac.johnson/Workspaces/fortran-cli/.scannerwork
INFO: Load project settings for component key: 'SAMPLE'
INFO: Load project settings for component key: 'SAMPLE' (done) | time=98ms
INFO: Load project repositories
INFO: Load project repositories (done) | time=98ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=101ms
INFO: Load active rules
INFO: Load active rules (done) | time=913ms
WARN: SCM provider autodetection failed. Please use "sonar.scm.provider" to define SCM of your project, or disable the SCM Sensor in the project settings.
INFO: Indexing files...
INFO: Project configuration:
INFO: 4 files indexed
INFO: Quality profile for f90: Sonar way
INFO: ------------- Run sensors on module SAMPLE
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=71ms
INFO: Sensor fr.cnes.sonar.plugins.icode.check.ICodeSensor [icode]
WARN: Results file result.res has not been found and wont be processed.
INFO: Sensor fr.cnes.sonar.plugins.icode.check.ICodeSensor [icode] (done) | time=1ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=1ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 0 files
INFO: CPD calculation finished
INFO: Analysis report generated in 73ms, dir size=12 KB
INFO: Analysis report compressed in 15ms, zip size=3 KB
INFO: Analysis report uploaded in 106ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://168.61.36.201:9000/dashboard?id=SAMPLE
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://168.61.36.201:9000/api/ce/task?id=AWpxavS8-5ofCQKUskvk
INFO: Analysis total time: 1.834 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2.822s
INFO: Final Memory: 10M/198M
INFO: ------------------------------------------------------------------------

We can now see the project has successfully been indexed and is visible:

Applying Stash

Now let’s use stash to start to backup this application (we’ll be expanding on their offline guide).

We need some details first:

[1] the the pod namespace and name - verify the label

$ kubectl get pod -n default -l app=dusty-butterfly-postgresql
NAME                                         READY   STATUS    RESTARTS   AGE
dusty-butterfly-postgresql-865f459b5-jhn82   1/1     Running   0          4d9h

[2] let’s double check the containers database directory

$ kubectl exec -it dusty-butterfly-postgresql-865f459b5-jhn82 -- /bin/sh
# su postgres
$ psql
$ psql sonarDB
psql (9.6.2)
Type "help" for help.

sonarDB=# SHOW data_directory;
         data_directory          
---------------------------------
 /var/lib/postgresql/data/pgdata
(1 row)

$ kubectl exec  -n default dusty-butterfly-postgresql-865f459b5-jhn82 -- ls -R /var/lib/postgresql/data/pgdata | head -n10
/var/lib/postgresql/data/pgdata:
base
global
pg_clog
pg_commit_ts
pg_dynshmem
pg_hba.conf
pg_ident.conf
pg_logical
pg_multixact

Next step is to create a secret that stash can use to encrypt the backup data.  Note, you need to put all these in the same namespace (default in my case):

$ echo -n 'MyBackupPassword' > RESTIC_PASSWORD
$ kubectl create secret generic -n default local-secret --from-file=./RESTIC_PASSWORD
secret/local-secret created

And verify it's there

$ kubectl get secret -n default local-secret -o yaml
apiVersion: v1
data:
  RESTIC_PASSWORD: TXlCYWNrdXBQYXNzd29yZA==
kind: Secret
metadata:
  creationTimestamp: "2019-05-05T12:52:25Z"
  name: local-secret
  namespace: default
  resourceVersion: "496452"
  selfLink: /api/v1/namespaces/default/secrets/local-secret
  uid: a1b1839b-6f34-11e9-a14b-7af187f5df5f
type: Opaque

One more thing we are going to need, if you haven’t set one up already, is an nfs-server to receive the backups.  This could be exposed in many ways, but for easy, let’s just spin a simple containerized instance.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nfs-server
spec:
  replicas: 1
  selector:
    matchLabels:
      role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      containers:
      - name: nfs-server
        image: gcr.io/google_containers/volume-nfs:0.8
        ports:
        - name: nfs
          containerPort: 2049
        - name: mountd
          containerPort: 20048
        - name: rpcbind
          containerPort: 111
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /exports
          name: exports
      volumes:
      - name: exports
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: nfs-server
spec:
  ports:
  - name: nfs
    port: 2049
  - name: mountd
    port: 20048
  - name: rpcbind
    port: 111
  selector:
    role: nfs-server

While we should be able to refer to this with nfs-server.svc.cluster.local, I found some issues as have others (mount.nfs: Failed to resolve server nfs-server.default.svc.cluster.local: Name or service not known)  so i just pulled up the IP:

$ kubectl describe pod nfs-server-676897cb56-d9xv7 | grep IP
IP:                 10.244.0.65

So now we know all the specifics we need for the restic file definition.  The restic spec is a yaml object that stash uses to describe what it plans to backup in offline mode:

Note the areas we changed (file in markdown below image):

apiVersion: stash.appscode.com/v1alpha1
kind: Restic
metadata:
  name: offline-restic
  namespace: default
spec:
  selector:
    matchLabels:
      app: dusty-butterfly-postgresql
  type: offline
  fileGroups:
  - path: /var/lib/postgresql/data/pgdata
    retentionPolicyName: 'keep-last-5'
  backend:
    local:
      mountPath: /safe/data
      nfs:
        server: "10.244.0.65"
        path: "/"
    storageSecretName: local-secret
  schedule: '@every 5m'
  volumeMounts:
  - mountPath: /var/lib/postgresql/data/pgdata
    name: data
  retentionPolicies:
  - name: 'keep-last-5'
    keepLast: 5
    prune: true

Now apply it:

$ kubectl apply -f ./restic_offline_sonar.yaml 
restic.stash.appscode.com/offline-restic created

The backups should start soon.  Let’s check that the stash “init-container” has been injected:

kubectl get deployment -n default dusty-butterfly-postgresql -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
    old-replica: "1"
    restic.appscode.com/last-applied-configuration: |
      {"kind":"Restic","apiVersion":"stash.appscode.com/v1alpha1","metadata":{"name":"offline-restic","namespace":"default","selfLink":"/apis/stash.appscode.com/v1alpha1/namespaces/default/restics/offline-restic","uid":"a478b03e-6f35-11e9-a14b-7af187f5df5f","resourceVersion":"513231","generation":4,"creationTimestamp":"2019-05-05T12:59:39Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stash.appscode.com/v1alpha1\",\"kind\":\"Restic\",\"metadata\":{\"annotations\":{},\"name\":\"offline-restic\",\"namespace\":\"default\"},\"spec\":{\"backend\":{\"local\":{\"mountPath\":\"/safe/data\",\"nfs\":{\"path\":\"/\",\"server\":\"nfs-server.default.svc.cluster.local\"}},\"storageSecretName\":\"local-secret\"},\"fileGroups\":[{\"path\":\"/var/lib/postgresql/data/pgdata\",\"retentionPolicyName\":\"keep-last-5\"}],\"retentionPolicies\":[{\"keepLast\":5,\"name\":\"keep-last-5\",\"prune\":true}],\"schedule\":\"@every 5m\",\"selector\":{\"matchLabels\":{\"app\":\"dusty-butterfly-postgresql\"}},\"type\":\"offline\",\"volumeMounts\":[{\"mountPath\":\"/var/lib/postgresql/data/pgdata\",\"name\":\"data\"}]}}\n"}},"spec":{"selector":{"matchLabels":{"app":"dusty-butterfly-postgresql"}},"fileGroups":[{"path":"/var/lib/postgresql/data/pgdata","retentionPolicyName":"keep-last-5"}],"backend":{"storageSecretName":"local-secret","local":{"nfs":{"server":"nfs-server.default.svc.cluster.local","path":"/"},"mountPath":"/safe/data"}},"schedule":"@every 5m","volumeMounts":[{"name":"data","mountPath":"/var/lib/postgresql/data/pgdata"}],"resources":{},"retentionPolicies":[{"name":"keep-last-5","keepLast":5,"prune":true}],"type":"offline"}}
    restic.appscode.com/tag: 0.8.3
  creationTimestamp: "2019-05-01T03:07:53Z"
  generation: 64
  labels:
    app: dusty-butterfly-postgresql
    chart: postgresql-0.8.3
    heritage: Tiller
    release: dusty-butterfly
  name: dusty-butterfly-postgresql
  namespace: default
  resourceVersion: "513248"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/dusty-butterfly-postgresql
  uid: 4f6afc14-6bbe-11e9-a14b-7af187f5df5f
spec:
  progressDeadlineSeconds: 2147483647
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: dusty-butterfly-postgresql
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        restic.appscode.com/resource-hash: "9340540511619766860"
      creationTimestamp: null
      labels:
        app: dusty-butterfly-postgresql
    spec:
      containers:
      - env:
        - name: POSTGRES_USER
          value: sonarUser
        - name: PGUSER
          value: sonarUser
        - name: POSTGRES_DB
          value: sonarDB
        - name: POSTGRES_INITDB_ARGS
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgres-password
              name: dusty-butterfly-postgresql
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        image: postgres:9.6.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - exec pg_isready --host $POD_IP
          failureThreshold: 6
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: dusty-butterfly-postgresql
        ports:
        - containerPort: 5432
          name: postgresql
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - exec pg_isready --host $POD_IP
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/postgresql/data/pgdata
          name: data
          subPath: postgresql-db
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - backup
        - --restic-name=offline-restic
        - --workload-kind=Deployment
        - --workload-name=dusty-butterfly-postgresql
        - --docker-registry=appscode
        - --image-tag=0.8.3
        - --pushgateway-url=http://stash-operator.kube-system.svc:56789
        - --enable-status-subresource=true
        - --use-kubeapiserver-fqdn-for-aks=true
        - --enable-analytics=true
        - --logtostderr=true
        - --alsologtostderr=false
        - --v=3
        - --stderrthreshold=0
        - --enable-rbac=true
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: APPSCODE_ANALYTICS_CLIENT_ID
          value: de2453edf3a39c712f20b4697845f2bb
        image: appscode/stash:0.8.3
        imagePullPolicy: IfNotPresent
        name: stash
        resources: {}
        securityContext:
          procMount: Default
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: stash-scratchdir
        - mountPath: /etc/stash
          name: stash-podinfo
        - mountPath: /var/lib/postgresql/data/pgdata
          name: data
          readOnly: true
        - mountPath: /safe/data
          name: stash-local
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: dusty-butterfly-postgresql
      - emptyDir: {}
        name: stash-scratchdir
      - downwardAPI:
          defaultMode: 420
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels
            path: labels
        name: stash-podinfo
      - name: stash-local
        nfs:
          path: /
          server: nfs-server.default.svc.cluster.local
status:
  conditions:
  - lastTransitionTime: "2019-05-01T03:07:53Z"
    lastUpdateTime: "2019-05-01T03:07:53Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 64
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

Next, let’s see if we’ve been creating backups:

$ kubectl get repository
NAME                                    BACKUP-COUNT   LAST-SUCCESSFUL-BACKUP   AGE
deployment.dusty-butterfly-postgresql   16             4m                       1h

Restoring

Backups aren't that useful if we can't use them. Let's explore how we can get our data back.

First, we need to create a persistent volume claim to dump a backup into:

Create restore-pvc.yaml :
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: stash-recovered
  namespace: default
  labels:
    app: sonardb-restore
spec:
  storageClassName: default
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi
$ kubectl apply -f restore-pvc.yaml 
persistentvolumeclaim/stash-recovered created

Quick-tip: you can use “managed-premium” for storage class instead of default if you wish for faster ssd drives in AKS (for a cost, of course)

Then we can verify it:

$ kubectl get pvc -n default -l app=sonardb-restore
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
stash-recovered   Bound    pvc-e95c72e3-6f4f-11e9-a14b-7af187f5df5f   1Gi        RWO            default        17s

Next, we need to recreate a recovery Custom Resource Definition (CRD).  This is where the restore action actually takes place:

$ cat ./recovery-it.yaml 
apiVersion: stash.appscode.com/v1alpha1
kind: Recovery
metadata:
  name: local-recovery
  namespace: default
spec:
  repository:
    name: deployment.dusty-butterfly-postgresql
    namespace: default
  paths:
  - /var/lib/postgresql/data/pgdata
  recoveredVolumes:
  - mountPath: /var/lib/postgresql/data/pgdata
    persistentVolumeClaim:
      claimName: stash-recovered

$ kubectl apply -f ./recovery-it.yaml 
recovery.stash.appscode.com/local-recovery created

Check on the recovery:

$ kubectl get recovery -n default local-recovery
NAME             REPOSITORY-NAMESPACE   REPOSITORY-NAME                         SNAPSHOT   PHASE     AGE
local-recovery   default                deployment.dusty-butterfly-postgresql              Running   16s
$ kubectl get recovery -n default local-recovery
NAME             REPOSITORY-NAMESPACE   REPOSITORY-NAME                         SNAPSHOT   PHASE       AGE
local-recovery   default                deployment.dusty-butterfly-postgresql              Succeeded   2m

Now let’s verify the files:

$ cat ./verify-restore.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonar-restore
  name: sonar-restore-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sonar-restore-test
  template:
    metadata:
      labels:
        app: sonar-restore-test
      name: busybox
    spec:
      containers:
      - args:
        - sleep
        - "3600"
        image: busybox
        imagePullPolicy: IfNotPresent
        name: busybox
        volumeMounts:
        - mountPath: /var/lib/postgresql/data/pgdata
          name: source-data
      restartPolicy: Always
      volumes:
      - name: source-data
        persistentVolumeClaim:
          claimName: stash-recovered
$ kubectl apply -f ./verify-restore.yaml 
deployment.apps/sonar-restore-test created

Now just get the pod and check that the files are mounted:

$ kubectl get pods -n default -l app=sonar-restore-test
NAME                                  READY   STATUS    RESTARTS   AGE
sonar-restore-test-744498d8b8-rsfz6   1/1     Running   0          3m26s

$ kubectl exec  -n default sonar-restore-test-744498d8b8-rsfz6 -- ls -R /var/lib/postgresql/data/pgdata | head -n10
/var/lib/postgresql/data/pgdata:
lost+found
postgresql-db

/var/lib/postgresql/data/pgdata/lost+found:

/var/lib/postgresql/data/pgdata/postgresql-db:
PG_VERSION
base
global

Clearly at this point, we could delete the sonar deployment and restic backup process (kubectl delete deployment dusty-butterfly-postgresql and kubectl delete restic offline-restic) then start the process of restoring with a full redeployment. However, a more common situation would be to load a new database with the data and point a sonar instance to it to verify it's what you were expecting.

In the case of postgres, we could just restore manually by copying the files locally and then to the pod (or a new Azure Database for Postgres database):

locally:

$ mkdir /tmp/backups
$ kubectl cp sonar-restore-test-744498d8b8-rsfz6:/var/lib/postgresql/data/pgdata /tmp/backups
$ du -csh /tmp/backups/
 48M	/tmp/backups/
 48M	total
$ ls /tmp/backups/
lost+found	postgresql-db

Next steps:

Stash can do more than offline backups. You can backup an entire deployment at once as well (see guide: https://appscode.com/products/stash/0.8.3/guides/backup/).  

Summary:

In this guide we dug into Appscode Stash and used it to create regular scheduled backups of a live containerized database.  We then created a restore process and verified the contents.  Lastly, we showed how we could take those files and move them around (such as copy locally).