Published: Feb 2, 2023 by Isaac Johnson
Often I’m asked for how best to apply secrets to CICD Workflows, most often in Github Actions and/or Azure DevOps. Today we’ll dig into using Secrets in Azure Pipelines (AzDO) and Github Actions. We’ll demonstrate GCP Secret Manager and Azure Key Vault; both reading and writing secrets as well as how to sync between systems.
Secrets Overview
Generally, when creating a repeatable Continuous Integration (CI) workflow, there exists a need to have secret information. This can include database passwords, the keys to access cloud resources, or perhaps a revision control system PAT in order to update a GIT Tag. In all these cases, we wish to keep that secret, well, a secret - that is, not publicly readable in source control or typed-out plain text on a webpage everyone can read.
Regardless of system, there are three common ways people address secrets in CI systems.
1. Bespoke Variable
Going back to Jenkins and other old tools, the idea of just adding a “Password” field or a “Pipeline Variable” is a simple, but not really scalable approach.
It has the advantage of being the fastest, but with the consequence of lacking re-use or history. For instance, if I have the “AWS” key on a build pipeline for Team A as a variable, I need to copy and paste it to every other team. And when it rotates, I need to do all that by hand again - which can lead to mistakes. Additionally, without history, there is no way to really know who changed it, when or even why.
That doesn’t mean you should never use them. Rather, use them for ephemeral mutable data such as the passed in “new” password for a manual pipeline (something we would not want to save), or the “default” password that will be reset on login.
2. Vendored / External Secrets
Here is where we will explore today - the use of a secrets provider. In our case, we’ll explore Azure Key Vault (AKV) and GCP Secret Manager. Both are extremely low cost and scalable. We can do more things with AKV, but with that has some cost implications.
For instance, in Azure DevOps, we can fetch AKV secrets via Group Variables
AKV
Azure Key Vault will exist within our Subscription which itself is within our Azure Tenant. In most organizations, access is controlled via AAD or a federated Identity Provider (IdP) like Okta.
We have IAM access, standard with any Azure Resource controlled at the Subscription, Resource Group and AKV instance level. Then we have “Access Policies”, which is unique to AKV that ties Identities (Service Principals or Users/Groups) to resources (Certificates, Secrets, HSM) and actions (Get, List, Set, etc).
For secrets, we pay US$0.03 per 10k operations; essentially free if used in simple CICD pipelines.
GCP Secret Manager
GCP Secret Manager focuses only on secrets. Unlike Azure, and more similar to AWS - Certificates are handled by “Certificate Manager” (AWS uses ACM). If you need an ASM, in GCP you use the “Cloud HSM” which is a feature of Cloud Key Management Service (Cloud KMS).
GCP Secret Manager gives you 6 secrets for free a month, but then is US$0.06 per secret beyond that. Like AKV, you get 10k operations a month for free and US$0.03/10k beyond that.
Github GCP Secret Manager
Let’s dig into using Google Clouds Secret Manager. The first time through, you’ll need to enable the API - see docs
Creating the SA
Before we get started, we’ll need a Service Account that can access Secrets.
We’ve done this in the Cloud Console before, so this time, let’s use the gcloud CLI.
Login and set the project
$ gcloud auth login
$ gcloud config set project myanthosproject2
Updated property [core/project].
Create the SA and grant ourselves access to it
$ gcloud iam service-accounts create ghsecretaccessor --description "GCP Secret Accessor for Github Actions" --display-name ghsecretaccessor
Created service account [ghsecretaccessor].
$ gcloud iam service-accounts add-iam-policy-binding ghsecretaccessor@myanthosproject2.iam.gserviceacco
unt.com --member="user:isaac.johnson@gmail.com" --role="roles/iam.serviceAccountUser"
Updated IAM policy for serviceAccount [ghsecretaccessor@myanthosproject2.iam.gserviceaccount.com].
bindings:
- members:
- user:isaac.johnson@gmail.com
role: roles/iam.serviceAccountUser
etag: BwXzlt2QRXo=
version: 1
Now we add the SecretManager Admin role as well as secretmanager.secretVersionManager role
$ gcloud projects add-iam-policy-binding myanthosproject2 --member="serviceAccount:ghsecretaccessor@mya
nthosproject2.iam.gserviceaccount.com" --role="roles/secretmanager.admin"
Updated IAM policy for project [myanthosproject2].
bindings:
- members:
... snip ...
$ gcloud projects add-iam-policy-binding myanthosproject2 --member="serviceAccount:ghsecretaccessor@myanthosproject2.iam.gserviceaccount.com" --role="roles/secretmanager.secretVersionManager"
Updated IAM policy for project [myanthosproject2].
bindings:
- members:
... snip ...
Lastly, create the Service Account JSON
$ gcloud iam service-accounts keys create mysecretaccessorsa.json --iam-account=ghsecretaccessor@myanthosproject2.iam.gserviceaccount.com
created key [4asdfasdfasdfasdfasdfasdfadfasdfasdf9] of type [json] as [mysecretaccessorsa.json] for [ghsecretaccessor@myanthosproject2.iam.gserviceaccount.com]
I’ll need a simple secret we can use as well
$ cat > mysecretvaluefile << EOF
> hello from GCP Secret Manager
> EOF
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ cat mysecretvaluefile
hello from GCP Secret Manager
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ gcloud beta secrets create mytestsecret2 --data-file=./mysecretvaluefile
Created version [1] of the secret [mytestsecret2].
In a Github Repository I’ll use, I add a new Variable for Actions
I’ll create a “GCP_CREDENTIALS” secret of which the value is the contents of that JSON file
Github Workflow
I’ll clone my repo and create a workflows dir
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ mkdir -p .github/workflows
(not tested)
on:
push:
branches:
- main
env:
SECRET_NAME: <SECRET_NAME>
jobs:
read_secret:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- id: 'auth'
uses: 'google-github-actions/auth@v1'
with:
credentials_json: '$'
- name: 'Set up Cloud SDK'
uses: 'google-github-actions/setup-gcloud@v1'
- name: Read secret from GCP Secret Manager
run: |
secret=$(gcloud beta secrets versions access latest --secret=$SECRET_NAME)
echo "::set-env name=SECRET_VAR::$secret"
- name: Use secret
run: |
echo "Secret value is: $SECRET_VAR"
Which for me, would look like
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ cat .github/workflows/gcptest.yml
on:
push:
branches:
- main
env:
SECRET_NAME: mytestsecret2
jobs:
read_secret:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- id: 'auth'
uses: 'google-github-actions/auth@v1'
with:
credentials_json: '$'
- name: 'Set up Cloud SDK'
uses: 'google-github-actions/setup-gcloud@v1'
- name: Add GCloud Beta
run: |
yes | gcloud components install beta
- name: Read secret from GCP Secret Manager
run: |
secret=$(gcloud beta secrets versions access latest --secret=$SECRET_NAME)
echo "::set-env name=SECRET_VAR::$secret"
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
- name: Use secret
run: |
echo "Secret value is: $SECRET_VAR"
I can now add and push it
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ git add .github/workflows/
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ git commit -m 'gcp workflow'
[main 9b797ed] gcp workflow
1 file changed, 31 insertions(+)
create mode 100644 .github/workflows/gcptest.yml
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ git push
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 729 bytes | 729.00 KiB/s, done.
Total 5 (delta 0), reused 0 (delta 0)
To https://github.com/idjohnson/secretAccessor.git
1b541ea..9b797ed main -> main
we can see it was successful
and we can see the value
I’ll now add a step to set a variable
- name: Add GCloud Beta
run: |
yes | gcloud components install beta
- name: Set secret from GCP Secret Manager
run: |
date > ./mycurrentdate
gcloud beta secrets create mytestsecret2 --data-file=./mycurrentdate || gcloud secrets versions add mytestsecret2 --data-file=./mycurrentdate
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
- name: Read secret from GCP Secret Manager
run: |
secret=$(gcloud beta secrets versions access latest --secret=$SECRET_NAME)
echo "::set-env name=SECRET_VAR::$secret"
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
which we can see worked
I can check the Secret Manager in GCP to see the value
as well as in the Github Actions log
Azure Key Vault (AKV)
Like with GCP, I’ll need an Service Account to access Azure. However, in Azure, we call them Service Principals.
I’ll create a Service Principal with contributor rights on the AKV Resource Group
$ az ad sp create-for-rbac --name "mysecretsreadersp" --role contributor --scopes /subscriptions/d955c0ba-1111-1111-1111-8fed74cbb22d/resourceGroups/idjakvrg
Found an existing application instance: (id) 7095e3ea-1111-1111-1111-1111111111. We will patch it.
Creating 'contributor' role assignment under scope '/subscriptions/d955c0ba-abcd-abcd-abcd-abcdabcd/resourceGroups/idjakvrg'
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
"appId": "e6c86cb1-1111-1111-1111-11111111",
"displayName": "mysecretsreadersp",
"password": "zasdfasfdasdfasdfsafsadfsadfI",
"tenant": "28c575f6-1111-1111-1111-2222222222"
}
Later i realized I should have also added “Reader” on the subscription as a whole and did that:
$ az ad sp list --display-name "mysecretsreadersp" | jq '.[] | .id'
"2e7b6dd7-9828-4177-ab44-3e074a0c690f"
$ az role assignment create --assignee "2e7b6dd7-9828-4177-ab44-3e074a0c690f" --role "Reader" --subscription "d955c0ba-abcd-abcd-abcd-abcdabcd"
{
"canDelegate": null,
"condition": null,
"conditionVersion": null,
"description": null,
"id": "/subscriptions/d955c0ba-abcd-abcd-abcd-abcdabcd/providers/Microsoft.Authorization/roleAssignments/3a91321e-0660-4925-aa1e-8be6200bd91f",
"name": "3a91321e-0660-4925-aa1e-8be6200bd91f",
"principalId": "2e7b6dd7-9828-4177-ab44-3e074a0c690f",
"principalType": "ServicePrincipal",
"roleDefinitionId": "/subscriptions/d955c0ba-abcd-abcd-abcd-abcdabcd/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
"scope": "/subscriptions/d955c0ba-abcd-abcd-abcd-abcdabcd",
"type": "Microsoft.Authorization/roleAssignments"
}
Now that we have the AppId, password and tenant, we will want to partner that with the subscription to create the auth
- uses: Azure/login@v1
with:
creds: '{"clientId":"$","clientSecret":"$","subscriptionId":"$","tenantId":"$"}'
I put in the secrets. Noting the differences below
We can create a quick test secret
$ az keyvault secret set --vault-name idjakv --name idjtestsecret --value "this is a test"
{
"attributes": {
"created": "2023-02-01T03:11:15+00:00",
"enabled": true,
"expires": null,
"notBefore": null,
"recoveryLevel": "Purgeable",
"updated": "2023-02-01T03:11:15+00:00"
},
"contentType": null,
"id": "https://idjakv.vault.azure.net/secrets/idjtestsecret/641ebe3f701a4fc8a194efb8f5a80ed4",
"kid": null,
"managed": null,
"name": "idjtestsecret",
"tags": {
"file-encoding": "utf-8"
},
"value": "this is a test"
}
Then add the steps to read from the AKV in our Github action workflow:
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ cat .github/workflows/azuretest.yml
name: Read secret from Azure Key Vault
on:
push:
branches:
- main
env:
VAULT_NAME: idjakv
SECRET_NAME: idjtestsecret
jobs:
read_secret:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- uses: Azure/login@v1
with:
creds: '{"clientId":"$","clientSecret":"$","subscriptionId":"$","tenantId":"$"}'
- name: Read secret from Azure Key Vault
run: |
secret=$(az keyvault secret show --vault-name $VAULT_NAME --name $SECRET_NAME --query value -o tsv)
echo "::set-env name=SECRET_VAR::$secret"
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
- name: Use secret
run: echo "Secret value is: $SECRET_VAR"
When ready, we add, commit, then push the committed workflow to Github
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ git add .github/workflows/
builder@DESKTOP-72D2D9T:~/Workspaces/secretAccessor$ git commit -m "Test AKV"
[main dc133df] Test AKV
1 file changed, 32 insertions(+)
create mode 100644 .github/workflows/azuretest.yml
I’ll need to add a Secrets Access policy for the Object ID of the SP as well
$ az ad sp list --display-name "mysecretsreadersp" | jq '.[] | .id'
"2e7b6dd7-9828-4177-ab44-3e074a0c690f"
$ az keyvault set-policy --name idjakv --object-id "2e7b6dd7-9828-4177-ab44-3e074a0c690f"
--secret-permissions all
{
"id": "/subscriptions/d955c0ba-abcd-abcd-abcd-abcdabcd/resourceGroups/idjakvrg/providers/Microsoft.KeyVault/vaults/idjakv",
"location": "centralus",
"name": "idjakv",
"properties": {
"accessPolicies": [
{
"applicationId": null,
"objectId": "1f5d835c-b129-41e6-b2fe-5858a5f4e41a",
...
Which I can now see reflected in AKV
Now when testing the workflow, we see we can retrieve the secret
We’ve seen we can read a secret. Let’s move on to updating secrets.
We will now add a “secret set” in our workflow
- name: Set secret in Azure Key Vault
run: |
date > ./mycurrentdate
az keyvault secret set --vault-name $VAULT_NAME --name $SECRET_NAME --file ./mycurrentdate
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
which makes our workflow
name: Secrets from Azure Key Vault
on:
push:
branches:
- main
env:
VAULT_NAME: idjakv
SECRET_NAME: idjtestsecret
jobs:
read_secret:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- uses: Azure/login@v1
with:
creds: '{"clientId":"$","clientSecret":"$","subscriptionId":"$","tenantId":"$"}'
- name: Set secret in Azure Key Vault
run: |
date > ./mycurrentdate
az keyvault secret set --vault-name $VAULT_NAME --name $SECRET_NAME --file ./mycurrentdate
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
- name: Read secret from Azure Key Vault
run: |
secret=$(az keyvault secret show --vault-name $VAULT_NAME --name $SECRET_NAME --query value -o tsv)
echo "::set-env name=SECRET_VAR::$secret"
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
- name: Use secret
run: |
echo "Secret value is: $SECRET_VAR"
We can see the flow in action
Azure DevOps
I’ll head to the Pipelines area of Azure DevOps and click “New pipeline”
I’ll pick GitHub YAML
Then pick the repo
I’ll just create a Starter Pipeline
We can either access the AKV via an AKV-backed Group Var or an AKV action.
AKV Step
We can start with the AKV step
I’ll authorize
We’ll pick the vault
The handy part is we can get all secrets or a posix filter
We’ll set it in the Azure Pipelines YAML
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- task: AzureKeyVault@2
inputs:
azureSubscription: 'Pay-As-You-Go(d955c0ba-abcd-abcd-abcd-abcdabcd)'
KeyVaultName: 'idjakv'
SecretsFilter: '*'
RunAsPreJob: true
- script: |
echo Show the idjtestsecret Secret
echo $(idjtestsecret)
displayName: 'Show the secret'
We can save and run
We need to approve its access first
Click Permit
We can see that we are missing secrets list permission
As before, I’ll add secrets permission
$ az keyvault set-policy --name idjakv --object-id "6e7329f9-3221-474f-b5ee-0db3ac3bc993" --se
cret-permissions all
{
"id": "/subscriptions/d955c0ba-abcd-abcd-abcd-abcdabcd/resourceGroups/idjakvrg/providers/Microsoft.KeyVault/vaults/idjakv",
"location": "centralus",
"name": "idjakv",
"properties": {
"accessPolicies": [
{
"applicationId": null,
"objectId": "1f5d835c-b129-41e6-b2fe-5858a5f4e41a",
"permissions": {
"certificates": [
"get",
...
Which unblocks it. We can see the results in the log
We can then create/set the secret as well in our Pipeline YAML
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- task: AzureCLI@2
inputs:
azureSubscription: 'Pay-As-You-Go(d955c0ba-abcd-abcd-abcd-abcdabcd)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az keyvault secret set --vault-name idjakv --name idjtestsecret --value "this is a test 2"'
- task: AzureKeyVault@2
inputs:
azureSubscription: 'Pay-As-You-Go(d955c0ba-abcd-abcd-abcd-abcdabcd)'
KeyVaultName: 'idjakv'
SecretsFilter: '*'
RunAsPreJob: true
- script: |
echo Show the idjtestsecret Secret
echo $(idjtestsecret)
displayName: 'Show the secret'
This sets a secret inline
Azure DevOps Group Variables
The other approach we can take is to let AzDO pull in secrets for us using a “Group Variable”. The small advantage here is the authentication pieces stay over in Pipelines/Library meaning our users need not know the specifics of how they access the AKV backed secrets.
We can Create a new GroupVariable from Pipelines/Library and pick the Subscription and Vault name
Now what is different from the AKV step we used before; this will not pull in all secrets. You have to use the “+Add” to pick a subset of secrets to expose in our Group Variable instance.
I can pick just the one secret I wash to share
Then save
We now add the “group” type variable at the top of file (or in the stage/job depending on your setup) and use it in a step.
trigger:
- main
pool:
vmImage: ubuntu-latest
variables:
- group: testAkvGroup
steps:
- script: |
set -x
echo $(idjtestsecret) | base64 > ./t.o && cat ./t.o
displayName: 'show secret'
I’m choosing to base64 it so we can see a value. AzDO will mask all known secret values so just echo’ing it would not work.
Unless you set “All Pipelines” on permissions (which is not the default), we’ll need to approve this pipeline on the first run. We will only need to do this once
Click permit
I noticed they added a double-confirm (a bit annoying)
I can now see the output:
GCP Secret Manager
In order to use GCP Servics, one way or another, we will need to get a GCP Service Account JSON into our pipeline.
The easiest way is to just base64 the JSON file as use a pipeline variable.
$ cat mysecretaccessorsa.json | base64 -w0
ewogICJ0eXBlIjogIn......
Then we set that in a New Variable
Which we’ll set to “Keep this value secret”
Then save.
Now we can use in the Azure Pipeline YAML
- task: Bash@3
inputs:
targetType: 'inline'
script: |
# Create GCP SA JSON file
echo $(gcpspjsonb64) | base64 --decode > ./gcpsp.json
# Download and install the GCloud binary
- script: |
set -x
wget https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz
tar zxvf google-cloud-sdk.tar.gz && ./google-cloud-sdk/install.sh --quiet --usage-reporting=false --path-update=true
PATH="google-cloud-sdk/bin:${PATH}"
gcloud components update
yes | gcloud components install beta
gcloud auth activate-service-account --key-file ./gcpsp.json
gcloud beta secrets versions access latest --project myanthosproject2 --secret=mytestsecret2
displayName: 'Auth and show secret'
which we can see reflected in the results
We can just as easily add a secret set. I like to do the secret create that fails over to the update, just to make the code idempotent
- script: |
set -x
wget https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz
tar zxvf google-cloud-sdk.tar.gz && ./google-cloud-sdk/install.sh --quiet --usage-reporting=false --path-update=true
PATH="google-cloud-sdk/bin:${PATH}"
gcloud components update
yes | gcloud components install beta
gcloud auth activate-service-account --key-file ./gcpsp.json
# Set the secret
date > ./mycurrentdate
gcloud beta secrets create mytestsecret2 --project myanthosproject2 --data-file=./mycurrentdate || gcloud secrets versions add mytestsecret2 --project myanthosproject2 --data-file=./mycurrentdate
# Then read the secret
gcloud beta secrets versions access latest --project myanthosproject2 --secret=mytestsecret2
displayName: 'Auth and show secret'
And the successful output
We note that it created version 12 of the secret, so let’s check that out
Summary
In this post we tackled Azure and GCP Secrets in both Github Actions and Azure Pipelines. We covered creating and using GCP Service Accounts with the downloaded JSON file then covered using Azure Service Principals and the components of the JSON, namely the Client/App ID and Secret.
While I didn’t cover AWS, the patterns are similar.
Azure DevOps has both a SSM task:
- task: SystemsManagerGetParameter@1
inputs:
awsCredentials: 'AWS-FB'
regionName: 'us-west-2'
readMode: 'single'
parameterName: 'mysecret1'
as does Github Actions
- uses: "aws-actions/configure-aws-credentials@v1"
with:
aws-access-key-id: $
aws-secret-access-key: $
aws-region: "us-west-2"
role-to-assume: "arn:aws:iam::111111111111:role/secretAccessor"
role-duration-seconds: 1800 # 30 mins
- uses: "marvinpinto/action-inject-ssm-secrets@latest"
with:
ssm_parameter: "/mysecret1"
env_variable_name: "mysecret1"
AWS is still the leading Cloud Provider. However, I tend to skip it as most of my day-to-day is with Azure and GCP.
When it comes to providers, my advice would be to lean in to where you use the secrets the most.
If most of your CICD lies in Azure DevOps or Github Actions, there is merit to sticking with AKV. If you leverage a lot of GCP Cloud Build and Google Cloud Deploy, you may have reasons to leverage GCP Secret Manager.
I checked my own bills and found I had US$0 for GCP (though I rarely use it) and in the last 3 months, US$3 for AKV. So both options are essentially low cost.