Published: Oct 24, 2023 by Isaac Johnson

I don’t get that many ‘feedback’ form responses from that upper right feedback form, but know that I do treat them as top priority when they come in.

I had someone use it last week and I noticed, sadly, the body got garbled when it arrived. Luckily I have logs and double enter into JIRA and GH Issues (from when I converted out of AzDO), but I figured I might show how I improved things.

First, we’ll fix the issue on tokens, then do some tests with a new token. Then we’ll use a narrowly defined one for use with a secrets manager. Lastly, I’ll show you some handy AKV scripts I use daily (listpass, getpass and setpass).

The Problem

The real issue is I had a temporal token that I needed to remember to keep up to date. Systems that require my meat-memory are destined to fail.

And when someone gave me a message in the early hours, the Github part nulled out

(Luckily I have redundancy for a reason)

Now, I do not know the prior value. When I updated it this morning to the current 3 month token, things worked again:

Solution

This comes down to a source of truth issue. Something I have brought up in my role at work as well. If we have a rotatable token, it is important to minimize sprawl and use. IF you must save it to multiple places, those places must be tracked somewhere or you will forget to update (as likely was the case here).

Ideally, we should pull tokens, in a secure and controlled manager, from a secret repository like SSM, AKV or GSM. As I am an Azure man at heart, let’s use AKV.

The workflow repo is public so you can follow along, but the workflow presently pulls from a Github Actions Secret (GH90DTOKEN):

          curl -X POST -H "Accept: application/vnd.github+json" \
              -H "Authorization: Bearer $" \
              -H "X-GitHub-Api-Version: 2022-11-28" \
              https://api.github.com/repos/idjohnson/jekyll-blog/issues -d @emailJson2.json > gh.o.json

Since at the end of the day, I store this in AKV for use in systems (and frankly, my own damn memory):

$ listpass.sh idjakv | grep -i 90
GithubToken-MyFull90d                      https://idjakv.vault.azure.net/secrets/GithubToken-MyFull90d
                                                       True       2025-11-13T12:08:33+00:00
$ getpass.sh GithubToken-MyFull90d idjakv
ghp_xxxxxxxxxxxxxxxxxxxxxxxx

However, if we look up a bit, we realize we abandoned using a GH Secret for JIRA which is why it’s rotation worked

- name: 'Az CLI login'
        uses: azure/login@v1
        with:
          client-id: '$'
          tenant-id: '$'
          subscription-id: '$'
      - name: 'JIRA Create'
        run: |
          az keyvault secret show --vault-name wldemokv --name jiraapitoken -o json | jq -r .value > jiraapitoken

I could do the same

$ az keyvault secret show --vault-name idjakv --name GithubToken-MyFull90d -o json | jq -r .value
ghp_

The thing is, that is a big key with a big blast radius! Instead, we need a narrower scoped token. I’ll need to grant repo for private repo access

I did a quick test to ensure things worked:

$ jq -nc --arg title "This is a Test" --arg body "This is a Test Body :: Requested by isaac@freshbrewed.science" '$ARG
S.named'
{"title":"This is a Test","body":"This is a Test Body :: Requested by isaac@freshbrewed.science"}

$ jq -nc --arg title "This is a Test" --arg body "This is a Test Body :: Requested by isaac@freshbrewed.science" '$ARGS.named' > body.json

$ curl -X POST -H "Accept: application/vnd.github+json" -H "Authorization: Bearer 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" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/idjohnson/jekyll-blog/issues -d @body.json
{
  "url": "https://api.github.com/repos/idjohnson/jekyll-blog/issues/284",
  "repository_url": "https://api.github.com/repos/idjohnson/jekyll-blog",
  "labels_url": "https://api.github.com/repos/idjohnson/jekyll-blog/issues/284/labels{/name}",
  "comments_url": "https://api.github.com/repos/idjohnson/jekyll-blog/issues/284/comments",
  "events_url": "https://api.github.com/repos/idjohnson/jekyll-blog/issues/284/events",
  "html_url": "https://github.com/idjohnson/jekyll-blog/issues/284",
  "id": 1954120469,
  "node_id": "I_kwDOF-hlms50eYMV",
  "number": 284,
  "title": "This is a Test",
  "user": {
    "login": "idjohnson",
    "id": 6699477,
    "node_id": "MDQ6VXNlcjY2OTk0Nzc=",
    "avatar_url": "https://avatars.githubusercontent.com/u/6699477?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/idjohnson",
    "html_url": "https://github.com/idjohnson",
    "followers_url": "https://api.github.com/users/idjohnson/followers",
    "following_url": "https://api.github.com/users/idjohnson/following{/other_user}",
    "gists_url": "https://api.github.com/users/idjohnson/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/idjohnson/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/idjohnson/subscriptions",
    "organizations_url": "https://api.github.com/users/idjohnson/orgs",
    "repos_url": "https://api.github.com/users/idjohnson/repos",
    "events_url": "https://api.github.com/users/idjohnson/events{/privacy}",
    "received_events_url": "https://api.github.com/users/idjohnson/received_events",
    "type": "User",
    "site_admin": false
  },
  "labels": [

  ],
  "state": "open",
  "locked": false,
  "assignee": null,
  "assignees": [

  ],
  "milestone": null,
  "comments": 0,
  "created_at": "2023-10-20T11:46:49Z",
  "updated_at": "2023-10-20T11:46:49Z",
  "closed_at": null,
  "author_association": "OWNER",
  "active_lock_reason": null,
  "body": "This is a Test Body :: Requested by isaac@freshbrewed.science",
  "closed_by": null,
  "reactions": {
    "url": "https://api.github.com/repos/idjohnson/jekyll-blog/issues/284/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
  },
  "timeline_url": "https://api.github.com/repos/idjohnson/jekyll-blog/issues/284/timeline",
  "performed_via_github_app": null,
  "state_reason": null
}

and saw it worked

I could have made this never expire, set as a secret in the workflow and been done. But it’s good to having a forcing function to rotate keys.

I don’t want to get too deep into my AKV strategies, but suffice to say that ‘wldemokv’ has a much smaller blast radius as it’s primarily notification tokens.

That said, I’ll set the new key there:

$ setpass.sh My90DayGHIssueWriter wldemokv 'Used in workflowTriggerTest GH action' ghp_zzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+ '[' wldemokv == work ']'
+ '[' My90DayGHIssueWriter == help ']'
+ az keyvault secret set --vault-name wldemokv --name My90DayGHIssueWriter --subscription Pay-As-You-Go --description 'Used in workflowTriggerTest GH action' --value ghp_zzzzzzzzzzzzzzzzzzzzzzzzzzzzz
{
    ... snip ...

Because I know me and that when i rotate, i lean into a different key vault, I’ll leave a reminder there for myself for next time.

$ listpass.sh idjakv | grep -i GH | grep 90
My90DayGHIssueWriter                       https://idjakv.vault.azure.net/secrets/My90DayGHIssueWriter                       UPDATE IN wldemokv AKV TOO                                               True

Lastly, while likely a bit overkill, reminders aren’t bad - I’ll drop a note in my call for January

Okay, we have that secret now all set and reminders in place. We can now update the workflow. By pulling it just at time of use, that is one less secret that could be left in a filesystem

curl -X POST -H "Accept: application/vnd.github+json" \
    -H "Authorization: Bearer `az keyvault secret show --vault-name wldemokv --name My90DayGHIssueWriter -o json | jq -r .value | tr -d '\n'`" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    https://api.github.com/repos/idjohnson/jekyll-blog/issues -d @emailJson2.json > gh.o.json

And test:

We can see it notified when done

AKV scripts

I actually use some scripts I wrote quite a lot, both at home and at work, to check on AKV.

listpass.sh - used to fetch the secrets in different vaults to find the one I’m looking for (since grep is my friend)
getpass.sh - what it sounds like, just get the AKV secret
setpass.sh - again, what it sounds like, set a password (or update if exists) in AKV

In my versions, AWORKKV is the name of a teams AKV we use and ANEMPLOYERSUBSCRIPTION is the subscription name. Felt it would be below the line to share that.

List

$ cat /usr/local/bin/listpass.sh
#!/bin/bash +x
if [ "$1" == "work" ]; then
  az keyvault secret list --vault-name AWORKKV --subscription "ANEMPLOYERSUBSCRIPTION" -o table
elif [ "$1" == "help" ]; then
  echo "$0 [vault-name/work]"
  echo ""
  echo "Here are your Keyvaults:"
  az keyvault list --subscription Pay-As-You-Go -o table
else
  az keyvault secret list --vault-name $1 --subscription Pay-As-You-Go -o table
fi

Get

#!/bin/bash +x
if [ "$2" == "work" ]; then
  az keyvault secret show --vault-name AWORKKV --subscription "ANEMPLOYERSUBSCRIPTION" --name $1 | jq -r .value
elif [ "$1" == "help" ]; then
  echo "$0 [secret name] [vault-name/work]"
  echo ""
  echo "Here are your Key Vaults:"
  az keyvault list --subscription Pay-As-You-Go -o table
else
  az keyvault secret show --vault-name $2 --name $1 --subscription Pay-As-You-Go | jq -r .value
fi

Set

#!/bin/bash -x
if [ "$2" == "work" ]; then
  az keyvault secret set --vault-name AWORKKV --subscription "ANEMPLOYERSUBSCRIPTION" --name $1 --description "$3" --value $4
elif [ "$1" == "help" ]; then
  echo "$0 [secret name] [vault-name/work] [description] [value]"
  echo ""
  echo "Here are your Key Vaults:"
  az keyvault list --subscription Pay-As-You-Go -o table
else
  az keyvault secret set --vault-name $2 --name $1 --subscription Pay-As-You-Go --description "$3" --value $4
fi

You can see an example of help which reminds me of my vaults

Summary

In this post we found the issue with an expired GH token and verified the fix. We then improved the process by moving to a common secret store and lastly, we reduced risk by lowering the permissions on the token and switching vaults. I also showed some common AKV scripts I use daily for fetching, listing and setting secrets.