Published: Jun 6, 2024 by Isaac Johnson
Recently Github rolled out preview of Github Attestation that let one create “signed attestations” for anything built in Github Actions.
Let’s dig into this Preview release and see how we can use it to authenticate binaries from our Github workflows. We’ll look at binaries, containers and local as well as remote verification.
Usage
To test this out, I’ll use a public project of mine, Express Uploader
$ git clone https://github.com/idjohnson/expressUploader.git
I know I need to add permissions
permissions:
id-token: write
attestations: write
contents: read
But I think I’m okay as I already have ‘write-all’ enabled
jobs:
build:
permissions: write-all
runs-on: ubuntu-latest
Let’s save my Docker image to an Attestation
name: Build and Push Docker Image to GHCR
on:
push:
branches:
- main
workflow_dispatch:
jobs:
build:
permissions: write-all
runs-on: ubuntu-latest
steps:
- name: 'Checkout GitHub Action'
uses: actions/checkout@main
- name: 'Login to GitHub Container Registry'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: $
password: $
- name: 'Build and Push Image'
run: |
docker build . --tag ghcr.io/$/expressupload:latest
docker push ghcr.io/$/expressupload:latest
- name: 'Save as a TGZ'
run: |
docker save ghcr.io/$/expressupload:latest | gzip > expressupload.tar.gz
- name: Attest Build Provenance
uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
with:
subject-path: "expressupload.tar.gz"
- name: 'Helm package and push'
run: |
helm package ./helm/
export HELM_EXPERIMENTAL_OCI=1
helm push expressupload-*.tgz oci://ghcr.io/$/
We can see it passed and is uploaded
The idea is I can now verify the binary ( expressupload.tar.gz), provided I have it somewhere, that it is indeed that which I pushed.
I pulled and recompressed and got a 404 on that new archive
$ gh attestation verify ./expressupload.tgz -o idjohnson
Loaded digest sha256:c0fb67da9c87b66aa3a62dd223f69106c7e6aee6b3f3391f6233c3adf8a4af14 for file://expressupload.tgz
✗ Loading attestations from GitHub API failed
Error: failed to fetch attestations from idjohnson: HTTP 404: Not Found (https://api.github.com/orgs/idjohnson/attestations/sha256:c0fb67da9c87b66aa3a62dd223f69106c7e6aee6b3f3391f6233c3adf8a4af14?per_page=30)
That is because this is not the same binary. We can see that from the attstation page
Let’s go another way, let’s test something we can ensure is the same, the tgz’ed helm chart as pushed as an OCI artifact.
I’ll change it up in the Github flow
name: Build and Push Docker Image to GHCR
on:
push:
branches:
- main
workflow_dispatch:
jobs:
build:
permissions: write-all
runs-on: ubuntu-latest
steps:
- name: 'Checkout GitHub Action'
uses: actions/checkout@main
- name: 'Login to GitHub Container Registry'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: $
password: $
- name: 'Build and Push Image'
run: |
docker build . --tag ghcr.io/$/expressupload:latest
docker push ghcr.io/$/expressupload:latest
- name: 'Helm package and push'
run: |
helm package ./helm/
export HELM_EXPERIMENTAL_OCI=1
helm push expressupload-*.tgz oci://ghcr.io/$/
- name: Attest Build Provenance
uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
with:
subject-path: "expressupload-0.1.0.tgz"
which I can see worked and uploaded a new SHA for verification
I can now pull it down locally
$ helm pull oci://ghcr.io/idjohnson/expressupload --version 0.1.0
Pulled: ghcr.io/idjohnson/expressupload:0.1.0
Digest: sha256:257b70bd3d03a5316477c53ce39a399812a921cb4ad965c59ca1daa91441e59a
As expected, this verifies successfully
$ gh attestation verify expressupload-0.1.0.tgz -o idjohnson
Loaded digest sha256:ff7ba1bb25585daaa3d2ac762f1edf0efefaa5a812e63ae311fb04e84a050b12 for file://expressupload-0.1.0.tgz
Loaded 1 attestation from GitHub API
✓ Verification succeeded!
sha256:ff7ba1bb25585daaa3d2ac762f1edf0efefaa5a812e63ae311fb04e84a050b12 was attested by:
REPO PREDICATE_TYPE WORKFLOW
idjohnson/expressUploader https://slsa.dev/provenance/v1 .github/workflows/build.yml@refs/heads/main
Validating in GH
We can now change the build YAML to pull down the artifact and validate it is indeed the right tgz using attest
name: Build and Push Docker Image to GHCR
on:
push:
branches:
- main
workflow_dispatch:
jobs:
build:
permissions: write-all
runs-on: ubuntu-latest
steps:
- name: 'Checkout GitHub Action'
uses: actions/checkout@main
- name: 'Login to GitHub Container Registry'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: $
password: $
- name: 'Build and Push Image'
run: |
docker build . --tag ghcr.io/$/expressupload:latest
docker push ghcr.io/$/expressupload:latest
- name: 'Helm package and push'
run: |
helm package ./helm/
export HELM_EXPERIMENTAL_OCI=1
helm push expressupload-*.tgz oci://ghcr.io/$/
- name: Attest Build Provenance
uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
with:
subject-path: "expressupload-0.1.0.tgz"
validate:
runs-on: ubuntu-latest
needs: build
steps:
- name: 'Fetch Chart'
run: |
export HELM_EXPERIMENTAL_OCI=1
helm pull oci://ghcr.io/$/expressupload --version 0.1.0
- name: 'Validate'
run: |
set -x
ls -l
gh attestation verify ./expressupload-0.1.0.tgz -o idjohnson
env:
GH_TOKEN: $
- name: 'Validate2'
run: |
set -x
ls -l
gh attestation verify ./expressupload-0.1.0.tgz -R idjohnson/expressUploader
Above I am validating two ways; one against the organization and one against the specific repo (Validate2)
This should kick off a build
However it fails for a good reason; we didnt set any auth for the GH CLI
We can use the ephimeral Github Token provided to our workflows (the provided secret GITHUB_TOKEN). I’ll add to the top of the validate build job.
name: Build and Push Docker Image to GHCR
on:
push:
branches:
- main
workflow_dispatch:
jobs:
build:
permissions: write-all
runs-on: ubuntu-latest
steps:
- name: 'Checkout GitHub Action'
uses: actions/checkout@main
- name: 'Login to GitHub Container Registry'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: $
password: $
- name: 'Build and Push Image'
run: |
docker build . --tag ghcr.io/$/expressupload:latest
docker push ghcr.io/$/expressupload:latest
- name: 'Helm package and push'
run: |
helm package ./helm/
export HELM_EXPERIMENTAL_OCI=1
helm push expressupload-*.tgz oci://ghcr.io/$/
- name: Attest Build Provenance
uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
with:
subject-path: "expressupload-0.1.0.tgz"
validate:
runs-on: ubuntu-latest
needs: build
env:
GH_TOKEN: $
steps:
- name: 'Fetch Chart'
run: |
export HELM_EXPERIMENTAL_OCI=1
helm pull oci://ghcr.io/$/expressupload --version 0.1.0
- name: 'Validate'
run: |
set -x
ls -l
gh attestation verify ./expressupload-0.1.0.tgz -o idjohnson
- name: 'Validate2'
run: |
set -x
ls -l
gh attestation verify ./expressupload-0.1.0.tgz -R idjohnson/expressUploader
We can now see it is validated without issue, both using my organization (just me) and the specific repo
Container images
I did show using container images with attest as just a tgz’ed file, however there is a much simpler way.
name: Build and Push Docker Image to GHCR
on:
push:
branches:
- main
workflow_dispatch:
jobs:
build:
permissions: write-all
runs-on: ubuntu-latest
steps:
- name: 'Checkout GitHub Action'
uses: actions/checkout@main
- name: 'Login to GitHub Container Registry'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{github.actor}}
password: ${{secrets.GITHUB_TOKEN}}
- name: 'Build and Push Image'
run: |
docker build . --tag ghcr.io/${{github.actor}}/expressupload:latest
docker push ghcr.io/${{github.actor}}/expressupload:latest
- name: 'Build and Push Image'
run: |
docker build . --tag ghcr.io/${{github.actor}}/expressupload:latest
docker push ghcr.io/${{github.actor}}/expressupload:latest
echo "======================= TESTS ======================="
docker images --no-trunc --quiet ghcr.io/${{github.actor}}/expressupload:latest
docker inspect -f '{{.Id}}' ghcr.io/${{github.actor}}/expressupload:latest
- name: 'Set Image SHA0'
run: |
CONTAINERSHA=`docker images --no-trunc --quiet ghcr.io/${{github.actor}}/expressupload:latest`
echo "CONTAINERSHA=$CONTAINERSHA" >> $GITHUB_OUTPUT
id: consha0
- name: 'Set Image SHA'
run: |
CONTAINERSHA=$(docker inspect -f '{{.Id}}' ghcr.io/${{github.actor}}/expressupload:latest)
echo "CONTAINERSHA=$CONTAINERSHA" >> $GITHUB_OUTPUT
id: consha1
- name: Generate artifact attestation
uses: actions/attest-build-provenance@v1
with:
subject-name: ghcr.io/${{github.actor}}/expressupload
subject-digest: ${{ steps.consha0.outputs.CONTAINERSHA }}
push-to-registry: true
- name: 'Helm package and push'
run: |
helm package ./helm/
export HELM_EXPERIMENTAL_OCI=1
helm push expressupload-*.tgz oci://ghcr.io/${{github.actor}}/
- name: Attest Build Provenance
uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
with:
subject-path: "expressupload-0.1.0.tgz"
validate:
runs-on: ubuntu-latest
needs: build
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: 'Fetch Chart'
run: |
export HELM_EXPERIMENTAL_OCI=1
helm pull oci://ghcr.io/${{github.actor}}/expressupload --version 0.1.0
- name: 'Validate'
run: |
set -x
ls -l
gh attestation verify ./expressupload-0.1.0.tgz -o idjohnson
- name: 'Validate2'
run: |
set -x
ls -l
gh attestation verify ./expressupload-0.1.0.tgz -R idjohnson/expressUploader
I actually tried this a few ways. It does upload, but errors on verification. I can see it is adding a “v2” to the URL
But I can see results were uploaded on the SHA
Reviewing the docs, I question if I really need the ‘push-to-registry’ step
I removed it
...
id: consha1
- name: Generate artifact attestation
uses: actions/attest-build-provenance@v1
with:
subject-name: ghcr.io/$/expressupload
subject-digest: $
- name: 'Helm package and push'
...
And this time it worked
And we can view the attest on the repo as it’s public
The point
So what is the point? Why create attestation entry? As the Github Docs say:
Artifact attestations enable you to create unfalsifiable provenance and integrity guarantees for the software you build. In turn, people who consume your software can verify where and how your software was built.
Think of block chain, but for build artifacts. It is a way to guarantee the thing you downloaded was that which was created from the originating pipeline.
The highest profile attack (of which we are aware) was the Solarwindws Hack of 2019 in which threat actors attacked the Orion build pipeline with malicious code known as ‘sunburst’.
Near the end of that article, in the section “Need for software bill of materials highlighted in aftermath of attack” it is noted why SBOMs are so important:
Modern software applications no longer rely on a monolithic stack of discrete software components. Developers now build applications out of many components that can come from many sources. Any one of the components that makes up an application could potentially represent a risk if there is an unpatched vulnerability. As such, it is critical for developers, organizations they work for and end users that consume applications be aware of all the different components that make up an application. It’s an approach that is known as a software bill of materials (SBOM). An SBOM is like a “nutritional label” that is present on packaged food products, clearly showing consumers what’s inside a product.
A Software Bill of Material (SBOM) that can then be checked against a secure store, such as Attestation, is key to minimizing these kinds of vulnerabilities.
A bit of commentary
We have many ways to check and store SHA256 hashes. The idea of a Container Digest is just that - a unique generatable key that matches a container to guarantee that is the same container.
In past systems I have often used the SHA as the image tag to guarantee authenticity, even if it makes image tags harder to sort and trace (you can always have more than one tag on a container image repository).
Another way I minimize risk is to use more than on Artifact store. For a small extra price for redundant storage, I’ve long held the line about storing containers in multiple clouds (or cloud and on-prem). Containers can be identified by SHA and any outage or threat detected can be mitigated by comparing to the others. At one prior employer, the AzDO system would store containers in Azure, AWS and an on-prem Artifactory repo. I could then guarantee quorum at any point. It also let me minimize retention policies in certain clouds to save on spend.
Summary
The announcement just came out this May with promises of more deep dives in Github Universe in October. Right now, it’s just part of the Github deal - no additional pricing.
I see Attestations as a great (?free) way to add some SBOM authenticity checks to our pipelines, provided one builds in Github.
There are non-commercial options, mostly using the blockchain like Tacos and OpenAttestation which we might explore in future posts. (If interested, start here)
