Published: Dec 5, 2024 by Isaac Johnson
Sometimes we have to do a little care love and feeding of our Home Lab. Today I wanted to share a few recent instances of that kind of thing.
We will cover some MinIO and DSM (Synology) updates and Synology disk checks. We’ll look at Beszel and remediate some disk space and container waste issues. I’ll talk about the Mystery Page of 3AM and upgrading hosts. I’ll also show what happens when one doesn’t physically check old laptops from time to time. The findings could be explosive.
MinIO and DSM Updates
I saw a page first thing in the morning:
It took me a moment to realize 116 is my newer NAS (SirNasilot).
When I checked Uptime Kuma, I could see it had started to fail a ping test
I circled back to the original article that I covered putting Minio on a NAS and saw it was via Docker compose.
I did a quick SSH there and saw it was up and “healthy” but what really caught me was “Up 32 minutes”
builder@DESKTOP-QADGF36:~$ ssh ijohnson@192.168.1.116
ijohnson@192.168.1.116's password:
Using terminal commands to modify system configs, execute external binary
files, add files, or install unauthorized third-party apps may lead to system
damages or unexpected behavior, or cause data loss. Make sure you are aware of
the consequences of each command and proceed at your own risk.
Warning: Data should only be stored in shared folders. Data stored elsewhere
may be deleted when the system is updated/restarted.
ijohnson@sirnasilot:~$ sudo docker ps
Password:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4d5cf797f616 quay.io/minio/minio:RELEASE.2024-08-29T01-40-52Z "/usr/bin/docker-ent…" 2 months ago Up 32 minutes (healthy) 9000-9001/tcp docker-compose-minio4-1
7e8e33e33932 quay.io/minio/minio:RELEASE.2024-08-29T01-40-52Z "/usr/bin/docker-ent…" 2 months ago Up 32 minutes (healthy) 9000-9001/tcp docker-compose-minio2-1
517b2c03bae9 quay.io/minio/minio:RELEASE.2024-08-29T01-40-52Z "/usr/bin/docker-ent…" 2 months ago Up 32 minutes (healthy) 9000-9001/tcp docker-compose-minio1-1
b1d94a99655c quay.io/minio/minio:RELEASE.2024-08-29T01-40-52Z "/usr/bin/docker-ent…" 2 months ago Up 31 minutes (healthy) 9000-9001/tcp docker-compose-minio3-1
5fa76248b69d portainer/portainer-ee:latest "/portainer" 16 months ago Up 32 minutes 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp, 9000/tcp portaineree
83af22fb4b32 registry:latest "/entrypoint.sh /etc…" 16 months ago Up 32 minutes 0.0.0.0:5050->5000/tcp, :::5050->5000/tcp registry-1
Why 32 minutes? We didn’t have a local power outtage and I have that NAS on my newest UPS.
Logging into the Synology OS, I realized the culprit. There was a major OS update applied and it self-restarted
My guess is Docker tried to restart containers, but Minio, with Docker compose has a few services and containers and I’m guessing they just didn’t start up clean
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ cat docker-compose.yaml
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:RELEASE.2024-08-29T01-40-52Z
command: server --console-address ":9001" http://minio{1...4}/data{1...2}
expose:
- "9000"
- "9001"
environment:
MINIO_ROOT_USER: admin
MINIO_ROOT_PASSWORD: drumpf8poop
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
minio1:
<<: *minio-common
hostname: minio1
volumes:
- data1-1:/data1
- data1-2:/data2
minio2:
<<: *minio-common
hostname: minio2
volumes:
- data2-1:/data1
- data2-2:/data2
minio3:
<<: *minio-common
hostname: minio3
volumes:
- data3-1:/data1
- data3-2:/data2
minio4:
<<: *minio-common
hostname: minio4
volumes:
- data4-1:/data1
- data4-2:/data2
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9001:9001"
depends_on:
- minio1
- minio2
- minio3
- minio4
## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
data1-1:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data1-1
data1-2:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data1-2
data2-1:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data2-1
data2-2:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data2-2
data3-1:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data3-1
data3-2:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data3-2
data4-1:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data4-1
data4-2:
driver: local
driver_opts:
o: bind
type: none
device: /volume1/minio/data4-2
This was also affirmed trying to reach the Administration page
I did a docker compose stop then a docker compose start to see that indeed it would come up
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo docker-compose down
[+] Running 6/6
⠿ Container docker-compose-nginx-1 Removed 2.6s
⠿ Container docker-compose-minio4-1 Removed 14.6s
⠿ Container docker-compose-minio1-1 Removed 16.6s
⠿ Container docker-compose-minio2-1 Removed 12.7s
⠿ Container docker-compose-minio3-1 Removed 17.6s
⠿ Network docker-compose_default Removed 2.5s
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo docker-compose up
[+] Running 6/6
⠿ Network docker-compose_default Created 1.6s
⠿ Container docker-compose-minio3-1 Created 6.7s
⠿ Container docker-compose-minio4-1 Created 7.2s
⠿ Container docker-compose-minio1-1 Created 6.1s
⠿ Container docker-compose-minio2-1 Created 5.5s
⠿ Container docker-compose-nginx-1 Created 1.1s
Attaching to docker-compose-minio1-1, docker-compose-minio2-1, docker-compose-minio3-1, docker-compose-minio4-1, docker-compose-nginx-1
docker-compose-minio2-1 | INFO:
docker-compose-minio2-1 | You are running an older version of MinIO released 2 months before the latest release
docker-compose-minio2-1 | Update: Run `mc admin update ALIAS`
docker-compose-minio2-1 |
docker-compose-minio2-1 |
docker-compose-minio2-1 |
docker-compose-minio2-1 | API: SYSTEM.grid
docker-compose-minio2-1 | Time: 12:02:50 UTC 11/20/2024
docker-compose-minio2-1 | Error: grid: http://minio2:9000 re-connecting to http://minio4:9000: dial tcp 172.18.0.3:9000: connect: connection refused (*net.OpError) Sleeping 1.293s (3) (*fmt.wrapError)
docker-compose-minio2-1 | 6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio2-1 | 5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio2-1 | 4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio2-1 | 3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio2-1 | 2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio2-1 | 1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio2-1 | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio2-1 | INFO: Following servers are currently offline or unreachable [http://minio2:9000->http://minio4:9000 is unreachable: remote disconnected http://minio2:9000->http://minio1:9000 is unreachable: remote disconnected http://minio2:9000->http://minio3:9000 is unreachable: remote disconnected]
docker-compose-minio2-1 | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio2-1 | INFO: Following servers are currently offline or unreachable [http://minio2:9000->http://minio4:9000 is unreachable: remote disconnected http://minio2:9000->http://minio1:9000 is unreachable: remote disconnected http://minio2:9000->http://minio3:9000 is unreachable: remote disconnected]
docker-compose-minio4-1 | INFO:
docker-compose-minio4-1 | You are running an older version of MinIO released 2 months before the latest release
docker-compose-minio4-1 | Update: Run `mc admin update ALIAS`
docker-compose-minio4-1 |
docker-compose-minio4-1 |
docker-compose-minio2-1 | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio2-1 | INFO: Following servers are currently offline or unreachable [http://minio2:9000->http://minio4:9000 is unreachable: remote disconnected http://minio2:9000->http://minio1:9000 is unreachable: remote disconnected http://minio2:9000->http://minio3:9000 is unreachable: remote disconnected]
docker-compose-minio4-1 | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio4-1 | INFO: Following servers are currently offline or unreachable [http://minio4:9000->http://minio3:9000 is unreachable: remote disconnected http://minio4:9000->http://minio2:9000 is unreachable: remote disconnected http://minio4:9000->http://minio1:9000 is unreachable: remote disconnected]
docker-compose-minio4-1 |
docker-compose-minio4-1 | API: SYSTEM.grid
docker-compose-minio4-1 | Time: 12:02:54 UTC 11/20/2024
docker-compose-minio4-1 | Error: grid: http://minio4:9000 re-connecting to http://minio3:9000: dial tcp 172.18.0.4:9000: connect: connection refused (*net.OpError) Sleeping 1.518s (3) (*fmt.wrapError)
docker-compose-minio4-1 | 6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio4-1 | 5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio4-1 | 4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio4-1 | 3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio4-1 | 2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio4-1 | 1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio2-1 | INFO: Unable to use the drive http://minio1:9000/data1: drive not found, will be retried
docker-compose-minio2-1 | INFO: Unable to use the drive http://minio3:9000/data1: drive not found, will be retried
docker-compose-minio2-1 | INFO: Unable to use the drive http://minio1:9000/data2: drive not found, will be retried
docker-compose-minio2-1 | INFO: Unable to use the drive http://minio3:9000/data2: drive not found, will be retried
docker-compose-minio4-1 | INFO: Unable to use the drive http://minio1:9000/data1: drive not found, will be retried
docker-compose-minio4-1 | INFO: Unable to use the drive http://minio3:9000/data1: drive not found, will be retried
docker-compose-minio4-1 | INFO: Unable to use the drive http://minio1:9000/data2: drive not found, will be retried
docker-compose-minio4-1 | INFO: Unable to use the drive http://minio3:9000/data2: drive not found, will be retried
docker-compose-minio4-1 | INFO: Waiting for quorum READ healthcheck to succeed retrying in 827.580805ms.. possible cause unhealthy sets
docker-compose-minio4-1 | INFO: (Pool: 0 Set: 0 Healthy: false)
docker-compose-minio2-1 | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio2-1 | INFO: Configured max API requests per node based on available memory: 714
docker-compose-minio2-1 | INFO: All MinIO sub-systems initialized successfully in 5.657111ms
docker-compose-minio2-1 | INFO: IAM load(startup) finished. (duration: 5.131644ms)
docker-compose-minio2-1 | ---------------------------
docker-compose-minio2-1 | MinIO Object Storage Server
docker-compose-minio2-1 | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio2-1 | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio2-1 | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio2-1 |
docker-compose-minio2-1 | API: http://172.18.0.2:9000 http://127.0.0.1:9000
docker-compose-minio2-1 | WebUI: http://172.18.0.2:9001 http://127.0.0.1:9001
docker-compose-minio2-1 |
docker-compose-minio2-1 | Docs: https://docs.min.io
docker-compose-minio2-1 | ---------------------------
docker-compose-minio4-1 | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio4-1 | INFO: Configured max API requests per node based on available memory: 708
docker-compose-minio4-1 | INFO: All MinIO sub-systems initialized successfully in 2.901171ms
docker-compose-minio4-1 | INFO: IAM load(startup) finished. (duration: 27.235076ms)
docker-compose-minio4-1 | ---------------------------
docker-compose-minio4-1 | MinIO Object Storage Server
docker-compose-minio4-1 | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio4-1 | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio4-1 | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio4-1 |
docker-compose-minio4-1 | API: http://172.18.0.3:9000 http://127.0.0.1:9000
docker-compose-minio4-1 | WebUI: http://172.18.0.3:9001 http://127.0.0.1:9001
docker-compose-minio4-1 |
docker-compose-minio4-1 | Docs: https://docs.min.io
docker-compose-minio4-1 | ---------------------------
docker-compose-minio4-1 |
docker-compose-minio4-1 | API: SYSTEM.storage
docker-compose-minio4-1 | Time: 12:02:55 UTC 11/20/2024
docker-compose-minio4-1 | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio4-1 | Error: Write quorum could not be established on pool: 0, set: 0, expected write quorum: 5, drives-online: 4 (*errors.errorString)
docker-compose-minio4-1 | maintenance="false"
docker-compose-minio4-1 | 5: internal/logger/logger.go:268:logger.LogIf()
docker-compose-minio4-1 | 4: cmd/logging.go:156:cmd.storageLogIf()
docker-compose-minio4-1 | 3: cmd/erasure-server-pool.go:2658:cmd.(*erasureServerPools).Health()
docker-compose-minio4-1 | 2: cmd/healthcheck-handler.go:71:cmd.ClusterCheckHandler()
docker-compose-minio4-1 | 1: net/http/server.go:2171:http.HandlerFunc.ServeHTTP()
docker-compose-minio3-1 | INFO: Unable to use the drive http://minio1:9000/data1: drive not found, will be retried
docker-compose-minio3-1 | INFO: Unable to use the drive http://minio4:9000/data1: drive not found, will be retried
docker-compose-minio3-1 | INFO: Unable to use the drive http://minio1:9000/data2: drive not found, will be retried
docker-compose-minio3-1 | INFO: Unable to use the drive http://minio4:9000/data2: drive not found, will be retried
docker-compose-minio3-1 | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio3-1 | INFO: Configured max API requests per node based on available memory: 697
docker-compose-minio3-1 | INFO: All MinIO sub-systems initialized successfully in 6.928239ms
docker-compose-minio3-1 | INFO: IAM load(startup) finished. (duration: 4.665781ms)
docker-compose-minio3-1 | INFO:
docker-compose-minio3-1 | You are running an older version of MinIO released 2 months before the latest release
docker-compose-minio3-1 | Update: Run `mc admin update ALIAS`
docker-compose-minio3-1 |
docker-compose-minio3-1 |
docker-compose-minio3-1 | ---------------------------
docker-compose-minio3-1 | MinIO Object Storage Server
docker-compose-minio3-1 | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio3-1 | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio3-1 | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio3-1 |
docker-compose-minio3-1 | API: http://172.18.0.4:9000 http://127.0.0.1:9000
docker-compose-minio3-1 | WebUI: http://172.18.0.4:9001 http://127.0.0.1:9001
docker-compose-minio3-1 |
docker-compose-minio3-1 | Docs: https://docs.min.io
docker-compose-minio3-1 | ---------------------------
docker-compose-minio3-1 |
docker-compose-minio3-1 | API: SYSTEM.grid
docker-compose-minio3-1 | Time: 12:02:58 UTC 11/20/2024
docker-compose-minio3-1 | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1 | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.192s (3) (*fmt.wrapError)
docker-compose-minio3-1 | 6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1 | 5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1 | 4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1 | 3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1 | 2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1 | 1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio1-1 | INFO:
docker-compose-minio1-1 | You are running an older version of MinIO released 2 months before the latest release
docker-compose-minio1-1 | Update: Run `mc admin update ALIAS`
docker-compose-minio1-1 |
docker-compose-minio1-1 |
docker-compose-minio1-1 | INFO: Unable to use the drive http://minio3:9000/data1: drive not found, will be retried
docker-compose-minio1-1 | INFO: Unable to use the drive http://minio3:9000/data2: drive not found, will be retried
docker-compose-minio1-1 | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio1-1 | INFO: Configured max API requests per node based on available memory: 692
docker-compose-minio1-1 | INFO: All MinIO sub-systems initialized successfully in 4.162384ms
docker-compose-minio1-1 | INFO: IAM load(startup) finished. (duration: 11.987113ms)
docker-compose-minio1-1 | ---------------------------
docker-compose-minio1-1 | MinIO Object Storage Server
docker-compose-minio1-1 | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio1-1 | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio1-1 | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio1-1 |
docker-compose-minio1-1 | API: http://172.18.0.5:9000 http://127.0.0.1:9000
docker-compose-minio1-1 | WebUI: http://172.18.0.5:9001 http://127.0.0.1:9001
docker-compose-minio1-1 |
docker-compose-minio1-1 | Docs: https://docs.min.io
docker-compose-minio1-1 | ---------------------------
docker-compose-nginx-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
docker-compose-nginx-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
docker-compose-nginx-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
docker-compose-nginx-1 | 10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
docker-compose-nginx-1 | 10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
docker-compose-nginx-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
docker-compose-nginx-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
The goof, of course, was not adding -d so I had to test, stop, and start again with docker compose up -d
docker-compose-nginx-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Running 4/5
⠿ Container docker-compose-nginx-1 Stopped 5.3s
⠴ Container docker-compose-minio1-1 Stopping 14.6s
⠿ Container docker-compose-minio2-1 Stopped 11.1s
⠿ Container docker-compose-minio3-1 Stopped 14.5s
⠿ Container docker-compose-minio4-1 Stopped 11.9s
canceled
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo docker-compose up -d
[+] Running 5/5
⠿ Container docker-compose-minio3-1 Started 7.8s
⠿ Container docker-compose-minio4-1 Started 16.1s
⠿ Container docker-compose-minio1-1 Started 17.0s
⠿ Container docker-compose-minio2-1 Started 12.9s
⠿ Container docker-compose-nginx-1 Started
Now the Minio Admin Page is up
And Uptime Kuma is happy again too (so am I, of course)
Beszel checks
One thing I like to do on a periodic basis is check Beszel for my Dockerhost status. Mostly around Temps
While my temps tend to hover around 43C (110F)
I did notice I have some redundant loads active
Notably, I have two Rundecks and two Uptime Kuma instances going.
Now, I do use Uptime Kuma extensively to stay on top of things and that pairs nicely with Datadog which can catch the absence-of situations
Necessarily the “real” is the one served by the ingress
$ kubectl get ingress -A | grep uptime
default uptimeingress <none> uptime.freshbrewed.science 80, 443 268d
That is sending traffic on to uptime-external-ip on 80
$ kubectl get ingress uptimeingress -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"cert-manager.io/cluster-issuer":"letsencrypt-prod","kubernetes.io/ingress.class":"nginx","kubernetes.io/tls-acme":"true","nginx.ingress.kubernetes.io/proxy-read-timeout":"3600","nginx.ingress.kubernetes.io/proxy-send-timeout":"3600","nginx.org/websocket-services":"uptime-external-ip"},"labels":{"app.kubernetes.io/instance":"uptimeingress"},"name":"uptimeingress","namespace":"default"},"spec":{"rules":[{"host":"uptime.freshbrewed.science","http":{"paths":[{"backend":{"service":{"name":"uptime-external-ip","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}}],"tls":[{"hosts":["uptime.freshbrewed.science"],"secretName":"uptime-tls"}]}}
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.org/websocket-services: uptime-external-ip
creationTimestamp: "2024-03-02T16:30:47Z"
generation: 1
labels:
app.kubernetes.io/instance: uptimeingress
name: uptimeingress
namespace: default
resourceVersion: "55993"
uid: 6599ac1a-bc15-489e-a430-eea77ff79adb
spec:
rules:
- host: uptime.freshbrewed.science
http:
paths:
- backend:
service:
name: uptime-external-ip
port:
number: 80
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- uptime.freshbrewed.science
secretName: uptime-tls
status:
loadBalancer: {}
When I see ports but no targeted lables, that’s generally a clue that we have an endpoint at play
$ kubectl get svc uptime-external-ip -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"uptime-external-ip","namespace":"default"},"spec":{"clusterIP":"None","internalTrafficPolicy":"Cluster","ports":[{"name":"uptimep","port":80,"protocol":"TCP","targetPort":3101}],"sessionAffinity":"None","type":"ClusterIP"}}
creationTimestamp: "2024-03-02T16:30:47Z"
name: uptime-external-ip
namespace: default
resourceVersion: "55989"
uid: 726a0847-ec53-4301-ac73-19df5bf6497b
spec:
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
- IPv6
ipFamilyPolicy: RequireDualStack
ports:
- name: uptimep
port: 80
protocol: TCP
targetPort: 3101
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
I don’t even need to look at the YAML - as it’s sending on to the Dockerhost on 3101, that is what I needed to know
$ kubectl get endpoints uptime-external-ip
NAME ENDPOINTS AGE
uptime-external-ip 192.168.1.100:3101 268d
Going to my Dockerhost, it’s clear the 3101 was the upgraded version and yet i still have the version 1 instance running
builder@builder-T100:~$ docker ps | grep uptime
1700cf832175 louislam/uptime-kuma:1.23.3 "/usr/bin/dumb-init …" 13 months ago Up 8 days (healthy) 0.0.0.0:3101->3001/tcp, :::3101->3001/tcp
uptime-kuma-1233b
0d276b305523 louislam/uptime-kuma:1 "/usr/bin/dumb-init …" 18 months ago Up 8 days (healthy) 0.0.0.0:3001->3001/tcp, :::3001->3001/tcp
uptime-kuma
Just to be safe, I paused after docker stop below to check the site was still up (just in case I made a mistake)
builder@builder-T100:~$ docker stop 0d276b305523
0d276b305523
builder@builder-T100:~$ docker rm 0d276b305523
0d276b305523
Rundeck is much the same way. I have a containerized instance in Kubernetes, but I also have a pass-forward instance
$ kubectl get ingress rundeckingress -o yaml | grep -C 3 service:
http:
paths:
- backend:
service:
name: rundeck-external-ip
port:
number: 80
That is kicking out to 5440
$ kubectl get svc rundeck-external-ip -o yaml | grep -C 3 'protocol: TCP'
ports:
- name: rundeckp
port: 80
protocol: TCP
targetPort: 5440
sessionAffinity: None
type: ClusterIP
As before, it would appear I have an upgraded instance serving traffic and the prior still running unneccesarily:
builder@builder-T100:~$ docker ps | grep run
6a8c46e9bfb9 rundeck/rundeck:4.17.1 "/tini -- docker-lib…" 13 months ago Up 8 days 0.0.0.0:5440->4440/tcp, :::5440->4440/tcp
rundeck3
0c89872282f5 rundeck/rundeck:3.4.6 "/tini -- docker-lib…" 18 months ago Up 8 days 0.0.0.0:4440->4440/tcp, :::4440->4440/tcp
rundeck
As before, I’ll clear them out
builder@builder-T100:~$ docker stop 0c89872282f5
0c89872282f5
builder@builder-T100:~$ docker rm 0c89872282f5
0c89872282f5
Lastly, there is at least one service I’m just not using in any capacity. That is Darktable which is a lightroom/photography app that i just beyond my needs so I’ll take that out.
I double checked I’m not serving it already
$ kubectl get svc -A | grep -i dark
$ kubectl get ingress -A | grep -i dark
$
Then removed it
builder@builder-T100:~$ docker stop 1389df4117a8
1389df4117a8
builder@builder-T100:~$ docker rm 1389df4117a8
1389df4117a8
While I’ll want to watch it over the next few days, I can already see quite drop in memory usage
At 3 AM
So.. at 3:45a I get woken from my slumber by PagerDuty
Uh oh. I think i forgot something. The next morning after breakfast, I looked into the error. It would see Datadog wanted to trigger a “status” endpoint and failed
I checked the Ingress and it looks to forward that to a cachet page
$ kubectl get ingress status -n cachet -o yaml | grep -C 5 service:
rules:
- host: status.freshbrewed.science
http:
paths:
- backend:
service:
name: cachet
port:
number: 80
path: /
pathType: ImplementationSpecific
However, this seems like an actual k8s service with the selector defined
$ kubectl get svc cachet -n cachet -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"cachet","namespace":"cachet"},"spec":{"ports":[{"port":80,"protocol":"TCP","targetPort":8000}],"selector":{"app":"cachet"}}}
creationTimestamp: "2024-07-30T00:48:36Z"
name: cachet
namespace: cachet
resourceVersion: "23300282"
uid: 4705b6df-8d98-4271-8baf-5df3e36f6022
spec:
clusterIP: 10.43.49.74
clusterIPs:
- 10.43.49.74
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- port: 80
protocol: TCP
targetPort: 8000
selector:
app: cachet
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
And I confirm that by checking endpoints and matching pods
$ kubectl get pods -l app=cachet -n cachet
NAME READY STATUS RESTARTS AGE
cachet-54995dbd64-czb2j 1/1 Running 0 17d
$ kubectl get endpoints -n cachet
NAME ENDPOINTS AGE
cachet 10.42.2.170:8000 119d
postgres 10.42.3.172:5432 119d
My hosted Cachet looks just fine
Interestingly enough, when I dug, the error was really Datadog trying to tell me my nightly build failed
And indeed, this was caused by a typo in my webhook (but the webhook to post to BlueSky)
Note: I cover this in the BlueSky post but if curious
curl -X POST "$PDSHOST/xrpc/com.atproto.repo.createRecord" -H "Authorization: Bearer $ACCESSJWT" -H "Content-Type: application/json" -2 @bsky.payload.json
# should be (d instead of 2)
curl -X POST "$PDSHOST/xrpc/com.atproto.repo.createRecord" -H "Authorization: Bearer $ACCESSJWT" -H "Content-Type: application/json" -d @bsky.payload.json
Updates and Upgrades
It’s wise to occasionally reboot systems - especially if running on consumer hardware.
Let’s start by just looking at the test cluster
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
builder-macbookpro2 NotReady <none> 219d v1.27.6+k3s1 192.168.1.159 <none> Ubuntu 20.04.4 LTS 5.15.0-105-generic containerd://1.7.6-k3s1.27
isaac-macbookpro Ready <none> 219d v1.27.6+k3s1 192.168.1.206 <none> Ubuntu 20.04.3 LTS 5.15.0-119-generic containerd://1.7.6-k3s1.27
anna-macbookair Ready control-plane,master 219d v1.27.6+k3s1 192.168.1.13 <none> Ubuntu 22.04.4 LTS 6.5.0-28-generic containerd://1.7.6-k3s1.27
As I can see one is already in the “NotReady” state, let’s start there
I might have to phsically go over and reboot it
$ ssh builder@192.168.1.159
ssh: connect to host 192.168.1.159 port 22: Connection refused
The other host was slow but alive
$ ssh builder@192.168.1.206
The authenticity of host '192.168.1.206 (192.168.1.206)' can't be established.
ED25519 key fingerprint is SHA256:EO/rzmw+J/mJTRIJXpkz2i/gZwibMO5bQyKvbVT6ts4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.206' (ED25519) to the list of known hosts.
builder@192.168.1.206's password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.15.0-119-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
180 updates can be applied immediately.
To see these additional updates run: apt list --upgradable
New release '22.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Your Hardware Enablement Stack (HWE) is supported until April 2025.
*** System restart required ***
Last login: Tue Sep 10 18:35:44 2024 from 192.168.1.160
I’ll then do an apt update and apt upgrade
builder@isaac-MacBookPro:~$ sudo apt update && sudo apt upgrade
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,054 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]
Hit:6 https://download.docker.com/linux/ubuntu focal InRelease
Hit:7 https://packages.cloud.google.com/apt gcsfuse-focal InRelease
Hit:8 https://packages.cloud.google.com/apt cloud-sdk InRelease
Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 DEP-11 Metadata [65.2 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,678 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 DEP-11 Metadata [212
... snip ...
It was taking a while so I fired off another update on the primary node
When things were wrapped, i rebooted
builder@anna-MacBookAir:~$ sudo reboot now
Connection to 192.168.1.13 closed by remote host.
Connection to 192.168.1.13 closed.
By the time I got home, I investigated the slow server - one of my old 2012 Macbook Pros.
I noticed it had a wee bit of a problem
I cracked open the laptop to see how bad it was
I pulled out the battery. It had inflated so much that it literally popped a screw out of the back. It was(is) nearly twice the thickness it should be.
Then put back on the bottom and booted back up
And it did boot up
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
isaac-macbookpro Ready <none> 219d v1.27.6+k3s1
anna-macbookair Ready control-plane,master 219d v1.27.6+k3s1
builder-macbookpro2 Ready <none> 219d v1.27.6+k3s1
I ran updates till all my nodes showed clean
builder@isaac-MacBookPro:~$ sudo apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]
Hit:4 https://download.docker.com/linux/ubuntu focal InRelease
Hit:5 https://packages.cloud.google.com/apt gcsfuse-focal InRelease
Hit:6 https://packages.cloud.google.com/apt cloud-sdk InRelease
Get:7 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,054 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,678 kB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 DEP-11 Metadata [65.2 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 DEP-11 Metadata [212 B]
Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 DEP-11 Metadata [159 kB]
Get:13 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 DEP-11 Metadata [940 B]
Get:14 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 DEP-11 Metadata [276 kB]
Get:15 http://us.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 DEP-11 Metadata [212 B]
Get:16 http://us.archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [810 kB]
Get:17 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,239 kB]
Get:18 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 DEP-11 Metadata [446 kB]
Get:19 http://us.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 DEP-11 Metadata [940 B]
Get:20 http://us.archive.ubuntu.com/ubuntu focal-backports/main amd64 DEP-11 Metadata [7,976 B]
Get:21 http://us.archive.ubuntu.com/ubuntu focal-backports/restricted amd64 DEP-11 Metadata [216 B]
Get:22 http://us.archive.ubuntu.com/ubuntu focal-backports/universe amd64 DEP-11 Metadata [30.5 kB]
Get:23 http://us.archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 DEP-11 Metadata [212 B]
Fetched 8,151 kB in 8s (1,018 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'https://packages.cloud.google.com/apt gcsfuse-focal InRelease' doesn't support architecture 'i386'
and
builder@builder-MacBookPro2:~$ sudo apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]
Hit:5 https://packages.cloud.google.com/apt gcsfuse-focal InRelease
Hit:6 https://packages.cloud.google.com/apt cloud-sdk InRelease
Get:7 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,678 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 DEP-11 Metadata [65.2 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 DEP-11 Metadata [212 B]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 DEP-11 Metadata [159 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 DEP-11 Metadata [940 B]
Get:12 http://us.archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,054 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 DEP-11 Metadata [276 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 DEP-11 Metadata [212 B]
Get:15 http://us.archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [810 kB]
Get:16 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,239 kB]
Get:17 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 DEP-11 Metadata [446 kB]
Get:18 http://us.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 DEP-11 Metadata [940 B]
Get:19 http://us.archive.ubuntu.com/ubuntu focal-backports/main amd64 DEP-11 Metadata [7,976 B]
Get:20 http://us.archive.ubuntu.com/ubuntu focal-backports/restricted amd64 DEP-11 Metadata [216 B]
Get:21 http://us.archive.ubuntu.com/ubuntu focal-backports/universe amd64 DEP-11 Metadata [30.5 kB]
Get:22 http://us.archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 DEP-11 Metadata [212 B]
Fetched 8,151 kB in 3s (2,979 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'https://packages.cloud.google.com/apt gcsfuse-focal InRelease' doesn't support architecture 'i386'
Beszel on Hosts
Let’s attempt to monitor a Kubernetes node with Beszel.
I’ll “+ Add System” then give it the details
I’ll copy the docker compose to clipboard the click Add system
I then SSH’ed to the host and setup the Docker compose from my clipboard contents
builder@builder-MacBookPro2:~$ mkdir beszel
builder@builder-MacBookPro2:~$ cd beszel/
builder@builder-MacBookPro2:~/beszel$ vi docker-compose.yaml
builder@builder-MacBookPro2:~/beszel$ cat docker-compose.yaml
services:
beszel-agent:
image: "henrygd/beszel-agent"
container_name: "beszel-agent"
restart: unless-stopped
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
# monitor other disks / partitions by mounting a folder in /extra-filesystems
# - /mnt/disk1/.beszel:/extra-filesystems/disk1:ro
environment:
PORT: 45876
KEY: "ssh-ed25519 asdfasdfasdfsadfasfdasdfasdfasdfasdfasdfasdfasdfasdfasfdasdfasd"
# FILESYSTEM: /dev/sda1 # override the root partition / device for disk I/O stats
This host seems to be lacking docker
builder@builder-MacBookPro2:~/beszel$ docker compose up -d
Command 'docker' not found, but can be installed with:
sudo snap install docker # version 27.2.0, or
sudo apt install docker.io # version 24.0.7-0ubuntu2~20.04.1
See 'snap info docker' for additional versions.
builder@builder-MacBookPro2:~/beszel$ which docker
builder@builder-MacBookPro2:~/beszel$ which podman
builder@builder-MacBookPro2:~/beszel$
In most cases, I would use podman
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:projectatomic/ppa
sudo apt update
sudo apt install -y podman
But it does not support Ubuntu 20.04 (focal) anymore
$ sudo apt install -y podman
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package podman
I’ll do docker instead and then launch docker compose
builder@builder-MacBookPro2:~/beszel$ sudo docker compose up -d
[+] Running 2/2
✔ beszel-agent Pulled 3.6s
✔ 397a8890c205 Pull complete 1.2s
[+] Running 1/1
✔ Container beszel-agent Started
I can now see it showing details
Let’s be clear, I’m only going to get Docker and Host stats, not Kubernetes.
That is, I can see quite a lot of containerd processes hosting containerized workloads on this host
But Beszel will just show its own container and host stats
But for Kubernetes stats, that’s where Datadog comes in
Docker Diskspace
There is one glaring bit of red on our Beszel dashboard. Our disk usage is over 90% on the Dockerhost
And it’s pretty clear that unmitigated we are just weeks away from using up the full volume
My first step is to just get on the host and start using du to find fat folders
builder@builder-T100:~$ sudo du -chs /*
0 /bin
278M /boot
4.0K /cdrom
16K /dev
32M /etc
93G /home
0 /lib
0 /lib32
0 /lib64
0 /libx32
16K /lost+found
4.0K /media
5.0G /mnt
2.9G /opt
du: cannot access '/proc/4085867/task/4085867/fd/4': No such file or directory
du: cannot access '/proc/4085867/task/4085867/fdinfo/4': No such file or directory
du: cannot access '/proc/4085867/fd/3': No such file or directory
du: cannot access '/proc/4085867/fdinfo/3': No such file or directory
0 /proc
49M /root
4.5M /run
0 /sbin
9.4G /snap
4.0K /srv
2.1G /swapfile
0 /sys
296K /tmp
8.2G /usr
104G /var
224G total
Seems the big waste must be in /home and /var
I have a lot of backups loading up the nasilot (NAS) folder
builder@builder-T100:~/sirnasilot-backups$ ls -ltrah
total 56G
-rw-rw-r-- 1 builder builder 1.2G Oct 22 03:35 sirnasilot_backup_2024-10-22_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.2G Oct 23 03:34 sirnasilot_backup_2024-10-23_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.3G Oct 24 03:35 sirnasilot_backup_2024-10-24_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.3G Oct 25 03:35 sirnasilot_backup_2024-10-25_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.3G Oct 26 03:35 sirnasilot_backup_2024-10-26_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.3G Oct 27 03:35 sirnasilot_backup_2024-10-27_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.3G Oct 28 03:35 sirnasilot_backup_2024-10-28_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.3G Oct 29 03:35 sirnasilot_backup_2024-10-29_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.3G Oct 30 03:35 sirnasilot_backup_2024-10-30_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.4G Oct 31 03:35 sirnasilot_backup_2024-10-31_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.4G Nov 1 03:35 sirnasilot_backup_2024-11-01_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.5G Nov 2 03:35 sirnasilot_backup_2024-11-02_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.5G Nov 3 03:35 sirnasilot_backup_2024-11-03_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.5G Nov 4 03:35 sirnasilot_backup_2024-11-04_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.5G Nov 5 03:35 sirnasilot_backup_2024-11-05_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 6 03:35 sirnasilot_backup_2024-11-06_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 7 03:36 sirnasilot_backup_2024-11-07_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 8 03:35 sirnasilot_backup_2024-11-08_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 9 03:35 sirnasilot_backup_2024-11-09_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 10 03:35 sirnasilot_backup_2024-11-10_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 11 03:35 sirnasilot_backup_2024-11-11_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 12 03:35 sirnasilot_backup_2024-11-12_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 13 03:35 sirnasilot_backup_2024-11-13_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 14 03:36 sirnasilot_backup_2024-11-14_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 15 03:36 sirnasilot_backup_2024-11-15_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 16 03:35 sirnasilot_backup_2024-11-16_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 17 03:36 sirnasilot_backup_2024-11-17_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 18 03:35 sirnasilot_backup_2024-11-18_03h33m.tgz
drwxr-xr-x 73 builder builder 4.0K Nov 18 08:24 ..
-rw-rw-r-- 1 builder builder 1.7G Nov 19 03:35 sirnasilot_backup_2024-11-19_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 20 03:36 sirnasilot_backup_2024-11-20_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 21 03:36 sirnasilot_backup_2024-11-21_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 22 03:35 sirnasilot_backup_2024-11-22_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 23 03:35 sirnasilot_backup_2024-11-23_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 24 03:35 sirnasilot_backup_2024-11-24_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 25 03:35 sirnasilot_backup_2024-11-25_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 26 03:35 sirnasilot_backup_2024-11-26_03h33m.tgz
drwxrwxrwx 2 builder builder 16K Nov 27 03:33 .
-rw-rw-r-- 1 builder builder 1.7G Nov 27 03:35 sirnasilot_backup_2024-11-27_03h33m.tgz
A simple find and xargs should do it…
builder@builder-T100:~/sirnasilot-backups$ find /home/builder/sirnasilot-backups -type f -printf '%T@ %p\n' | sort -n | head -n -10 | cut -d' ' -f2- | xargs rm -f
builder@builder-T100:~/sirnasilot-backups$ ls -ltrah
total 17G
-rw-rw-r-- 1 builder builder 1.7G Nov 18 03:35 sirnasilot_backup_2024-11-18_03h33m.tgz
drwxr-xr-x 73 builder builder 4.0K Nov 18 08:24 ..
-rw-rw-r-- 1 builder builder 1.7G Nov 19 03:35 sirnasilot_backup_2024-11-19_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 20 03:36 sirnasilot_backup_2024-11-20_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 21 03:36 sirnasilot_backup_2024-11-21_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 22 03:35 sirnasilot_backup_2024-11-22_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 23 03:35 sirnasilot_backup_2024-11-23_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 24 03:35 sirnasilot_backup_2024-11-24_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 25 03:35 sirnasilot_backup_2024-11-25_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 26 03:35 sirnasilot_backup_2024-11-26_03h33m.tgz
-rw-rw-r-- 1 builder builder 1.7G Nov 27 03:35 sirnasilot_backup_2024-11-27_03h33m.tgz
drwxrwxrwx 2 builder builder 16K Nov 27 07:06 .
I’ll then add that to a nightly crontab
builder@builder-T100:~/sirnasilot-backups$ sudo crontab -e
crontab: installing new crontab
builder@builder-T100:~/sirnasilot-backups$ sudo crontab -l
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
30 1 * * 3 tar -zcvf /mnt/myfbsfuseblob/vaultbackups.$(date '+\%Y-\%m-\%d_\%Hh\%Mm').tgz /home/builder/vaultwarden/data
30 1 * * 3 tar -zcvf /mnt/myfbsfuseblob/filegator.$(date '+\%Y-\%m-\%d_\%Hh\%Mm').tgz /home/builder/filegator/repository2
10 1 * * * find /home/builder/sirnasilot-backups -type f -printf '%T@ %p\n' | sort -n | head -n -10 | cut -d' ' -f2- | xargs rm -f
The other bit of waste comes from Docker images (consequence of testing MANY MANY things on here)
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 108 59 66.06GB 42.69GB (64%)
Containers 70 17 1.28GB 640.7MB (50%)
Local Volumes 86 21 6.063GB 4.69GB (77%)
Build Cache 313 0 1.042GB 1.042GB
I can remove some unused images
builder@builder-T100:~/sirnasilot-backups$ docker image prune
WARNING! This will remove all dangling images.
Are you sure you want to continue? [y/N] Y
Deleted Images:
untagged: harbor.freshbrewed.science/freshbrewedprivate/mytest@sha256:54841e85a2e279c386483f89d32fa7b1ec1916cb2095aa683cc733b4cb512d63
deleted: sha256:1c561921fd9c71922151b9a102c719adc3e613a8626edf61af8debfd28431a28
deleted: sha256:d295d1486be32aa8f94b89f5effd49b39de976eb4d1f284b64319134d5b848b7
deleted: sha256:2573e0d8158209ed54ab25c87bcdcb00bd3d2539246960a3d592a1c599d70465
untagged: tensorchord/pgvecto-rs@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
deleted: sha256:2e2cb40c55b8b0f0c6ea23c5f505aee3a3bf6e14859178a5a2593b856d93f3a1
deleted: sha256:e433723c58b947dadadc46c7f9fad77c6e537be6afed3a08126c21e7f68219b7
deleted: sha256:da2a970bf5f6e547fbeeacbc84f1bc840597d9325ddb0bbb036031f28e0ff3df
deleted: sha256:6cc53aef4162a720da0b8fef9f0fd9650cff610b2f5af9169d9695f17fb6e8d2
deleted: sha256:c2da3f72fc1c61a6f4da836b23582d41d1661bc14da3184bc8617f2c135b9e0c
deleted: sha256:baaf27fe745398ce9630cd0f422bde308789f043c8a3c0cabad67917fae967f7
deleted: sha256:8c61978592ee874e2f241d8e99d6739810d60dd3cd36aedbbf0eae5db0f28627
deleted: sha256:89845b18648063a1aed6f880a1f4a086d7aac10f50ed78f52f4c2ebcb24f417b
deleted: sha256:ce18fb87a32d0854fad64e07787eb6fdea08ecabce82aa8c2f14db9f5f3d4b48
deleted: sha256:15f0a49f0bf50ee8385f3b6dfcba98e1edd289716f297d96c083a1291156bec6
deleted: sha256:da5e3acaca489c690768226d734cb5d7326f0acfd6e50909a58d6a901d095848
deleted: sha256:d9b03f3561cc415b10e91dc512a4b792ad31a896b0346da97f58b6bf0b3886d2
deleted: sha256:d5bae171e77d0b4f7b8091bcf42b945c9f4a5fe878681a249db834e65df4456f
deleted: sha256:bb0b33f8d53b428bb2d339a40f8eb2449d63b2dd5c66c805dd3630a5a0cc50cd
deleted: sha256:5e8b91515b053ccc4d3cf1fc79f6c67f9714303d7122f75031f4e4ae5eae77fb
deleted: sha256:216103ce9ec50b4cbf91226b4057b03c1dc35fa2156d5118850b6829b662fd11
untagged: redis@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
deleted: sha256:7de0dedd123b8cb2b105ace4e8d00b8bba5ad7be39617dfa229acff315fe4fbf
deleted: sha256:37e3616051976e621859857d9a4998068c312dc0e9625e9313bece50153e2e51
deleted: sha256:3b14e7f9a58a961a8b66813a298d3c8c648d94e5c303af53918ad56441d252d4
deleted: sha256:fc595f45aa80f596429753557bef5a545d18b74a0f277fbdc1fd756a14de8776
deleted: sha256:bb806e616e5e24a2b85882ba54a9cb1a5a5ae7e132e55f38e733fbc67a92ab01
deleted: sha256:cc160d8560c7c780fd8c4660d7ed5491b08878581d87ea4e7f9af355872b468f
deleted: sha256:11865b81f98ca2870b5b757b9687e340641670e9be8389615b9f26ea482ad43a
deleted: sha256:44369eb44239a1fa79f759aa958f7ceb764d851f060e66bb7ad4db8427be317a
Total reclaimed space: 701.1MB
I can also remove stopped containers
builder@builder-T100:~/sirnasilot-backups$ docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
b1f25e2a243a76e5b419321ce7cd68dd42de21cef9ede50734fb3ebb6f58b3a3
07addf17dafcf8954bdfb46bc53221c0435e737798c699ca072e7d1aea571f80
0c8ca545e0359cda513f3babe845f5d26d435e270d682fdd630fdb2db6e76b6d
7bac56d1c4977abb9d7754c15984378510fa6b0b010d079ed0e544043eeb42fc
1f0337971b9f800d2d1515cfc3b84c6d59ebd9ad82c707047e933a691539720b
810a2c05f761fddb0b14d4868e8b47dda259e2119e320cb69263ee51d417fb96
66fc44ec76fa023250fb1ff8c6ef96d72ea9f6f3b25097b3550a34d7e827186b
44345e8c762fce7619f26a0bf6e5d29ba482eb2f17b7e565d02f5865b9e0037d
ba18e7b167648138f6bf2643d232a0474b57b95bbb21730fb1f7a45bbafbbd46
c48638b81ff2d726bbefa5944d612c11081c6eb4f9d6dd69344914f33366b074
42f769f39aa30b449e3b8eb7b0397e8592a1a0a13f78dcdb34f5a11cdc5a1738
a471303c8093d8f8b13ea2f222e3b928d16c42cb77c3af5e47151f8a871f8bcd
4f683a96127697b0e5ae6c44ab69d2835be80bdcf0ed67a68d25625d7c066044
68b08b8d624bc364d6f3d26db3a6f44fcf103f01d1960ff6dc5a1fcb7e0d7abd
4e8353c6e5143cd8b623fed148934eb5c18872d71aae11875e9f3c38f0afea85
0c3076158195f15cfd41f03adc38a78530a965493625489557a408ce083e790e
0b2f15589778b595b741d6d9d23e75c671424da9c28ea5536346e9bdeaf81664
c194e587da2e59a17e4e2b0f8bad263217a9bb87808be3852e2bed1769f74062
4437fe5178a0d23af9012ce033e66912f5710add031ee3f6862120ce5767af24
c5654c05d2fabc18211c7ea37fa00ff7a54a57fb0c3450aa68e8fe8da8fcda1b
c0c81ce72625b8b5fa9c9de62f59ef269f4ca179d7f16cda00ff740e810550f3
708938da0e0f9e2c4a9ca14e93539a03356ae41a70bd59390f8841d52473202a
84c578380db99e5264d2d0d70e6bdf4bf4c144949260a2561b1bd68468b0ea3b
04e2c7fe5a55affa4398feda699768d819daaa1f0ba061ad412158e2effa56cc
93f1a3e3aea90cf5bc886728c84bce712cf943950563659eef2117fdbbcab916
6785b5c3bb3f7d1c5922d5e9d56cb24a55cdb892e32f9d1574b4afda59b7c5f7
225ec2abf2c74d97a30734f162fa5320fd18e59c723d01161ec4aaa2bf3d685d
c1f96fc0c41d61dc439e83c2c5fae2d3b5ce6e79768a8274f5505d33be5fbc68
ab1260db90f2cac06e9960195fbacc9bc47b7c6ee1ad259584967f1193600a31
b5afeda7083a8e590939734236c9bf99125fbf71aadb800d91ca5ead316fd05c
10cba48d94efb716b99fcfefb640f4ba854cea71c6313a25efa530c45abbb5a0
23bf94a2c8e0aea6bbdfa675f37fd739f60185aa0c250b12085d581811542b65
78c055877ee356bf2591919a9f53ba5cbbe54874aa3daefa411875a3b50023a6
52c0289c7c735a1e269ad622af3bd67596f5dbdeb286168299933a50c00db627
acd69511ada9c75cafec80425c418b6cdb0657cdeb0ad3132df1c596e05bff66
3acee50fe7d984bf5234a47eeb6335f15ae9677ce4af5fe3a1e7f262312d0310
5abdc928e11dd96a83662ea2735b83673df1d66532829f94def97b79b477e8ce
28f1ffa62653776aa53cc52dad4d99bb5823e124ebf0fc5b058f1e86a30870e8
33dd87f329d7588410b43c043cbde66dcdba2691920e5c6e41cfe03daf773d75
9a4794b5653f672c7521f04138d3309f92aba8fb16a3b6c69865e800c52931e9
9ee0478f5ada33d355e690ab2e7c5dd7d324f373e998c27cc3e27737a748e1c9
3c3f3bea3fdbcbf7f2a1c09f5ebd08d1c469522687ad23da78b9099a4b52eb7c
0f1371ef3f3e8f7448acdd6f7f962e664143b4341556d0632c2c196f31f34ad6
5d8a6d764e2703e25e422c4620de35a13b71445af731f2f14063f10a399b8355
30b3d3701e35c908111ee551050a8966a5e5f43765c8abebc757d0a034b95a8f
e06fe95713409877fa4ed98d9e55ceda0ddc411c43bab1b87b0df70c57b2f347
c47891efdb53e427bc660e48123d7006649c188040d45f2a0718f7d4fb1defd4
67e13b29fd43c7cee1e10cc2e99e9f5cfb30e3042e9ba152348b00f09ed5343a
34903c6058ca30d05bfa098e2c8d333c6041c8db19fad8324d27b502b3e9ef91
1f3ada372a43267476d137f18d3a6641ff806c6a73d21454da718a09e24b6f2b
26e4f801ed151eb01d0b18d49b1da0682a43abccf4d6c16b7c69bee877eeadce
8ed93813eeca2dc17205825623fcc0fd8af07b62c771acc0fc309a4356af8ca3
767c9377f238147fa85e1f1c7d194e22fd1eb16b1b1988c0537711da3e41ef7a
Total reclaimed space: 640.7MB
Lastly, removing dangling volumes
builder@builder-T100:~/sirnasilot-backups$ docker volume prune
WARNING! This will remove anonymous local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
0b280fb769993331ad69772fc0eb906a7a0f0735570216e6739fb1cad51f4fed
9070f8622521782ccd490e9c8e32bbdca0273bcd3d6662baad08bed440ccd73b
a6f481ae43036fbf0297a85795a1a7b5c3ed8cc27b9f6baf565204bc6f3de4cf
b81f808a24372d7aac968831f5ff72d3bf0cbdc94eedb62c5a19651c0bbcfb6a
c210586454983e2e6e4485d76e514e989883fef2136c082a16934fae757a28c6
f8ded16deff2c0d21ddce2fe3c2dd7d7e5d6fac308de1b81e25727731548e00b
10268fadf7bbdd54f36b6775ef58a8a0ac731a5ad7c36f2ed2de19a817e34e37
40f383bd86d26fb6b4e86feda48889411ca7dbafa9ae5f01336fe69eb959a4be
4674e9d30e88f1aa3de097e4b4d7f91289eac29baf331da0a83aed0eb9daac1b
89f5e078bf41c3d39614393d200b0bb512fd8992df38de39d96e181483331539
ab1367681820537c003fa85e1d6231e1fbfe085975ef213f97d95168ec832a8a
f8cb037aef5c7f0b3386fa65f7413868cc2238331460c2e4a0c0b1717167f93c
0d3742de58b2cf4725de5521cb5060dbe4a5867d297213e8e3f51f9c38f467b4
e888097091aa5f1e8c9d442d2ab3efa954009dde5ae877febed73ceda3f7c0bc
ea42f9864d7a9d50095a4b781a7170a3d42f3cb9ab183b646f21834d109c6be1
62747bf0443259c2afe432fb2e4a34daa0c3ed5a58af3c1cb6d5f5a49a19e142
8b214373b5fdaf1403f765691e6e9e8eaca509e8441ac4f6c8f8b4ebecadf0ee
f053eaaa7ccee9941459c4175eb461fd05a592cf7ebff88d04a20bec00042ef0
4ce96f4bd7895cf207f3a10a98da88e1cc89d11e983e35952f93a7e1fd8dae44
dc16d149ba0b306a615f5dfd0070f3ddd7260a78f376ff37c697b41c9ab81280
e738539596f18d392b909f4a35a70de16af433cbcfe3d2902c33ac436993aadf
352d0738b1d1bb5b2056e9f719b095a75cc5bebbd8629a3ba728fdcd7dfa0992
63f03917853fa3ddf9be82eeab060676ecc455a0bc211e8f3054d5df3c2d666d
9e52f84554cdd94e2547489cc5fd408930c2f8c4585a586872eb5e01fe3cdea9
421e04e1ed294d66dc9d15f31c557762b0bb6dba6d59acfe79727dab946fb709
aecac79bf70c541f30b33ed48841e250e2e98d0e8f18975bfeddda71d3ca1ee3
b24a79f7cc8507956303c40bf5dd657fe949cfff27d7bb6b4d69b91b76dff9d6
873838615f334d4f7759c5f434376a5167f27e910b98558f27417b4c6adfb53b
b4cd32abbe1e0ddfba0ec79f36c9e05ddfd7955ec4f2266927c14d0a01d4e17c
0fb8b3c522a4f99be7155c0de23d803522f8b666612d1aafb78e76b44fa4a043
aa750d749f4fe43d2290324ca560547df2fb56ba9a5bb32ff5147126f0f12b31
bbc0dee8a227a0bda1cf47d0dd8c60c701e7da12ca3645d1731e41bc7e24b12b
f3f35bc1072751a9bfa5cf0c78c4496a13b026083229eff96fd5f49a3c46667b
1cc439c475f5d6c23ba6f759d0028cfefe7c73dd1833dc9e9ed4449296d8c3cf
2794a8d7a447336c1019d767d53ba8fba7720c35bcc3dff4eaaff1a889281d41
1c0c96d30d9990fd759786b10397dca1cec827353758d257ef19b57840f55744
c474612226ff306e4e20d9b1b8043a94891bec091292b5ed15df7fa0446da667
efaf50747ee9282431898a3fd5875740b0938226b440eb63557d3ede29ffc7a6
18366ff68136c7be50fbfefd68e9a108c2d67f339b7bee6b2ea5a584e56fd26b
2110463f2ad82dd45fe89bc4f2c0e125db95e788552a06b84096f710db4a1bbc
dbe0eb1707db65750622720fa1910f699cd38406644b904dd123a4abd463e431
e62226e65cfec37e017ae7c1875a269614c348fd642cd8b43afb30d8a00aa084
4e5a5c25640a9a710ee5a18eaad700105d697b7b8bebfec25e121d5703600bbc
d9f0f47d6ae9708b96cf05d035fc93f1c4962e954334be803d4267b03990961f
Total reclaimed space: 3.984GB
Now, to really reclaim space, I’ll remove unused images including ‘non-dangling’:
builder@builder-T100:~/sirnasilot-backups$ sudo docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
This just means some old containers would need to be re-fetched if I redo those items
builder@builder-T100:~/sirnasilot-backups$ sudo docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: plane-web:latest
deleted: sha256:970046198c0465a7fbfa27b3c0940f41a6f4bcc3d6fe128c18f5896bf8513121
untagged: 1dev/server:latest
untagged: 1dev/server@sha256:e37c8bf9902cb643bbb2f6bda698290204f8297149d49f8630b749c63fa89aee
deleted: sha256:e06e8f319a1f8381bb03fb75534b3ad29c0ce2b7659181fd2a5c4524b6ef6437
deleted: sha256:b103dc28f0f3613c70776d85e7afd6f655f6144ded668f7e1d9ba4f7b3e97bf2
deleted: sha256:0f99a449bea525d8d4c18f7d66b486978d7fcc3ce6a57db9dbe690d06e6c019c
deleted: sha256:ab441a93155fe40570a32bf29426e5cc9fbea532383d13b4afdada15df003970
deleted: sha256:0dd2318bcfa7d70229ad8959a5e6a49c09c460054ff35d6e2f002a0f5140ae08
deleted: sha256:ddc7817373486ee0d450bac2a0432d138dcb7873d0e1557e3672f1a4beea424a
deleted: sha256:b9e4112a94b8531c91b5de4f91330c30bbb788b142a51900e65060b6d85611f6
deleted: sha256:6ec8f20deba66a025bb9170d420887a33fd01b427c89a9fc2950ed877d67f3fb
untagged: noisedash/noisedash:latest
untagged: noisedash/noisedash@sha256:7aab65f9c467a1c036f6ec8510f781afdd42af8b2acaf349ff4c41cebf78c945
deleted: sha256:5482f494bf4e04cccc25f08147e310d2394922d5318669011efbdb0b8ff5bd41
deleted: sha256:9ba46f3b29e9ac9d038ba6d3e660c5f54e1027ae918d77d850efa3958d4a7c52
deleted: sha256:c5841a710abfac0495cb4433beff9579d21dc6441ae9ad31f27419dd08d59341
deleted: sha256:6cfaf9154d0c35b1031c5b620841856debb8ab069fbdd83f6348845e433b2c3c
deleted: sha256:393ab4c1629bfd41ba847f7aee55272268f846de229e31d2fcff00d9809ff043
deleted: sha256:0bf6bfceb11a09236db71ed4bd2db86d45ea9987ef0cd933dedd502e6b89d974
deleted: sha256:0322e47fe75b740cbf986b1750c334107718ab6d73bdbfedd71157cceccee639
deleted: sha256:da4a3a907432cf27b923ea845c1ed7560f9ce3dd23bc15bed71379e9d2b60eda
deleted: sha256:7d85e5185a181f323d84f04aea9584abb9049ae9e5dd43fab9974dfc3ce5dd89
deleted: sha256:eb298abd8219fcc019b01fa0cc7f57f925f7a4bfa904cabc9dc1de82d9d5ac7b
deleted: sha256:e384bd3e36d0d7d0e7ff836d2334856424095ef1874b02c092ed72edd07ab77e
deleted: sha256:c209cc51a67ea34ac3d3157f02e32a92020bf5b74a9d739f3fd15ee4b0ba675c
deleted: sha256:797b420c580e87b9747eaf2f4a0c545b56f0ae435ee01a4cda45da8e46840c70
deleted: sha256:9af5f53e8f62b46127a1e27488a8d501013df31fbb21d81fbce4cc55654fa910
untagged: harbor.freshbrewed.science/freshbrewedprivate/fluffychat:0.0.6
untagged: harbor.freshbrewed.science/freshbrewedprivate/fluffychat@sha256:686cf5ec4c595080e7ce76171c881c4d209cbc8a0033c99b630555d964d6670d
deleted: sha256:2ffddfa31486c2ad35427e33a9c962a10b39e5211305b6a1de4f506d08f5b714
deleted: sha256:465e24b30800ecf65a1a4b543146e0ecbc5775d525cc042716e218776956c916
deleted: sha256:2b0be095d1c9bb3ac7b2b1bad23951dd348d7999cc78d118fbe2a222ee942349
untagged: supabase/postgres-meta:v0.68.0
untagged: supabase/postgres-meta@sha256:31a107dcfe9257792b49f560a5527d5fbd7128b986acad5431b269bac4d17f12
deleted: sha256:16851ad99c8f8865c2fe41a6718576a5c1ec95846ef96c667e3d49c5fcef4039
deleted: sha256:7bc3f7f11cfa4e9264e38520c1d6426d055b446da246aa0fda3c8e163f312a27
deleted: sha256:b9386354510135f1ac57cf7a1c30a8da4a3efadf89b8c77d2965c868a1e0a83c
deleted: sha256:c1179fffaac4c978fa2f4ef1fdef3c7d5c775193c3a4f6c6b3f5baf09dade7f1
deleted: sha256:396c311a7ab6266c5d4d8a22a31eb5c75f5eadcbbd49da66f7380cab37ef9814
deleted: sha256:c41bf149753eea7020be6d0421ee3a04e92149291ddbfeb42e3a02cac79973a4
deleted: sha256:dae2ca211f51d5b914a6bc4e57cd0325b4843631dae450d868f427349e38c35a
deleted: sha256:d8fe0714f71f5ec75bed7ab0f8e59fa77cee7363f5da6e17d58b9f116a144b50
deleted: sha256:d7221edb54017e3660e01416386081799648441afc56aa9d040dbda11f22c464
deleted: sha256:8ce178ff9f343a37169f68dd0df03099524afb71a879551c5f17e493c7b1d3ec
untagged: neo4j:5.12
untagged: neo4j@sha256:211c4afab9a7b082980ab1eb9001361715f77ee178f7f090714422a4becb3c6e
deleted: sha256:61f7a2ea94a8fcefe9bba2810c226c0b16746112357ce44fce18c7971afee2e6
deleted: sha256:ad918ad58cbf4860db764768bd3cf5c3e6d68e2ba0ae342709bfd45f606b6749
deleted: sha256:7adbbf9ea7cc208f6308b2cdcdea82fbaab326395d97b4631cc20f33d981dca2
deleted: sha256:7751dc9ad80ee0f90169708736a9a08ff9defd1bc74733f19909760b4277f6b2
deleted: sha256:d110739629fb2bb16916ea5bb3423feff2fc391b68db6092206e0587307f987d
deleted: sha256:633f5bf471f7595b236a21e62dc60beef321db45916363a02ad5af02d794d497
untagged: postgres:16-alpine
untagged: postgres@sha256:36ed71227ae36305d26382657c0b96cbaf298427b3f1eaeb10d77a6dea3eec41
deleted: sha256:01100f48660a46a6c3fffb71f672da75347bb12f7b264dee381cf0e985d9e63c
deleted: sha256:9d425d348e579b74125e5d8e8d777e61d9052771b043311ba8ee3146c229ddd1
deleted: sha256:24a5e050a460c6f0ed4bed3b9a65255555d14bb45f27ea736b3ccb0c53530b39
deleted: sha256:d628ade6005c7c547624d53d0a601cbdb44efa833ac02626becc120171c4b8b1
deleted: sha256:1682d54e929255b084825153d4f7cfc380b956cd57ae3fa83be5b12314241941
deleted: sha256:8fd64670e784aee0ec97c3f180d4a7fcf7e5d57bce82c71cc32976db02ac9c27
deleted: sha256:186f2cb6ca47639d104ee3971e56fdee9c6a4563a48d74586f5a34f6325109a7
deleted: sha256:770cd448b690d774f8d34c01fca75f402e4e0c874ed3ec526b068fc5db8e3050
deleted: sha256:ab30370fc44e5cfc9cadf154ed11558ff75eb76c3592e6ddd3c2ac1db58b50e9
deleted: sha256:cd623b0fac75891e87ffd619b5e63758d7971bc776a56b79c8b6d27e414321e2
deleted: sha256:f9a12c660106f34a92c5e5ec8d74f7182f715fa57127fb3c1706cedc6ed0983d
untagged: memcached:latest
untagged: memcached@sha256:4eadb0818735f04b96c1cdc855df43fda3cced8faeb85f6081cef6eaef07d4b6
deleted: sha256:e89b92b1e7ff13f14853ae2b16915de3b3afbf465b958005d848a01a249df57c
deleted: sha256:409e03a004a64d5b13dbd1a5086a0b86ccb82fc51013b516a97e4de2ef673e1e
deleted: sha256:da7436a6ced7a6736b6a423c2dd2de296d8654f18edd3fa02c1080cd5c17d82f
deleted: sha256:849bc228d11c459f247544e35e065ec545ce1b4dc75d13d34c789777fe69bcb1
deleted: sha256:1cd3f516bdf0ca63147e0c86d2a90b8af4ef9da7d15021e799f06a4734da54ae
deleted: sha256:62379107447e015b7f1a0c1c9c2f8a25622146c00730387ffc05e92ee40bb485
untagged: postgrest/postgrest:v11.2.0
untagged: postgrest/postgrest@sha256:9a38d9565ab92675e0ad5db8e2a24f8aa2d2f98accb07a517a61e598edcbc98e
deleted: sha256:152931ef82804bba0ae24295b5593612359994c1112d7cb02059f7bf5e7617e2
deleted: sha256:afc89f9f388136ed50c6bff865386ccffaf3f4a61fee12bf0f66a6fd13fb4070
untagged: mattermost/mattermost-preview:latest
untagged: mattermost/mattermost-preview@sha256:cc133a74724fc1e106d035ac1e6345ce1ba295faaca47612b634fba84f106ccb
deleted: sha256:2655dd55e4a015458ac5ce789a272abf29b56695ff09911ac882200478c0925e
deleted: sha256:21327efd675300179b015cee60867c8e1163730e938abecff8750501c3bbba57
deleted: sha256:162396cb010c4d926ea8f0a721b32631d21004424c52ca084909ba10be438d41
deleted: sha256:3ee2b2033b1abcaba4c0a307b920fd767bc2bd2a161c15fe4bb44612a14a705b
deleted: sha256:295ebbdf8064edc7d158d0f99b9473479e6fa16ed54d6d4260243003bab054ee
deleted: sha256:7d5d65af70b16191a0930435cba6019772f4cd3308318cef3e6bd3c66e567ab2
deleted: sha256:e6354bc112d66a60b80ea1a0597dfd70c9b43b7deb3039e02a0bbd85f4a9b806
deleted: sha256:8ea36ac1dfa92d08c9bf9bedd11e7abe878924ebe5d3238afb43a2665ff1b3f3
deleted: sha256:60eb0dc7c5858e9c4f1cab803a12e921add360469fe6a07f6b68ff06e1bed77c
deleted: sha256:52c83b575d200da99427f0a82e588a02de489ab62e9ebfb71f2eb0e42e318298
deleted: sha256:15e201bffa53cd39ad7f00c970fc2b9034089e9f4e067c90b32e5ee1554b3a58
deleted: sha256:2e0eedbe5810163273c1ffcdd43f4a1114434d48b8c7f85389b24e54c3d925b2
deleted: sha256:0f337fb83e170b58be142eab9276d0b16a2803684d350dc9a4026d822e9ddebc
deleted: sha256:0f6b52b9c5b099d6ccd2eb3843fe19af19d263d65e9daf7ea2bd0b4740ac1bee
deleted: sha256:713f4c85f20e449a60e5ab96e8c696d99f2e20ec11fc0c0f467562dae2fc59b8
deleted: sha256:6ad8cdf3a7481caff2a0dfffe7adf98b42f5da66da5feb26718e3956c531b494
deleted: sha256:fb09271fd450a7137424d96d9e995ae5984fe5990ae405b7b52105e6e72a929d
deleted: sha256:c97bfcc9f7f818f342b033d5d698dd851acc9f3e74f8cce9855de2e3bbea7388
deleted: sha256:f85672a02842a88a8173058dad867c6be66148454c9ad0a3e5dcc332788cbf45
deleted: sha256:f7af973c0dbb003a234db58575e74dd9167e4d1c513a0008971c37567a1922df
deleted: sha256:ea57d65f26bdd6304a6dc88a51ebe01517d4bad0b0b09f683f4036bf7a032d91
deleted: sha256:1abdc22fbfd95c19190a4748a1a8f4756f803e56ec1a88b93282757775d19022
deleted: sha256:7fc91448763d28f7f1db1f8521c7b0c8cac618718af6130165f0436c88a2f710
untagged: lscr.io/linuxserver/darktable:latest
untagged: lscr.io/linuxserver/darktable@sha256:bbe472d2a12a677d0ba596a1ee098b26b29cdef3f95e4cf9f06d3007f4c2ef75
deleted: sha256:0e7e71611cde5ed955e8ce6cd4a23735f19b5f3d94f7786292cffd166544cce9
deleted: sha256:29bcedd4d1dcd252a1f81a766e94a90a12869e2bbe08db255a0cd3246e4386f9
deleted: sha256:f75191f9cdc7fe0ba1aca7ac593b2ecdf142df056efb202030425c824bd697cd
deleted: sha256:5e1c056e8441169b89861fbaa849f0f795ea1804b1abdeac6f1733256a44ba25
deleted: sha256:ae9bb6f2e48353eecae4594355b119507a74ba68eceea7c3e35a10fb097b775a
deleted: sha256:75cbce05c70c4230412362aba4f7cc52c8be249093640a5062e52ba7975e2dab
deleted: sha256:7a29f1fa7b76c50f29456f34c8ba6e1bf186eb2d8749792b4cb0e57fb7ae152e
deleted: sha256:44f38c0f168f77e8c632006c16169411596640b15db2d973ba096d8d820ecf73
deleted: sha256:e6a34e4b513be11619b848caa40f59f216cea2bbd17e7741f8e8a5c112d94a52
deleted: sha256:6121169f612b35f0af735c1608d95c7068ad633b1d0f38849458809e90e3318d
deleted: sha256:222fd1a68e4662d97b8753ea0fcdb1fb421793ee4d9586efb33c2e979a69801a
deleted: sha256:8f9998c0d90266af44536ff534a0ceb3424dc54daad1c842e8ffc80ce8070483
untagged: harbor.freshbrewed.science/freshbrewedprivate/docusaurus:dev01
untagged: harbor.freshbrewed.science/freshbrewedprivate/docusaurus@sha256:b19c403e337e056d7dc2a5f6093330d7c6ae5a57e7be1d8e51c6c6c28ca05159
deleted: sha256:06a14264f55ed413fce19ccd6e60d0244af8222aefbae06925e0c727e9a706e3
deleted: sha256:1c9e0572c795d2d725ec13a7e2946c9ddc0c5a4af2cdfd5438ab6efc597b0817
deleted: sha256:d706ccb5fc5b9c6ee6c76ca87eada0acb04cf50507ef1c6f556196bf28842451
deleted: sha256:5fe008742c89a465e09d47d58f5643e64417796211e4a5324fe5ab171e772f0b
untagged: ghcr.io/codex-team/codex.docs:latest
untagged: ghcr.io/codex-team/codex.docs@sha256:fc932818383a59193669ae46953bcc6f1baafee1a659a1200508c69aed0dbad5
deleted: sha256:02a9679605a02038c0530b49304747671121b23985ee1d8f7ce02a1a77ca926a
deleted: sha256:120a9cebb9ef25e1dcbabf949d06da05985b951f383eab62c1a77d9b8c1a6f52
deleted: sha256:3a8523b26b097ac72db6b4e92ca31c504ad690d4874286a16a5aaeb243530b47
deleted: sha256:79f7716e08d81e6800e2b363718311a5fb29031f41b52fb8513a2679750c72c5
deleted: sha256:39485c477fac50845488a74e992d40d963becb930bbbf4d6415a0815542ea314
deleted: sha256:e4c6267884b459ac67f4088b1c8a8347c37b154055e7c6b717f6af63d9410339
deleted: sha256:afc914927708989de80814883cc8fe67bdfcd110f86ec7e55b24ae2b9ee4714f
deleted: sha256:ce1a7261e245f9afecb92dffd51c26b6f8bc5708be904ad3d613f0beddd89232
deleted: sha256:8b1eef4765b942d8154719378aa1830340967f4042822843f3ffac31fa1a1970
deleted: sha256:5e03d8cae8773cb694fff1d55da34a40d23c2349087ed15ce68476395d33753c
untagged: louislam/uptime-kuma:1
untagged: louislam/uptime-kuma@sha256:1630eb7859c5825a1bc3fcbea9467ab3c9c2ef0d98a9f5f0ab0aec9791c027e8
deleted: sha256:9a2976449ae2d48388c8ae920311a49b82b76667042b2007281a77f694c151dd
deleted: sha256:e0b5c8d3e82ba75daf8f8263d721266d7ce51a975fa304422844ce077dc5d961
deleted: sha256:074b46d3f0c36a152e365f7c590733533cb789d65fcb45af0e0700021d7107ab
deleted: sha256:aa2e956069aa64c36666b25ef058aa50538a066e271b7da6cbd5dc9f66d6df99
deleted: sha256:e5d89483341a7374cf05ebe796d2fe486bcb6b981e3fc81d9f25d32ee08b8fda
deleted: sha256:05b40974f3268824da654a42c6f41388f0bff6751b700cefa0bbf8040bfc8fa1
deleted: sha256:b70c7d3fcd76bfffc4513d04b947deb0fa318dfd94e3e7ff81e9f97eecb22575
deleted: sha256:76890f656620a7d3086e3a5337e658bca93b35f8dbdcdf63ae4e44088f516324
deleted: sha256:3d91731672b024e6c46fb1ca99e581268ff88804087c7783fef56039c55f4502
deleted: sha256:9372d9868a7909c18f4a6b951043ab49f11e95fc3dcc824e50384dfdcfa85f07
deleted: sha256:61a5c84a1270a2da38fd32b5dd395bc404fb6c02940fa3b302a94d1c105ce499
untagged: chibisafe/chibisafe:latest
untagged: chibisafe/chibisafe@sha256:79d5f6776476f64257e24d78830315b0bf746b3711662f2285b2eb2afb87b6b4
deleted: sha256:274cb7901e59c405ea2a58b7c4648c87212c02523aa444bac2a8f97b31e4f82f
deleted: sha256:86f98fcdc2a6ece2f413ca2d4343084aa45e22652f15f412cd503040676d038a
deleted: sha256:6cf8e3788cb7bf225a831fb51691d1ca3d40453967b5b73c52677ee7e1da34d5
deleted: sha256:40b23d2ccc6436bf6a3bfdcfc967ae4817673226e90f3fda4f0901eb987bd120
deleted: sha256:0535192b4378205bbce8b19375e172341e040572800a67694892202652a23be1
deleted: sha256:3720ea9c69440cd805cc4a6431dfdc573452fed52219932f19363d41b6a85215
deleted: sha256:0ff88f8142ee53aa055979b1013e476026be93b04b0bf372c11692af50a272dc
deleted: sha256:67be81b6de6e5eb71a3ec52a7e912b7f6f24819c6b7821255ad0543567dbafab
deleted: sha256:76d21cf0dcf05236c3a615ecd35a0ea35e56e5884818edd431b84012fb761701
deleted: sha256:2b4bbba9bd67cf0dbb1045a92a9cbd620a43ddc62cff5737986c1a4f6f19d639
deleted: sha256:7af3594b7afb6243c95c588ecc6cb1ad6bad5a61a0f942f82d20829ccee3226a
deleted: sha256:976287d38c2b59a7f430fe8d601d13c612434bd6db23c07d1cc6b1dfc1a6be34
deleted: sha256:5028b0694c5fa6424f9526973a5c4e7a2f03bbee4bfde36b6cf6ee17b782b7c7
deleted: sha256:8299e058ff3c7d09eded60890aa7dd2df2ee60422eb384e73e372172f5449a74
deleted: sha256:6ad56f1961981e5e1b31e11038e161cca3b71fee73f719d4d502eb91e9b7def0
deleted: sha256:f416c8c375971c03d22022e31a622b9aa55d52a312121d1524d503e83241d6b6
deleted: sha256:9594a9038004e8bbcf766aa12570de281412d82798390447164a1d7c6619548e
untagged: grafana/alloy:latest
untagged: grafana/alloy@sha256:c3dac4e26471472a3a7167ff9ce1a05190866fc9ac1cdb7dca9cf4181bf526e3
deleted: sha256:af84da9ca4a536fa8add4e59ed90da8b6a7195505a66fbb6ab52f4d25006de90
deleted: sha256:2eb8f4710cc4eef108adbfbcd3d9f6d630f0a427c580cb4d5eb8a90addde3799
deleted: sha256:ddadfb7a611f4c49c637fba707e3d8b428e5d9d980ab548680dfbfac9cff746a
deleted: sha256:121caa091c50f839500b04173d83391ad71db6ab0b1256e3ca1de673ea2287f9
deleted: sha256:7b4f4598eab85f7df26f2fae6bd58bb24b1d22f0f5ca5ce4bf7c4fd55ee2133e
deleted: sha256:396e00da786beb6460b7283824eaf802a04c9afaea36b51fb47ed36f415ee3a4
deleted: sha256:7c79de828e0e23c2f4520babc62f4c95bd2683a129a57ce7bbe73cba6cc16594
deleted: sha256:f6ec936913df41354db43b79da218c362b29da5818e5bd21adfa1330ff945faf
deleted: sha256:bb57e9ce36a2c232d3194ae711793ec1528249afbcd5ff6ccb0b21a6ff46a5ec
deleted: sha256:95ad1df54f2391af54181d4fa39fb349ca92483553928e87c59acd7e68311d3e
deleted: sha256:b453511cac8ea9e64fd9cbf309f1299e7306dbad275f51b81c867d2052f26bf3
deleted: sha256:97f383acf79d194e4b447b5c631e3abc4470c56fa29ed3bb37730a727c70662c
untagged: mysql:5.6
untagged: mysql@sha256:20575ecebe6216036d25dab5903808211f1e9ba63dc7825ac20cb975e34cfcae
deleted: sha256:dd3b2a5dcb48ff61113592ed5ddd762581be4387c7bc552375a2159422aa6bf5
deleted: sha256:8c5e3b50c7085016b7c692611fdedacba442aa0f4ccf8739f4ecf4d5e49b7d91
deleted: sha256:2ff31bb829f16adc374868d9ef59254f394816aadfeb6b45da5bec5dcde45540
deleted: sha256:91bcc75c6967ec46f3f19fa96f248ab3a23589788f9e2c4a1a2e032112def863
deleted: sha256:127296d2ee30c8716069dc23d015d9ecaa1544d0a2c44712db128ae6a9cb1431
deleted: sha256:fe36f45d1760f56c23a9957254682d74aa6e7fba931dc8f11addf75684b1aa69
deleted: sha256:8d6550c8cb1e10f9d0cf1ee3061ecef9e2cb333cdfbf58a02ee27cf2dd5fc35a
deleted: sha256:64e278d98d2502416e0c55646e919f0080a50f9a342b0e28aa9c43655518d3aa
deleted: sha256:013f70419be70a06c8e4ce898f2dac82be8749cb9979d9e5b1c98dd82519f627
deleted: sha256:c49bbc9eeced1247fcd625da3e2c2886e89b15a6264899a9ce32cd21d927912e
deleted: sha256:031797b961ee18c70bff1f3b53cea52f643dbaef6a5ecb0e98fc2696e8b039a5
deleted: sha256:2b83e5699838047f936a3875bcce7fe1b169983bf86785ae7519c5bc488558ae
untagged: testbuild:latest
deleted: sha256:0c51c72c0de6f722d909cc45e34021deaacd253d8e32d6069a6d17e39726cf13
untagged: chocobozzz/peertube-webserver:latest
untagged: chocobozzz/peertube-webserver@sha256:eecc8f2024fb6e4c38e2656fd95c02c928251202347151d886318541ecd8a837
deleted: sha256:55a2fdc70ce88c2daee13697579249db21a0c03749061698486715eea7211ece
deleted: sha256:f1a15dbddf407e0a5df2fff4249625ff790cce353fdac3fe846414a0ac29d8c6
deleted: sha256:c3eaa8dc14094d454be622ead66e94b0d23a41fda66bd74ef980c40541f42053
deleted: sha256:e147e9a32ff5e22582c92edb73193bd5b2219eb25ba7a28e5c1385fdc61eb6a6
deleted: sha256:12369c7fe5ffb31bc592a24c0cd081c85f34702da4b747ede00543e6f7f54a74
deleted: sha256:2b3208f4feef2df0b1c11744e87d2a5c41a1ef41a1217f7d90f1e7c1dab2ee30
deleted: sha256:97912e57274d7772d7f052fe2d671c5e0ac193863e9d5d02d2575949c17e1cd0
deleted: sha256:8d49f96bd3dac9f64c8b46bda71c268caa7eafb1d9fde95b93a36133a1e805fc
deleted: sha256:2765f389f779d9903825e36b704119da1da13faa4e73a44478fd86f577f4b738
deleted: sha256:baeb76f1ff72a2a650534f62c17308491c058905a82289971c604dea72fe54ed
untagged: b3log/siyuan:latest
untagged: b3log/siyuan@sha256:1e609b730937dc77210fbcc97bd27bcb9f885d3ddefd41a1d0e76a5349db9281
deleted: sha256:18590542671468c7ca827c1a42dd305b13e21d5e3fb93b8b528a0e3ffd9f3a55
deleted: sha256:19ea87ba8aabc33e369b254fb1e00c6d00f7808d771b0293ae11f4aa1bfbdd0d
deleted: sha256:37d00df7f3c460e2816f2be9c3d66f2e4cd82d5963016567f54c69499cedb25c
deleted: sha256:f0c8bda3fdeb875a4adfb2fc475103aaf0eee6aa4beedb58182297ede5abcc90
deleted: sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820
untagged: gcr.io/datadoghq/agent:7
untagged: gcr.io/datadoghq/agent@sha256:58d53ba7fa4a7c291602fd69f983417f000142a6a78fa0ebf9bf0f9c67a62c0e
deleted: sha256:25ddee4e7fa90c768c260d358a5b3436f3ba24bcefb7df0f3270019d4e041f79
deleted: sha256:aa463005fcbcbefb0ec41ec4c574a12a7214f5783afd7dc9a79424665515f343
untagged: eugeneparkhom/pomodoro:1.0.48
untagged: eugeneparkhom/pomodoro@sha256:dd07a4e830caf50e8a29f8479aa8cfe79a1ec6c08d4426c7fc54b20d895b4659
deleted: sha256:251bed656706e0239cc9be28928890e2a0e38d5b78d8c90b61d894dc39b63441
deleted: sha256:00b2a8c426c7bb2bfd11127b4ad2c70ae67516531cf2d6cba3ef8408e60007d5
deleted: sha256:d5844849ddb8027038cec120349180c0d4eb22c2a5f747ef62bbcdec031068db
deleted: sha256:bc32ed7434f45cc008df5f09266e2a81b25f33c771cfd87aec9e4655a500de76
deleted: sha256:589bcc284f24d6548cd3cef06ace5f6ebc4f23a48a5763f2f9d3d30b0f9dadf5
deleted: sha256:b3addc7069fafd183d88d1a40bb3dfe51227d45e4fe8e59b81a2fda7614ebbc1
deleted: sha256:5bf28af6a2188fa2d657e451213761b03e115e4c24ee72c41da34a241fe81ca1
deleted: sha256:2496134da21702d935bee1334ae42baf26d0197af91275e5c1a11eee31299121
deleted: sha256:c7e60968a54882c23483c3acb0ff1f415ce0f98184dfbed3fb9080447d79b313
deleted: sha256:49bfd4a4ea578aefcacdfd87efdc4999d6a4e4b7f00322484cac67ff7671389e
deleted: sha256:511780f88f80081112aea1bfdca6c800e1983e401b338e20b2c6e97f384e4299
untagged: 1dev/ssh-client:1.1.0
untagged: 1dev/ssh-client@sha256:b4521937011de01b6cbb6518c89cad533ff879b6340c3c77274e26760d6a4adb
deleted: sha256:7a4574c94f0fd4a456c848909f3d37e1fed29b1196a5edac3e174a8730df00ab
deleted: sha256:e60019f657bbd314bd725dba172cfc22263be4df6bcc83b41ed34e9e79da351e
deleted: sha256:8ceb9643fb36a8ac65882c07e7b2fff9fd117673d6784221a83d3ad076a9733e
untagged: pyk8singresssvc:29
untagged: harbor.freshbrewed.science/freshbrewedprivate/pyk8singresssvc:29
untagged: harbor.freshbrewed.science/freshbrewedprivate/pyk8singresssvc@sha256:588da840c695ec4690daac7f95d6e2f0043f53295161fcef54db81c60344e415
deleted: sha256:98d6b3ccbab40b1a9a7719c9f3c81945ab9bc986ff4a56565fadcce0dc6adf81
untagged: azdoagent:0.0.2
deleted: sha256:a5afc63a0a519d85f5817b3384d2dca680c337bfedd4cf79fa4fd40bc4553c90
untagged: redis:7.2-alpine
untagged: redis@sha256:0bc09d9f486508aa42ecc2f18012bb1e3a1b2744ef3a6ad30942fa12579f0b03
deleted: sha256:97ed3031282dd6ae5bc80c7168d7290e79a11871f29fd059b38aed8807d8e7ae
deleted: sha256:a676606653421115cb2c2ed2550955ae55a25ee85f58a0442b0144e3a748b6e4
deleted: sha256:f54e67e15e97b37e44bdda2be6cc9ac04dd081bade8f1f5660d254681f193a04
deleted: sha256:554ab4e9c6977e05f0859eb8abcd1c5a2527ef01f4a3ec123755ec3fb6e85403
deleted: sha256:50152d8712f02cefc42db337862bb5f92ed0568a16cbf0d4ea1bb4ec9444ef46
deleted: sha256:e6c37eaeb267d09a9c87c664166849e7c4f04c7d7183c737cfab0254715bba4a
deleted: sha256:dd39606edcb8484b51fb117e0878fb167dd3cc5edfd7cb473ecf43125e09a984
deleted: sha256:456edce6f329afda1c8fb468ba38c0d3d87c07d4cbd553753eb5c594f4937973
untagged: ghcr.io/idjohnson/coboladder:0.0.3
untagged: ghcr.io/idjohnson/coboladder@sha256:882705b34d7a7fcb47b4ac2a1155fae2b07b4cc0307d252f5b31d0731b3afafb
deleted: sha256:592075849eab012623dc183ff16f9efc93fdd7784bf1f635f26e6889774b9315
deleted: sha256:59a4113b9f73da7572d70c92715e39644dfe0027fd3100437b7ea8ad708992c9
deleted: sha256:440c4bdd8b5ddfcbe1885dd7769eb8bad0a5c9564e7f13fa65a38c5ccc390f22
deleted: sha256:a3779495ab3fca20a69959330e079992e9dc00dbbec0e88c4a83ae1f1dcc0f4a
deleted: sha256:a29f6ba95b3ff9b67ccd6d1f5c8c025132334b29d689139c44b568f2b74d257a
deleted: sha256:22a5769daca3c501d2e6322793a5dcdd1807847409519d72cd542e7c6059e14d
deleted: sha256:79f75e29f66ecd66d93d488868eb550e31bd790d79f12d9b19b4a3b6d20bab1c
deleted: sha256:f6a1fd07cb04e91ba09ba141a31cd7eb66650db8706323ebc6b72ea1ccc72227
deleted: sha256:a1ee1513cf4d32bb87decabec3f65b9d5a8b8516d9dac068ebd74990bf4ea016
deleted: sha256:bdd14fe246b6db2220ecc668361214d7ddee41d4b288306e19e013389ce29793
deleted: sha256:884e724cd438c62e84a580e0ad0f6b2b05ddf4879e71c7baf3c6aa88fd6095e2
deleted: sha256:65abb98c6dc1e93bfc62f84a217c1e8e8d35eb3459e91508d63e90c5ee27ae3f
deleted: sha256:7ab53d525d4f6dd99be4a5b930fe7d8519121db4ab75ba95ce30938a3a033133
deleted: sha256:574566ce36d6cffbeaf4c3326365ee982ecea18700b16569dd8303c444fe4f61
deleted: sha256:73c09cd545aaa14a1542c570c2953f2e2184102acfbb13e07aa6ae349cd80886
deleted: sha256:f046bf9e51ed6acb3d05c69600e79e91485bf62f495feabf2c1f269e618301e4
deleted: sha256:74f6042b32c1223c40563cf9c502d7cc066bae801b952f76383c014ee117bad4
deleted: sha256:3bdd514cc984fcfbde5e25f30c5ce8c4107ada8a1982567416391fe9fafc1452
deleted: sha256:29d0da7409ab0b2c77248c322d927b6178a7ae74780c15ebd300819aa49a80be
deleted: sha256:249a01813fb4f80d61131cbdfc63712bde1ef3716c9335cbc41a0e835a11bbbd
deleted: sha256:92ee295f45cf1bafc3b9dca8c3a9db8740956922cccae595e762758e776f44b0
deleted: sha256:4ca9b06b744df56df97b0f4f2e173b608a0c3c8030f32a261403f3a649071def
deleted: sha256:74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59
untagged: corentinth/enclosed:latest
untagged: corentinth/enclosed@sha256:d57b54553c10195c6deee92ea07bee5b7dc20e2d45da7c3bc009c3dd3901c570
deleted: sha256:efc0e300210f6596a37cc7c3fdacb636d2d34d8489484cb34a8d05549389ce48
deleted: sha256:e4fdb281b11fedca87ee9f113b81109aa375b3f45f222f4dcd1578756edcbba1
deleted: sha256:5ee91f0d7c182f63c6d5fbb913d5821df9ba76a75a81ba0f9c76f7ee72e6d143
deleted: sha256:20fbaf4a251f421061b03454be6e86f77c0c5c7a3f8b13488541a0f97e6933fe
deleted: sha256:7ad4de27844abe9de39fa89f31b7079fb846eb48d88f88f6428f989f63414aa3
deleted: sha256:eacb7407868d7f34b314a993cac1754307ac2a96aa0e189c4410838ac6193e97
deleted: sha256:4f6109e3e9df843f0767864c0d5da9a18e8fe26fa364bdb0aa30b924aaf885ae
deleted: sha256:c4fc8c8022fbe78202c34019982d8cc01375e1b8906b8ba7b1381cca9aa6402d
untagged: ghcr.io/immich-app/immich-server:release
untagged: ghcr.io/immich-app/immich-server@sha256:ac9fb82f2be3cc60d4e0c7f1ceb371d401d52573e387834d7a82776116458242
deleted: sha256:588fd1ca221cac52b501eb73fdfac8886dd33f4eb486adcb090eb023d77cb179
deleted: sha256:9de17bb4cf4d541e41d80380657e61aeb987b0fba2116004c3377f8d32fe0c2e
deleted: sha256:faa1d6b3a1105b0cce74e045604dc33bf986ae92e25cdbd5794e6eafffdc5823
deleted: sha256:55100275ae5505d79538d3926bc9d6620e93e2a6b8356f59284eefa6b813c3cd
deleted: sha256:d579f4ec7c4dec4c6841f86dbfcee6e3ee6d3d7625c14b5a36deeb0c3ac66cb0
deleted: sha256:d19196f0ec11ec62f837041cfb13e4ac072593bc905fdab0dc2a9547a4c29ef0
deleted: sha256:743ccc5ddfb3ca0e345fa01248d91a2ccb17513d77d2edae933d5cacd59b8f9e
deleted: sha256:34bb12c36501df3517fcd9dfd1f217b8831461a20652264608aba9d8f341e42b
deleted: sha256:9f5a1146859c2b1f1133e7726dcf0de66c0fe53bd9b8706bd7c88f5b068edbd2
deleted: sha256:3cddb258be1d2cb6ec3db135bdcc7f7f3227549fdb9d1ca89d50cc2912dd3a73
deleted: sha256:867878c216d9586ad0b6e225539b907984725235ab0e7361365f6c8a8f4f7554
deleted: sha256:d535a01812f8fcec391c4610d5d44a338518958dab0ee92034b6c430f9770a0c
deleted: sha256:c1e446d40dfde1e63ffa454208eb7629fbd9cb749035f5815e966cabbd5cc131
deleted: sha256:07bc9b12c644da8fa7084c317e10c323c05989b5a118799fdcd65dc22f49e47a
deleted: sha256:c6362fadcb6ee522e352d8a509086595af33c36017b15eb31833929cfb55d36a
deleted: sha256:66e945ed825b2261d61e63155bd9200ce7e3173cee1fdd24a2c0c688d0881b49
deleted: sha256:1a10f9962f889a35bd3ca31c4d21ca402eb11ceb8872c6dda7e60e786d763ae0
deleted: sha256:b4e5919b441bcff740fcfcbbbbb2e1a0e769f8c2ab8a767cba73b5b3069ce88f
deleted: sha256:74439cc3ae4ea08e70370e0ed026fe69960da1d00d5d79e3d80d9900bf6a1ea8
deleted: sha256:87f98d9660c1a0e5421abd182afca6a4baf367ff8fd9ad203e34217184aea8b2
deleted: sha256:3451a1e630bb6cc5aa7b391948fe608c899b2b8e9a63a386c63ea4fef2117f75
deleted: sha256:c0ed9d3a8a50d3d101f6cc01234fdea4723631eb41ac080af0ae178c071827ad
deleted: sha256:f75f05518c77fc53b84cc538f112d3bc88540ae373d653a9456ebf3b9ddcbaa4
deleted: sha256:9fb443e518a5b8b26f0add336675dbf2aa25c135a32872814d65d3d0cbbe14e4
untagged: plane-beat-worker:latest
deleted: sha256:2aa998e8bcd109b5bdcca7d864760c98ae708065f2220efde3ee5ffdc49def69
untagged: redis:6-alpine
untagged: redis@sha256:80cc8518800438c684a53ed829c621c94afd1087aaeb59b0d4343ed3e7bcf6c5
deleted: sha256:c9052246611fd2d9efcf620a05898778c045696a9ef664a13d2454e073dd05cc
deleted: sha256:d1eb35f91188e90b5460f63089b7cd3d6dc2db5b63364cd80cad49ca3e094606
deleted: sha256:9d9d5a3fed1a2ef3e9cda1e21f2ffed9810adc1a760061237424eb5a14f09b72
deleted: sha256:f49c1085a1e563323b001c193f9fbe8d2cbe4a93c6a986d217f11ca4dcaeeb4a
deleted: sha256:e19037a0afc94638253a57696706b78b13da558222473e38007ac73a8abaa992
deleted: sha256:08b63ed7d44b1dbdd8fe862e9a0e834102bc30264aa4283cbb135b9508a970bb
deleted: sha256:6721e8737c81749874005425bd25a1b84e0b6c6c8d882de8d6cee5bdcd841f46
untagged: mwader/postfix-relay:latest
untagged: mwader/postfix-relay@sha256:91bdd69d1aa1b7a5476dc7416c4b629c0e21147843483c901ad92f6dfd3afcd3
deleted: sha256:dfead71fd58c206ec631ad1f010229018e2dd5fcb68ee139041b37bd69b5ca77
deleted: sha256:5632c0aa5f108ef08375df7882974d0517c11fab627dfde119bafc6cec3e7661
deleted: sha256:288a68d3833c176825efe7ffa846974fe97d13afe4b3cdb264d1c1753174c7ec
deleted: sha256:03023dbc16c2b63ef166cd5e18a2eef8fc0a68f6bd431aa0586195ebb6fc3cdb
deleted: sha256:cfa496fa54c6477031dae0660ff7471870c094c326de4214e8c6423ce6298e64
untagged: amerkurev/doku:latest
untagged: amerkurev/doku@sha256:145084d4d910877e1b0a2ce80da2caca1abbf5ba58ea732c589100b519c5d96b
deleted: sha256:33e67eb2f819f27b8ea5c9940331adf21d255f9d09f5ed86307e24eece01b212
deleted: sha256:90cf3800b14bfe1f9ab6fa27588fa626c6d2f5d547792dbc5c97e5d43a492e75
deleted: sha256:c29040e4e280f4da19a600cdd98a517722c8de1cc8baaf2d7d8fe0a53bdd8dd7
deleted: sha256:8c056fed09068922ac03bb18176e2e8353aa0ccba7d1d5be4bea6e88c6758317
deleted: sha256:281795bd5ccc19ffc4b6da99e21d6176c32c18fa3d7e13ba701c06b14007aa29
deleted: sha256:f532ee424da435cfdec198dd9f9820534ed52a44b80a189c47ff563ef17aedef
deleted: sha256:8e4480beb80c4ec529664146c9cdd90aef2a351a666fb569f1318fadbe2a953a
deleted: sha256:b0fca3dab746b448489db7a7623130b70c7ca5c169d485ca79cc551d1ed79195
deleted: sha256:d95277f2e59ab63a2c3e48b472d69cdc6373586fa00a706590a3dc4ed5a0a976
deleted: sha256:9b9a012fe1c8c30e57bbabfedebd8e99bdd09cccb3cb62746272f27287de8991
deleted: sha256:6d062e72b951af3dd23c652e00e3610e4aada0d614ee0257cddd9d7c110f00a8
untagged: codeberg.org/forgejo/forgejo:1.20
untagged: codeberg.org/forgejo/forgejo@sha256:d665129e66cc04fa72aa6a284eb98b17c0eee642aeaad7c88edec21f9edbf519
deleted: sha256:a805d8499540f63b24e9e7f462542416227b2dcd647ed38ec5fcca0b51d9ce0b
deleted: sha256:ba57f87cb46f4b110236435cf9086ff934084d9d65b0ac4ae230fc101fbbaa7f
deleted: sha256:68711e0fabdd489b68b4f6aa4f8e259c3494de842e7c729ca9ed859e0ce9daa4
deleted: sha256:567ba6697a6185ece19d89e188905a0d7a3dd30980f3e8219b4c3f639c06ca78
deleted: sha256:f3b8851c730bf3a5408c4986701a28bcb86a337a07e55b82acb38ecc9d5861a6
deleted: sha256:af91b787e0c64c81de074b89f67936b52e9e5c8b7d9e0e1b523ae2b3521a8c1d
deleted: sha256:27985bc80c47184b580fbb92c113cea4c41a79623c3c91fd0461b1c8d1a35853
deleted: sha256:303a171ca77f35448f38a2fa5536f3a8b064ce595e642444003ecadf8fd4daa6
deleted: sha256:e9c49ff8678931661cb316225af3a0f644a19d644b937d21db02be406b869128
deleted: sha256:603f3610a8221dde0607a3dc6e25f75159da173f93a6eb78642d19d9f5e614fb
deleted: sha256:7bb5d81b83562da31f7eece33f4933e4407f624087bb2d144c691e497059087c
untagged: filegator/filegator:latest
untagged: filegator/filegator@sha256:9d40cebbeec0c7a8ee605deb6d8b8d44b08621e2158b8b54388e4dffafabd0cb
deleted: sha256:c96d866da505713e02e94345393eb0e5537848964e72e50d58cd7d2e8d8016fe
deleted: sha256:cf0b2a1b812f86d93e30d0a668427171335b3aabd9aa7cd1be770f425731d97d
deleted: sha256:93544cecb887acd2e6152c73c2d5f3dafb9e313350f8e191993e18c4f0e8d510
deleted: sha256:aab0a4b9b7b6159a3ac36886dcb0aa2a32693b968af8f0b7d4b63f6ba4803539
deleted: sha256:8ed9db2f4ca0817b78b2a59e66dd9428a50d2612f51320027aca4fd06b633d0a
deleted: sha256:79c9d6ff5dcaed785b9007d76988a85bfebe889f4c7f569e64fcd28e5f555323
deleted: sha256:c442dea235d6deb20debb89c04e71648dd836b3bfb2f4886dc40cacc4d7db98d
deleted: sha256:aaf954db460ffa188652291ef0b21fa0b2b7fadc93c9cee7f47c72bb367bf150
deleted: sha256:69980d180a95cd12b851a7c4970d5a92370436391c59eb33fb6a18b1a86f81d5
deleted: sha256:95dc4079515c5effba998c04f2caadce85197cd55c70c7e26a4f3ae68bf2213e
deleted: sha256:1e6d2b5613e7c40d599b9855767e16423f8cc70a68398648b2b95de663cd594a
deleted: sha256:84fc3c25473053d570d01eda682e175879b4bacb291b0dc560e74fa684535353
deleted: sha256:80904ec581980f81ff0bd6918a37a08ca036df17237b2e95d2cc213fd03ca68b
deleted: sha256:398846a16fa3e2b59d546250cb1c23a07af7dbfafcd97c54079022135d2156fd
deleted: sha256:be8c42ff0b094dee57f1ea55a12686aa81c10cb099a971727ebb4f2bbff501c1
deleted: sha256:fecdbd1324966cec48fa19a68477ef1f6d44343244ef7b69b10ddaf979eed33f
deleted: sha256:afee6ecd524fb8b197278914146cd451cde007928cff7e2f10cb14a37c0472ec
deleted: sha256:122db55175ddac781586e314ed58a192b2e3165df140f10dfadb01ba64d15fb4
deleted: sha256:75764c8607b9f9fe26133a1585d30b03e93b4c04c4b666904ef5fbcd25f2e9d2
deleted: sha256:63857b9daf9ede973d870ee2301373a9f9b42b00c60a2dfb98e824b649f9b8b1
deleted: sha256:51ab9e5352371b01c32109cb0cee0dd1a5b7a504e896f4732db294a21da24480
deleted: sha256:7885b3ac708405e39aa05e8ef81fe6c676337da880f9c7f6b9c2507cb4aae929
deleted: sha256:d520254c132b39f9300806582e8bf01cae0265d33a3845a3ef466f2c6edcca3a
deleted: sha256:3bf84dff1654ac027401b8bbbceda3849763efb9844d19698fd92e10e69df21d
deleted: sha256:58a939279f631c0058eb0c43b6c8f3d2d1bfa60450daed65d27455b8c403cb12
deleted: sha256:6d03cb5c0ad5ef46c9d198872f45173ba28add4d04da52cdcdd9672cd4b4a016
deleted: sha256:c4ba967cb9dcda38afe7f74cd5aeb345852ef4449372e672a7c39be04f4017a8
deleted: sha256:f28a3cd74dced91c0f40de6692238c7d4a016364fcbc4b8500de182dd611e808
deleted: sha256:a28e71738952e69bddd945e963f12428a1ed32fe103c7ff9404dde8e49f02dff
untagged: postgres:15.2-alpine
untagged: postgres@sha256:d9c304353c031b21e9a7e33dc4781e272a9fa802a2ab9703fe4199d72ba1422c
deleted: sha256:ddc12ac7fa27279bfcc41deaccd1368291ab7d71ca7cee0420b41f9974e9a468
deleted: sha256:b240a07d9992fb80e84549afa77f8b30b179a3bbd1e7e63eddb4e2b531b85310
deleted: sha256:424450fe7ca8478596993d579f05900e95de031e53ae90809a64d55be2b50a46
deleted: sha256:a7d1783cafb55c0a17827a94ad29a32dad8f9898f8eab3e4504cfb4404936541
deleted: sha256:9f78018e6a1ce9d1e8c9834c8913b191e92a92f7fe953f2015f4f3566726fc25
deleted: sha256:d02d36077f4aab6b91edba540c43655b3c2f80d979bdc4ac0a20970216eec415
deleted: sha256:0ad82cc2bc183de295e0d3cace4c05b4b2af3975ad72273a9b8515be9e7d88ed
deleted: sha256:602e77ac65b79530392927bd919b0d9f5bc06eac59c538c058061f6bb8844494
untagged: mbixtech/observium:21.10
untagged: mbixtech/observium@sha256:e1c3a917153a9a536cb9da2006b0bf0c0fc2e887dbca780865930457dd40c7bf
deleted: sha256:defc5fd072453121783069ba84a8f9ef8afc7c471a0627d86ef6f461b0482343
deleted: sha256:f26593751feb26e82f4a34cbb1a20a5d5efdedabb85726f848c7ee9e7d38a7ce
deleted: sha256:0e74f83ef053b2b02d27b3699987a84d5b87fd1c2bb4a1f9c574269484028400
deleted: sha256:ffdeece818556b731a1df4bb6a2f17dca8388e702e3e305b19dd4b5dfaaca37d
deleted: sha256:39779e447a33ed65fff2c96edf6116d9172b57fa8c7df9cbf3b605273466729e
deleted: sha256:37ef031e382d9ffa264a5199ff9a8a85daa7faae1103714205cd2c8a3b3bbac8
deleted: sha256:304f1b4f73686c52281de80c73785121975dfb399ebc0c46ba8e15214d12952b
deleted: sha256:ff5091ab3adf5638dc70d883be88892632ee7e0cd2fa223e78f4b090e4e1b9ec
deleted: sha256:bf6f0636263cbfdd4134d279b5161c0f990154806d0a73afae008c2a5ec6d063
deleted: sha256:f6c88f0527ce8b48687ba0eaae48d3d0cd440f93de9a3b7eab5a5ab9e8976940
deleted: sha256:2f86bd5971ece22287ff69821e58c529668eacdc7e2ace9b0f8267028d0edf69
deleted: sha256:089f620c94d012beb398de0946c48cc44fa4cad8b75ed56ad07f62a90ac01f4d
deleted: sha256:2358f1dabef98ec169a966d2bf3154fc3af1ca8de642ab453da7c375f72c7e86
deleted: sha256:2a384aef5498acc3b5abd506b1275685ff2c7d2a7d170784ead2285c27390335
deleted: sha256:49ab163b2fb2d9c7a38242459511ae325a43bd46e9a6e485850f32cfee9cd4f3
deleted: sha256:4942a1abcbfa1c325b1d7ed93d3cf6020f555be706672308a4a4a6b6d631d2e7
untagged: fireflyiii/core:latest
untagged: fireflyiii/core@sha256:b9905d30934be51453bbb956be88c9ff9db35edce6f3779f8fae1ba87e820a69
deleted: sha256:270dd52f5254fac28316956f5e8bb17689bca5757738113dd84032dea77c4ab7
deleted: sha256:1a7076cf2dc772092918458327738f22c64b2626ec9dafc700050e586484e443
deleted: sha256:6e29ec5c658b4dedf7d6d6edd0108da7c98eab413b331234b629bc275d88e3af
deleted: sha256:6ee604a472d06c74e440048794b186ebac06872c439bd4bc8482ff5d3057e0c8
deleted: sha256:b493cc0ef6bd0a52807555a39b3d0b3d67e0dcf54e0e735bceef4ba5e803ba67
deleted: sha256:b7bb9221a18028275aa8c1b89edc9a4300ae2a53b08e6fe6640a49814830f5ad
deleted: sha256:4c49a1549ae15f5530cbcea81b4f8c7d35f947aae8de33593d38b26d88464ff7
deleted: sha256:fb8b5a291457d9056f63e0861db1ca28facd9a6ce32da1c412fc7bfc17869a96
deleted: sha256:c586e8edb074dcbca3b123105f1034978f8eeaf8b9c396c4f742cb0f82c728c1
deleted: sha256:a466b22594c4c1a99dcbb3d8fdd59c8fbda17a40918b64b882ddc82b48933ff9
deleted: sha256:fb8f0bef733ebf6dc6cc5d588bb598734746aebe3cef78c0b1e08e5f0c1294d9
deleted: sha256:4194ca1ad87977d3b215748d753f7aca02e3544777714c80d3236081696c852f
deleted: sha256:1ab347f8447559a3b15aa6db85e1888cf22bd50014e611d30f9ddb2a46de1e5d
deleted: sha256:4cdd3ff5d918070f6e34951a567bdd7ea0b945e204babd3f10749491b0e7cc3a
deleted: sha256:186ca7f7f4017c84e637060554b2cb42cb9c511cd1709d85dd122c3dfde8f881
deleted: sha256:ff6bf9950311759d8fa11c5723f82fb40694faec4b84e7e4720e58f9f663b49c
deleted: sha256:d968739b9d093cf66204db215c8dbf06679afcd33c462abcbf2281616496ee67
deleted: sha256:6132da6c63f225bb1d55ec9411044cece41b4bb7210d3cdf5af254d65e8b07d4
deleted: sha256:cc66e95905b100ccfad4058f3cf559b517f7c11324d97be8431ac82badc4fac1
deleted: sha256:5c7dad5f8ee1fd0532c376921b0257166eb9ffff6edbea6af148165e6e0fbb2a
deleted: sha256:2e7b5eb7f44264455ae99666abbb9bb0868cbf37caf3a56a545bbc2c03f943ae
deleted: sha256:f622eb1ccd3d70ebe5f5b4a4e934fa014c8601eeb9084a1ae351f7f9f9d834bf
deleted: sha256:d478413409ee51fabbae5d5c5ffc8b122129630f298f69782494596055ea35f4
deleted: sha256:43b4c1a956987fadf31626034ffa78e589b72e794ab2dc02cded14bb7e4a5be6
deleted: sha256:3278f79ae6a35c64925b23a165130d4194d6584ee88a855d13f46134ea64f367
deleted: sha256:0147530070908d371f8aac0fc0ac5c1f3ba1f06de14853a755af2ebe5282d2b3
deleted: sha256:e301cc1d711b2dc0d394a2ef20d57499381b1820a7be8c5e9b1f5b9124c0cb0b
deleted: sha256:0f6e020588b992c8d437a8225f765191045cbedd9f2645e7fb768e2b489afbb5
deleted: sha256:9ff2c2ef2cb6a888a58fc688fd7ba784b3569bdc4c74eaba26377301e721936f
deleted: sha256:cc7543c9a1920618cbd3024c444b31ca2f8a54f6298b375b795fef5f31678db5
deleted: sha256:2a5ca866f38bd930e53d9f372c3f3ef6d46f13592256ceb36afb769814e63c53
deleted: sha256:61722bbbc1b44ecc9eff1efb48ef529d2a0adf7d5ece52a8d42681d0f238166e
deleted: sha256:e6cd39d4cea4051114c9bf1c61995efdb7b33a54c7b46b33b221f8371ae8c2b7
untagged: 1dev/kaniko:1.0.2
untagged: 1dev/kaniko@sha256:c13b73554dbfdd0376c525e6fcd48201f95ef443a906557c0cbb709f94118aa4
deleted: sha256:7dc0aaf1eb042668bbbc37154e2f1bee158c23e2ca23adb087b21d923657e108
deleted: sha256:53304cd3b35496ce5ad21538a90c3c022020837b96a6ee5334cc279d15ac3db8
deleted: sha256:66b904592f7b101c1d3c7c61ae404fc8c673e32b0dfa6c8ea801c652b70d52d4
deleted: sha256:3ff871cc18503928225cf6fe27ca727b2d78b7b11c4111f4616f290629647b87
deleted: sha256:5670aa799eb7e2838c18cf43cab622b5cbfe9806d2853ee13090531abe3395b2
deleted: sha256:e149714013887cbee9a7d4f474624d98dd84ff58a11eb47e177bfcaf6ae56d44
deleted: sha256:f7429ad8eb7bf4ff783b80cd54f3430ad7c89d9a95dd25f6eff32d09f1a29c3d
deleted: sha256:791afd92a22d82b1235715194e272ecd210ad6b9e6801b6a109cbd508b1ddc03
deleted: sha256:11879eabbaea979af072f21e6d7edd409711feb67371b8a2f44fad827c5c6376
deleted: sha256:66a189525acba8ed0421a4b439ba2d67d83a88959307d9f0ad38e6fdd6dc4605
deleted: sha256:95cb6166a6e956a05e1aed3d6d0502ad8288b841cd0c034e7d6fc6aaa8e88394
deleted: sha256:a4105d1f8725b87146687ce6c40ea70ea278af2f7f6144fd9d957eb3929c9679
deleted: sha256:f464742827b0d957f1dea07a3d72b22ad27501feff2ed2f3ee424dd3e5e5f080
deleted: sha256:c52c34bfd40e942f30d12694d429affafc7d3ad9523342d1a756891c1b089cc5
deleted: sha256:69389944e44a8f3a61e3bfb700c227ab0c83ea6a70164c2914531e4c24900889
untagged: plane-space:latest
deleted: sha256:eae88d0670cf36928a36004b8a52b21f0f64e2983f6813de5089f86f9d726c14
untagged: ubuntu:latest
untagged: ubuntu@sha256:278628f08d4979fb9af9ead44277dbc9c92c2465922310916ad0c46ec9999295
deleted: sha256:fec8bfd95b54439b934c5033dc62d79b946291c327814f2d4df181e1d7536806
deleted: sha256:27123a71e85e0540333291adccf7b9c340d089e8c3717c380d3b75cc8c7df90f
untagged: plane-proxy:latest
deleted: sha256:30b0e05f068e12d9c95458a187f78b4c0f75e01c8d682fb2a67b2167ae0d7cba
untagged: redis:latest
untagged: redis@sha256:a06cea905344470eb49c972f3d030e22f28f632c1b4f43bbe4a26a4329dd6be5
deleted: sha256:f02a7f56692880b88d5fabe631da5547ae1d8c30d7050c3d004347a9fa5e5a3c
deleted: sha256:685583ae93732862468e2a7edba217ed8165dfc730c363c30bb9405c4c81aac5
deleted: sha256:fd8efb49fd2763acfd6a07881e62713a246858a4cdfe4b9a31c9cb3336a9c710
deleted: sha256:3895fdb072213028eed326869c37d52d4b4003c6b0d0b4e6faf57bf18e5d439a
deleted: sha256:7c9bf6b85511bdf143188f915e200ff94b46dc75d6d218845a939bfa8701f1e2
deleted: sha256:ba40754911b8313098bdacb32d5044d6105f72a3b9cfe4a9ea16b8f05861614d
deleted: sha256:93588577b39273261aea628c7f6aab541a86ff37e1f2674ee421300a2ff34d02
deleted: sha256:1408254577b9d64420dcf7d4ba1b0df9a3ddd7a29d3d4736ad2ff4672baf4a8a
untagged: portainer/portainer-ce:alpine
untagged: portainer/portainer-ce@sha256:214cb7f6c4034420d4f74de81492bf8fddede0677d2f12171955e075ac2ac341
deleted: sha256:8d801f792bd3e47093ba97b9d11a5393735bdb2e3eca0f4f1dda3245c4a9b0e8
deleted: sha256:6bf9e3d171f1ab22a00545c5738e8628a4d50d912fb8384baa0ffb62621bc20c
deleted: sha256:fd1e9e621b115a6880e2f458e6bff5b601a5136e873ac21a11c582a5884bb887
deleted: sha256:9e93812a81af87def2f54c98eb078d4e1482267618f4b290625d279b79bf2afc
deleted: sha256:da6f54348a64b60d4092ff7572befa9c2cf14bf3fe2cc64ce4feca81ffaa9808
deleted: sha256:cd1881999747255f3c52511f681c0dbe1a0fe525426306a3b54303d553ad7a4d
deleted: sha256:9816d193ccb39f0fff95d1169e13259dfd47602d3c7cdb1d470ea221e013f2c7
deleted: sha256:32d897c177e22fe5e549e8b7450c86a97b9502186adc9b303d447d29f815fcac
deleted: sha256:1eaefa6c47f1ebeff4199bef76d88cb9b5149456bbfe7054898bbe6b0c1c2415
deleted: sha256:0521786f67e73a0f8197bf41bd4a979ae7d83550004f9134aba9e70c6b657297
untagged: plane-api:latest
deleted: sha256:b5ff5b50e413fcaff121190db029f2251e31ff43b393ef6c38c29474e95e2a8e
untagged: alpine:latest
untagged: alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
deleted: sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418
untagged: redis:6.2.7-alpine
untagged: redis@sha256:fe7cce7c87019e13a14a04eddf96d74c72ae64d503ebf6f8d38ad0393d13ead1
deleted: sha256:fd45e5a11a9d7583a57867f782459a9b04b4949a78bf13ab79b83017b4da2121
deleted: sha256:b9434c62313e39e657f6f154dde8908db0662e8557b75e75b42f5ca36e26482e
deleted: sha256:97ebd486fe7397910c590b7d7f64020c7238d0a36c52e9876be4350ed909c439
deleted: sha256:36d5726cae0312949db2baa9fe599699698fb4ea70128cdcfde4bb426a5b1ea5
deleted: sha256:2be9af83a20dd32a586181e7812c052dccd241221307e8b973d18edd652034b6
deleted: sha256:3b4e6842445154483a97b7dd1be34847a9813e95c68fddd2cbb8a23e2196d573
untagged: mbixtech/observium:latest
untagged: mbixtech/observium@sha256:dcf2b9c60ac1418f25a4505ff8e4acf45d3c41eeafe2ac7c0f6b2d6d1f23a9f1
deleted: sha256:578d83f3beb26eacd53e6423efa06458224183bf9449ebbbd8f80c80235199cf
deleted: sha256:30b424b68624e9e6eb1734b20a95278536dd1eecf2c2f6bb97b5d38ef1688f85
deleted: sha256:80f20fcdaf4ca1e11c7e89d9ce8d01371321c4fbf7991633643c475409ba76a7
deleted: sha256:4283d94f7105b2c7b0611cb32a041af9176349825829adc23e88caec4f5d3afd
deleted: sha256:6a4ba31df06950d1254314ce717f9260a8cc51f66e199d3a09dfe984ff737ed3
deleted: sha256:4fa8fcf4be433d702a0f11293a1ad1761ce7cde401d763b599ced0080f65ea9e
deleted: sha256:f249059892594ab9bc1d2d33f65c885de2ff28b318eda70de153c33ed15ae38c
deleted: sha256:300889791b0dbc4df75e1fab73d9a95b258011fa7a305b1f2c175f457488baaf
deleted: sha256:df7022de69542d070c67a42abe48b3bc99f5c5897e3e88197a7c841e57b150a0
deleted: sha256:5d1900a11b23ca02fc62fdee0a413116e5c4d9ff6a291c5e42966670e07b1608
deleted: sha256:0db865b59d8fe70f7ca0a09b38703dc5d4effc9cf43376a255e702e2ebffe66d
deleted: sha256:1aaaed7daef60caa69ef8e3615724223f3bc188c2743f3081dc46a6c205ab475
deleted: sha256:0de947dc788fc443b622920c0c8d36b319601ff4e83f71cb4f1d165f4d561e36
deleted: sha256:311701a0871d14a0f38b8e4036295e0d14a5e7958e5e87258b64453e75141d7b
deleted: sha256:e950e927f9d22488c868dafe33215332a764a6508159148d30e0fc0572339902
deleted: sha256:722efce25986c03c3db46d8eaf30c9942065e0149e3d1f955db28cc821b1c5fa
deleted: sha256:7a91184ca2ba987276aa980c9752299ff9dd4dde2be4d4bf3ddd8f1e7d8a2e34
deleted: sha256:24ec4bfa7f02be1cf758bfb3387096274050b4982569b0e8101e9b7b5dda3a56
deleted: sha256:a1360aae5271bbbf575b4057cb4158dbdfbcae76698189b55fb1039bc0207400
untagged: lissy93/web-check:latest
untagged: lissy93/web-check@sha256:a2ae048b601c7d44ab148d746f5836ace7b4e8514ba8f905c4890b90635c62c5
deleted: sha256:fa403c8cf45ca51e0eb3d64a8ab5f0cc3e3af7cb97a0d6f0098148a37dcbdcd8
deleted: sha256:6fb2b60d3d3295a0f5aea7fd6e9d61431a8b0c13a1c6963eb1ae98ea890348aa
deleted: sha256:8449f133f28b046e6b5159672afd162680da79b32fda2cf5f14953e1930817ca
deleted: sha256:64e750144bd1490d76a0e3dbb1e9183e652b50380db6ac415c2880bc396734b2
deleted: sha256:241e2c48b4fb8352ff29fbf70a2d870638fe86dcb2a85f9d346d524775ab6178
deleted: sha256:3e8987cf2781707ddc44a1eea7edda6f8f75cd0f3975deb4f2b4bcff5e151711
deleted: sha256:6eca432f3d44bbe332903686931286782caaf36ab4d6f03fe0f7a9bfb72440e3
deleted: sha256:07e9a6e46a35fe1022b946d169b93985ededf989e4045b72373d50ff822e6162
deleted: sha256:56e9293e9e0997ff28a15f10470ce0a9f305c5b4b820db155926a53979427255
deleted: sha256:024f1363f48e664e2c72cef6511d1ffadf5614a5b97d174cacde029b768ae6a8
deleted: sha256:beee237af16a2e57bffb2891e444ec1b86e98e6fb042d45b5a90c4444848cd33
deleted: sha256:9442508e9b3c39c50ad9aec81aafc4ace469888251aa3f63497772e19e92fd42
deleted: sha256:82677505c894548130a3c97e95faa57399f02ea2fb97667d934fd6b807ea7108
untagged: supabase/postgres:15.1.0.117
untagged: supabase/postgres@sha256:0746e0334189596ddfd0bd4bb53e3daeb7e876f745d3219f5d396707fc9c95c0
deleted: sha256:0ff08350249479474ec138e2f70cdb1943e1407f17b55501a28d4588d76e4abe
deleted: sha256:2c56af1079573aa3804c7380f4f9fa89cdcb6f6bed0009bb21c66987a4adf58f
deleted: sha256:897942fb9e9b6d0d13f1c0a846d06fcfe3b07bee49637b3828047f0108045e6d
deleted: sha256:0aa59b56b795dd8c353162cd67a367d433f065685d639740dc17728a6b823145
deleted: sha256:c3cd6599f493cee06f8f596441ddd9d2f8302b1a1f94cd47b09a528f4ea8dc10
deleted: sha256:0ad95209156e714059d8fd34e057ec8d5337397d3e7c09cb37cb9ce063c956f2
deleted: sha256:d22fe7474d014a145e1286bb76c871d655bb2bd5ba59213d825ab025c6adf1ee
deleted: sha256:d81eb0120b2f1180e5f3e5a145e52e7b96ca9a783278e6c17aed5a397ec6d640
deleted: sha256:8cde0a74c96db399f31a10d17815cd0cb9d74165d4717dbb6c64f365ae1cb75d
deleted: sha256:d82b969f872a4ae3527f45d56c7684022582faae6f98845103cdd0c8771fe12c
deleted: sha256:fcd48ec1b6685dfdc95f4287a3810e671ea388c4276d6990fcba4b89542acb1f
deleted: sha256:35996052b429470ec16fd081f394336f0ea9efd66530a26238edb177dcf93100
deleted: sha256:30ab5bcbb9f0bde60de77f96755da165d0ab0495ea0ba867a53c629b45c18948
deleted: sha256:50b62fb2edf8a5ea1d1a67dae81008415c21536f64140d8aecd25580c6f4a169
deleted: sha256:a7d90c1c61f2655065732f86d2ac50bcb91b8f93c6821ace48f22d93d71c0063
deleted: sha256:9d54e352869b3171e32cfe3d64b0eb204fbb48ac2f2e42885075c02300f2847c
deleted: sha256:8b740c0823ae21c6571e7621183f87277f6fda105355f34f814bbf1676a09c0b
deleted: sha256:499bf66b3dcb67501c19c99b538d2a35e8ccd33e8578b0c02cd89ee20b0a5cf4
deleted: sha256:cd2a3b7af6988ddf745c19c99413dbf4d56e46c1a40a1c9c7f609ba79d41d150
deleted: sha256:9de78ea9044987554cc4d2c7ca4e96d575e002bc6b85ceb1ed66fc20b852b940
deleted: sha256:e14778d6436d6a3155e11326f207f1f0e78298f697647848f1b76d9000d8c3bd
deleted: sha256:ff93cf6b809f3ca3de79292e956a28b4578229b911560de33707729b8dbf87c3
deleted: sha256:e0c2dbad6a56d5e394ebd314a409845a87bb0940629f0d557aa461d0f4962e33
untagged: certbot/certbot:latest
untagged: certbot/certbot@sha256:1a977cd8cf776e5dd1ebe7075ff3b978f13fa98f2330a500590ba2b7b67dcfb1
deleted: sha256:28ba94451daade77e5c3823247a8ef80af0993c1181663f2ca6d757b858f246b
deleted: sha256:a6bf1ff46cd2277db3d6b8affc3a3ee791558b5661a8a3d331bf9c2738797f99
deleted: sha256:f1d8651fb3b893b15e3baf49583a97ce3b4f25a0328649c56f8bebcf2f6b5721
deleted: sha256:50b4af13e37ac849c80247ddfd87b7d5057a81810751524973be935e7b83e121
deleted: sha256:fcb2e23b9fc8ac9a4a660f4631f4d0cd675a453b21f566e3ebe5275894d5ef1b
deleted: sha256:9c590e7f99340a437273777469ed28cfbd0fb391dfb67fe366c7603b686f3b13
deleted: sha256:591dbfe1630a09e29a01799582c78e80cb986b9d772a16560f3cae770eea8dcd
deleted: sha256:246bebbd1ae324454391cfba0348f5bf713ab289a51cb24f1dc21a7df8bac2b6
deleted: sha256:b085c7ca1a26b7e9378530a6365f3c4b05843dd05bf3e66ec92d1e7b8774dd96
deleted: sha256:0fa01f5da82860aca24291a6968b629a42a28ae7aefba6d6544edda30fa7f508
deleted: sha256:e42322c8332483942ac189547b9180245060f85e814ddc76108f1e9c47a75d31
deleted: sha256:2e554279a0ce5ca2453951c3e579a9fb9fc4378eda74a064a3e6defb2d931778
deleted: sha256:5bc340f6d4f5a3bc999dfbc790a0bdf0920b9103ef794645034de4260ee4e9c8
untagged: otel/opentelemetry-collector-contrib:latest
untagged: otel/opentelemetry-collector-contrib@sha256:0dffde3e58fc3f255cad861deb62d7c51505ce961a8a7e2107c7fc2668b774a3
deleted: sha256:1a1220cd53a63b0eda1f4caa223fd0d161bac8d376071f23e045cfc32f511528
deleted: sha256:ec3065b25212260c408d40b5c803db2bd88941693fca9c970fefae00eaa931da
deleted: sha256:8b4436532189e32213a0b9de37831cd639d31aa68dc159114eda2127de1e4d6a
deleted: sha256:9a6f616d0cc7f62f623bb83ab328a1ce9987a91f3b2ec19b0bac7114317b58e8
untagged: docmost/docmost:latest
untagged: docmost/docmost@sha256:57f1c29a75590ba100b2d769e0678c8b8fd7ee39fca09466bdb22cfaa324ffa9
deleted: sha256:fe8e00c4f83c50dc0ccca079a90dbbe3f425c6497779f417312b7f896edf23c3
deleted: sha256:98b46b136660fafe56740a6eb789c96164b305fec77073688265a41fa2b07b4b
deleted: sha256:030c811989df7079c6081b4728d1f3937d2eb80c8364a424e29fe9c76ebb17a4
deleted: sha256:323ae9c8b23fd10710b1aff1aa112e6075758fc78d30280a10fa2b7e4a8544b5
deleted: sha256:99cbcdb69cdd078e7d5fc2b4e3f535f7d12dd9cacf1ba3e4e8ddf2bd27e39282
deleted: sha256:e8aa5285ba7291ed8f71857f9be248da87a7f2cd16150b19d99584e5b8dece58
deleted: sha256:28c2ecc3f857e3c7cc3e1c1213256e2fad8b90e8413091c5c56ce261e425cc42
deleted: sha256:dd32fdb77ea0986a8991c23f84909d78c162448f72063dac16b4ed620c5a3312
deleted: sha256:74123d5aa1b80ec8e05ce74e4a67d835089b045cc668bd5064ff07f0c65746c4
deleted: sha256:72917b746eada50f29aa8c82a659a9b5fd070295036c0b1ad6d5f31aeac557bc
deleted: sha256:8458a4af568467c26df38e97d746e36e83d27cc250f1e11e6491ff8ca592141f
deleted: sha256:d178d7aba946e8563ba19b7d8479dc35a42632ffbef6e7e54fceb781d4d9eeb4
deleted: sha256:caf386872a3cf349b41d040fcda031c11c444ea1ab35198f0d5cc601cf2861ff
deleted: sha256:6655332caec4297afc82ff4eb3d90b1f69c686f6f26f2630b8005cf0bd566ed8
deleted: sha256:dcca8ed5742ac6575821506f132e964a48ffb931662212ceeb1835e06aaf90c7
deleted: sha256:5c984066030c52fa3bce9e79019af2c996f1504c62c30b7d9b4d5e86e30c6f7e
deleted: sha256:24300c8e604414ff92631e47793d27ac348ada50bb937ee57fa267f423414b62
deleted: sha256:02f2bcb26af5ea6d185dcf509dc795746d907ae10c53918b6944ac85447a0c72
untagged: harbor.freshbrewed.science/freshbrewedprivate/myghrunner:1.1.16
untagged: harbor.freshbrewed.science/freshbrewedprivate/myghrunner@sha256:53a11f4502976e9e8e2b68c5cbdb3129de223892eabfddb5b62c8b5508e961e6
untagged: registry.freshbrewed.science:8443/freshbrewedprivate/myghrunner:1.1.16
untagged: registry.freshbrewed.science:8443/freshbrewedprivate/myghrunner@sha256:53a11f4502976e9e8e2b68c5cbdb3129de223892eabfddb5b62c8b5508e961e6
deleted: sha256:1785768a7b85505fcf20005e3ae34add78ca5c0065ff5ba1be2d3d0f705f1b93
deleted: sha256:40fba8cca8f49a39752e44601277315e4379e717e4c669bd2d189a05660b9339
deleted: sha256:b0cd49f88332f7cc7a69352c0e0c2b0a138209d89041be4e7e9ea74844deff8d
deleted: sha256:4676261da88fecb9d15869eb991e7cd655c7ba230ba39ec1964cbf5427f77060
deleted: sha256:13e8183fa974f8c0bc59a0681d84aec5ddcf6f78dadd921fe0f4d48c8b89745f
deleted: sha256:c0a907f739a0681177bee9a92ff5a92f86a8fd0ed321c2aa4e1772473f283548
deleted: sha256:17c1fd0ffae17e51ffc3154d8d3f17e0573e8730449c50e402934440121d9cf3
deleted: sha256:e0da7e11acaeaaf65e17c410f1e5abe32f2d88e6ce79c788043838e8f73c6e40
deleted: sha256:d3259b9a9b92609d404c97c0d58d6d6b1d4e03ba4a0a486ba0c4fd7971f92579
deleted: sha256:ef981954873a0e03303da10fcba12b069c608e6cbc96e8544e61745b9fbc390b
deleted: sha256:30c7b23620146afc92458b16bc81c5b61a2ee29081634c873d3b3e1f492974e6
deleted: sha256:58699ec6aae9be631d9efae537dbc299561492098bb4d944f3514e02b10e6dab
deleted: sha256:ae8fd34014414097ab3a939a70052ab15c6d12534ab93eb12eea9f65a5e7ae90
deleted: sha256:d02ab944466e8e3dc5abc4b70d4ba5294bca2512f6d698f75c630d96635add49
deleted: sha256:246a831d29a8a79a95f6879d8d384ac4c122cb0c5573d35a20d401755f22efc5
deleted: sha256:99206c93be3c50896dc9ee1cc72e44907b2f8e94ba184c9a3b1bb1a431262cc5
deleted: sha256:df92da010899964aa635021b244c57b7df43219498aaad71709a04c28b3ce1ae
deleted: sha256:c61bf0d1a8bd60414cd3fc69eb475a7a988af25b67c3afc9d66a8b8345740b72
deleted: sha256:4a932cbf8110eeb15dbc1efacf05b10b86df7272632eaa58408532e5530264f8
deleted: sha256:422f957b32ad644a893facc3971d060a3838981a0b9e97be0540b03b7a6bfb54
deleted: sha256:5479e7e6dc408696a748354bccfa56ff204cd09cc738b30a24a5fb6fa49147a8
deleted: sha256:7dc3ed16ea664ef4c0e61f0a1b535d522c76e667702c20224c583bbb41a18eff
deleted: sha256:530c1b97d1fe00eb93b452b37779a00100395cdd5b65af0850dc729f11035f83
deleted: sha256:a890edd617165969a0160ff0821b1884aaa957e44219cf01df74414315ff676f
deleted: sha256:298e1ae651e8b8780cb3ac31e8fc1bf7dd9e0a24624cbd502cd4da2508c57711
deleted: sha256:b053e3bc756f23730e216d6e95ab8ff4f64fe4d20e25803a816094b0ff782160
deleted: sha256:1dd51962ee6aa7db9edfecef7f3282571ddc9fff9bb218e4bb793fd9a6591689
deleted: sha256:0bab0fbf5950dc3f73984e77204bb95bcf453536e6c55d9fc580fceeecf06681
deleted: sha256:97ea32edb76a038ffaef8320e9ef99126a0a68bb690b0f8fba77332f3711cf66
deleted: sha256:5ef2266864b1843a2e4960b6449a16535ab6682a43e29f17d01f03b2ee859f4e
deleted: sha256:28da0445c4497f3ecb56288bd74d91ed1ff6f86578d1d0f6f9cb2781915163b1
untagged: supabase/studio:20231023-7e2cd92
untagged: supabase/studio@sha256:8073e89c79f8823cdf9144127d391b86e60c22ed77b451812d0e5dfa2a517566
deleted: sha256:a0b210df16ab9563bbe874ec78d7955823cd856b03104db0969335a9fa466a9e
deleted: sha256:3f3cfcaad8b468cbff79979e9aabf6d31e7bacd194031f81aab154df738431c6
deleted: sha256:044768e8d00a531153ff05b269b43fdd4e69071d66e552f3f73289ac775de107
deleted: sha256:e23d10c211afd869b5588629b531caa64f2d833bad5d69c045d56307de40e816
deleted: sha256:a0af545aabf760df974c253ec952855b35faeccb0938a88ed25b17dfcfb33f29
deleted: sha256:9508142f1b9c2c9bb22027722ccc3a9f75b73a496e48a4563ea3dea28cdf3aa1
deleted: sha256:ae25302ddecea40f5fc6a6b1b40bad8982bd7dbb585e58798a9bd7f091db8002
deleted: sha256:886d42c811ea05c44c9f7c302805cf949e08186634b61d7fbeda622b370e8799
deleted: sha256:1ac9cf69fcd9041be91d6bf21757326836aba57c86cf57d89193d4bf72355fc6
deleted: sha256:3ce4d819517d441d27af9ca642e46610fe6f56e70879064563daae7e2cfb7da4
untagged: willfarrell/autoheal:1.2.0
untagged: willfarrell/autoheal@sha256:31f580ef0279eaced5b38d631b08c474d70d8403c1c2fdd6ddcf2e879d5f3f7c
deleted: sha256:487a6b982637737434edc432eb386d7d1d4f15a437275ada6261da640511c722
deleted: sha256:f324fd8c7ac167aed71567f5b45bc63c9b9667914cd213260ee24d2f9aa98796
deleted: sha256:2ec0948b672f79d39ace9d28272fd01a7e81aaa033d3d67b13b92e1d3d090c32
deleted: sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116
untagged: 1dev/crane:1.0.0
untagged: 1dev/crane@sha256:2d21b3c0d5533d4f6e722a05f1ccb078d4a414a4547ccdf881122cd2c6eb3103
deleted: sha256:3724c47968952775353c0061b2d82db16e8008db6a69ac05e71ba3c1da8ec6f6
deleted: sha256:92bb488b8bd4466a3b229768c6aedd414b549694bbeb7c300d35dc34a7c2cb45
deleted: sha256:ee0d219bcee9c80d529596b0f4a8fbb3384c60f4082782e095c4e0145ec1cc55
deleted: sha256:25161d4251f20786ccd6cd0dc2e8829eaa362e595a3955ef67d5429bca34cab4
deleted: sha256:c4d1cfefb2a1af664d2b6836bd7dcdfd52c28a2c0ef1818e230c8aa5c1521a60
deleted: sha256:c455ff9a6648ab90925bc799231d85776c8c373c25aed889e73c302e40c0b786
deleted: sha256:44043657805a72c00d6f483e5e1c05211be291b47bd5edb2019f8b10206b271c
deleted: sha256:1ba79e3210ffec19b0aac78261ef3b56c9d572a84ce171d99f8c5e9f8c8ceec9
deleted: sha256:92c2f7f2279d20f21e80a81748d3e535ac005414c173369bad07900665f4ce38
deleted: sha256:7c9bca19ee3fad0b4f850527fb2f75115a075b6ef771b57f995c3322cb2bb64a
deleted: sha256:d1619da8540b89bfc797c583754f20f42e0846611104a59fa13701cdf5842255
deleted: sha256:0ffd82d96dd6ccfbbbb6bbc2492bd3d256b8cdb01144e8140a4c55f12ae835dc
deleted: sha256:18fdb66c6a41b89620c96035277ec03d4069d27ae0b174b0e89f26530eeba864
deleted: sha256:f96114e9454bb8b5edf548870b385293d170efffaaf27ec6bca0df5396b830ef
deleted: sha256:9300705518a3ff9222e78cbbabf1108e0b3336c28a1d40b05cfea89bc41d1bd0
deleted: sha256:e023e0e48e6e29e90e519f4dd356d058ff2bffbd16e28b802f3b8f93aa4ccb17
untagged: darthsim/imgproxy:v3.8.0
untagged: darthsim/imgproxy@sha256:0facd355d50f3be665ebe674486f2b2e9cdaebd3f74404acd9b7fece2f661435
deleted: sha256:25d21ff36e2b7b66f8202e390b1919c5d03aab47fa98d33a47a3d33a7b89b4bd
deleted: sha256:82a950fdc3095342f84438b5edfa8925f8369a103f7fc43bccba8cde63eda0d4
deleted: sha256:b6c05784152b2d92cc73a1e600b9e50af607ed3b0c716091178fd3e6321e0373
deleted: sha256:3356ce12d1eee6b98f5cf236cfb7b8716f421e230100be00264184facbfe738e
deleted: sha256:f09fff3d32fb78f8d8fb090bf785e3b43ad112fb7338269135a11573fbe48f55
deleted: sha256:86a1ded6560dbe0d55d7c37c954a882ae85844ff381530c89832134922f2b434
deleted: sha256:fe7b1e9bf7922fbc22281bcc6b4f5ac8f1a7b4278929880940978c42fc9d0229
untagged: ghcr.io/medama-io/medama:latest
untagged: ghcr.io/medama-io/medama@sha256:1761c0f9881a8b90b1bc9175518c6c50122ac26f2899d825bb7d9ffa0a48ff8f
deleted: sha256:55b3405ed9da0fa3d144c7c4853c39de40b3097fe65caf0b9fdf6094984b9c92
deleted: sha256:aaf7c07a5a100ae85be5588e26dbfe26ff096bfa7c051771c67ce0998f84c983
deleted: sha256:75800b84875c29f738aaa8c26571c845411793467a33003dd24766f4a59271f1
deleted: sha256:695e061e19302ba4b1888d5d06c49ba6d87eddfad124f7ea7f2e9c05028d38ea
deleted: sha256:70b426ac8911ab9854f132e210935fadd0ade32e52bcf3192a3f10251264c8bf
deleted: sha256:5e71562fba2af69f5a5476bdaf9cb35afd3a7e6e39d830874558c8cc0f23539e
deleted: sha256:c88e1d191528cf57d9ac1435fa30e5edd858610f89582c1ce72aed53a0f819eb
deleted: sha256:e7a13f8291b4bcec5783e8e6e256afa42771faace25772e491052bee70955b68
deleted: sha256:02a6654b341f07345e6541f018a7e55db184b7cce0eba514b40493e1873680e3
deleted: sha256:cb11e9aa0d896a09a96ad874dc54f09160a74f323abebe825ead073c8b1dfe6e
deleted: sha256:4f48b8d75fc5eff343f3162ede57a2187f3865147bc65d05debc47255126a4c7
deleted: sha256:62d18fc69e93743de5560c1ec2685a9309965a138d38768db8ce7b4ef5285bc7
deleted: sha256:a80c7dd83858c74cd00412d86028d067092df090cc407456ec2d94280304c081
deleted: sha256:d64f02510e390ab0362fecf748bc296f16174bc2607838021c027a030f1b2c64
deleted: sha256:997c88ad29ee1fe7a8d3c3905beebbcde097f39d526d9efc1fa4ce1c2bce8f0d
deleted: sha256:99367afa9f8de4dfeed70ecbb586a0d015c20f0d1fde3f005898017d8d79ab8c
deleted: sha256:ad5fdf482a25c8eae66f52a9fe0118e1765f66d083c6b359787aff1b8eb5de7e
deleted: sha256:d5c88c2eeb066010069610c970cc3444201844f11006b5f00c1d5ee7463f7563
deleted: sha256:da0c4410e542ee7c2bbe746c2acb059e34fd05bd129d88274cd0148b0240e30f
untagged: postgres:13-alpine
untagged: postgres@sha256:d0fee67085a82b72d5b660d75dc82c60cef5fe5ee0ff652cc4cc64aa9da57cb4
deleted: sha256:d85ee7d22cb8aa109222cb78a2b28c385478de832c46b32d66e59154c35d2091
deleted: sha256:9a93da234dbbff62e13aa27d41048e1ff040deda889a6d9de4b7607be7018dd8
deleted: sha256:0d7a757256a2b8fdc2094891b43c0c64bdfa7fc4aa32097bee813f0e42914e79
deleted: sha256:aaf5056394e81d5d9a3ccd2a3dd0bfa18af9dd3a62ee8d296b6c4eff1d82ccd2
deleted: sha256:2d09363175ddca1acc2a85f5c7e8b9db3918ab3b710d2d28e063f3fd86aab417
deleted: sha256:64c7850f815081dba16063a9fb56a97b8bebc38d9ec98db0d0d3e58f93e2e446
deleted: sha256:960400ffbc19fd8576dcf278982932d0a2e1d4e0a5a8a6554ea46e04252e1fd7
deleted: sha256:83383b02dd4ae32678b35e44572c767fb7adfdef7682d564141052f8790a749e
untagged: gitea/act_runner:nightly
untagged: gitea/act_runner@sha256:de0e69d51958c9b27175bc392c205dafb04be6ab2b17b3b9e5c018c9042de9a5
deleted: sha256:ba7fc78926e030a1355ae0dfd1716bdcf456b6009138ed9de627d5a970d56443
deleted: sha256:e429c5ca7a8ae7d5bbe3a25b5bdb954f4875c1f23843feea866719b457823e2e
deleted: sha256:6be768e1f423df72715525b35224a8c5fb0b73538fbffb10c3e96561147551a3
deleted: sha256:9cc59626114bbd072fe5e30771e2824579ac2ac7d6911fd17d8250a2d376f64f
deleted: sha256:4693057ce2364720d39e57e85a5b8e0bd9ac3573716237736d6470ec5b7b7230
untagged: gcr.io/cloud-ingest/tsop-agent:latest
untagged: gcr.io/cloud-ingest/tsop-agent@sha256:b0efca60fa4dfa659e36bb88c5097ccca1d91958f97b5ec98bd6d8b69918a228
deleted: sha256:b3d071eefb00d13c62317b99915ec8eb2d90a79e278da15b7cd80dd23c35709b
deleted: sha256:dba68d012ea77bc5a44fcb608e50c100816e625dad0314fbdfc6040956d43d4f
deleted: sha256:e18e2bcc9504115235e22a411d1a270d08abf3b59ca06f57a994ed1f2fe98eea
deleted: sha256:0fd6cce84098d0e54b043dd172db9dfac2c7db2016dc0e8d4a640c17e2b34d5f
deleted: sha256:68cfdce2de244df2e4d7ee1b0370d652ae6142cb9e5f375e98f466f3fdb454f6
deleted: sha256:6fc8fbaef2c476ce27e143e8ec9f027d52ab887190583207a1a057dd51bac83d
deleted: sha256:0aacdd7203595ee636f587dc8a312270b4869e8db366a6ca7077e3edd28ac8a8
deleted: sha256:66907f7106bfca4e707599d663e431d578a015601c1da979f0458681f39b4266
deleted: sha256:e5d459736e56cac8c22d9db03c11b1968338f6c1e6d23402643df29f3f27b357
deleted: sha256:9996e1a64613d62aedd016f17921b0badf9b291ca07e301f2a8f686060ac1329
deleted: sha256:6f869df764571eff302c7c4993bcbdbb0947323d7332d74341c5e1414b1533cf
deleted: sha256:f224036345531ab219795f9a4fe160c8037f2bab4f6b0ac920b4d4770e27d9d1
deleted: sha256:c76b6113c9b7eead8b26c8be808c2a18e1ec90a46562bf7f20f09391b0f7019e
deleted: sha256:ac307eb16f0cd7d105ff7ca0f4af469124e07052fde1c1a98f32609321803542
deleted: sha256:f91ae064f7b4f850563d6baf23f3b240f0f41e7cfe00f972649e9440ad36bfff
deleted: sha256:b1ccb696758ca2747a3e29a6e091de9897130f0399fe3b65a7e53c6f755db0b1
deleted: sha256:7ed50e3f2f092cb6d3123cf2f949a7fc61f576000b13e5b1391b94622d9c6c72
deleted: sha256:e13fe89b2b8ff1ea172a16371ef34fcbb9f262ea6529a21f3dea760a2afc7db3
deleted: sha256:78561cef0761903dd2f7d09856150a6d4fb48967a8f113f3e33d79effbf59a07
untagged: drone/git:latest
untagged: drone/git@sha256:82182ff192f26236456951e13e8641eaed40f001f1461c1cd3f45140de0e37ed
deleted: sha256:b263385b74bdfe8cfd37e6b2415c08f0c40a71639e77bdcd60cc46a334627357
deleted: sha256:0c3b3a18acb035703f6fbdb652d4dac8972bcc0ad7dedd308a4c7ce6ac3d9fd3
deleted: sha256:2feedf014209559ba739ab1b34c2250b816e380dae4948e9dfa5940ddc517037
deleted: sha256:1ad27bdd166b922492031b1938a4fb2f775e3d98c8f1b72051dad0570a4dd1b5
untagged: easysoft/zentao:18.10
untagged: easysoft/zentao@sha256:65cd5f01db6d18e89c6594df30f7613a10cfea9016e2bc2adb05506f686991a2
deleted: sha256:4cae640bf3bf7f4102242a361fb011b6bd384669911b6e81df1e68a1f0c646bf
deleted: sha256:59ac5d1ce0f2f6875559a198d3bd131082a94a102ad438ea9f680659f22d924d
deleted: sha256:b1bea04e221bbdb396dc6e084e74cabb5b8bb8c88bcd34126d994f915924c21f
deleted: sha256:0ad45c2ffcc535909311ab61628bb1a87e53162a7374176df5b4e195cb1cda43
deleted: sha256:f4bfba3576945594389a65499ea4b938d2a9fc468482edbc72786de119c69cd4
deleted: sha256:6613470369321f66bc0abc081df95484a3098f34631d9dc42ff707c2d326d549
deleted: sha256:2a53682eefb7a2354dc8e7729609a972417bfa9d423e390a6f4fbfcd93dbc1c7
deleted: sha256:757e0b1f989ed97b9507dd19183aa32ae187b228e9512d35ecaca63a1682b11f
deleted: sha256:e135e35444eb793131077bc5049d259d490acaaa2feea32ddbf7e8dbdaee0e2f
deleted: sha256:9b4deba52efa0ef1e56745828f54deed747a7d6f4ac0c461527c7366aaf19af6
deleted: sha256:4e5da69e34158ff3d57b794f29b0aa1de23f2d1b4565a7d14b4c9afb50b901f5
deleted: sha256:8f8f23af474cf621d0162d6f76f82c877dbf3aadd3cc81a9268dd2bcddd59976
deleted: sha256:cd36bb5745ffc334e95b0f35bd1335eea07c424d18dd0c82ee823f1163ad322d
deleted: sha256:80d32ecdfd4c233f76ad1a2d2c4fc885b257139af5b728f4e3f5e1e11f36eb25
deleted: sha256:65e3cec1fda517b459a2d7c2e11bdd4368116e0d26e0907bfee37a90a7375ec8
deleted: sha256:aad126697940bc811b63fa880e0051d83c884c56d80c34cdc7a5d7bfb88158fa
deleted: sha256:349951e89bedb84e2093a00bd331b635ef76320b3b69b7abe1291a673deb6ace
deleted: sha256:90e3ec0690b1b39125c3c5ccf12b1f02732788e625b692b0040f9fddf88cfd50
deleted: sha256:1b6fd3ad4ce602924fffb84437331a255e2a9463531a1bd92a15e9e3c4d11523
untagged: supabase/logflare:1.4.0
untagged: supabase/logflare@sha256:e693c787ffe1ae17b6e4e920a3cdd212416d3e1f97e1bd7cb5b67de0abbb0264
deleted: sha256:989590782684e8502d1da3bf673089500483baab43682b296985abc6b67ddce0
deleted: sha256:fb3eee5e1e16a4c91b62010062472ca661475d29f6474844e691cb5a01da8515
deleted: sha256:baa917f0afd928cc437ede181096b58dd66dd3f08d0635e41e613d407247c038
deleted: sha256:5ec588fb0627f614d1ab61ef7f28d53b34dd6834a64cc6210e3f21bb185d1a18
deleted: sha256:60cc8b0001ad2da261640fac8cebe9dd67450efaae4e93c734f5f8ea64775245
deleted: sha256:5c518a35ba96b9b76ef07c7bf3035557db2650a72477b1e4d209bed23303d2a5
deleted: sha256:f4e3211d242e0b39683a6f02677ca11e61ae16d49043598aa8945999b383dd47
deleted: sha256:c290d053680c34a272b03d84b090340fa8acace2e31cd52b88cd1c5f21ea2397
deleted: sha256:29dd1fda68bd924a966c4e01bc483d0981f303720d1f21e18d92a5df11be1c51
deleted: sha256:ded7a220bb058e28ee3254fbba04ca90b679070424424761a53a043b93b612bf
untagged: idjohnson/alerthub:0.1
untagged: idjohnson/alerthub@sha256:8917f6097fd3fb3b07aca375be85b828e612ce8571d5786e391c5d0f49e7fdae
deleted: sha256:6465631c6a888987492072a71be8bbecc1f4fd2258cd5db4e4733b4c4ff89dae
untagged: timberio/vector:0.28.1-alpine
untagged: timberio/vector@sha256:4bc04aca94a44f04b427a490f346e7397ef7ce61fe589d718f744f7d92cb5c80
deleted: sha256:f0494e814793db7c889d85e35d12cf5613a4e080a8cbe82d6f6ece79c9f2b4a6
deleted: sha256:e8d83c01d41d65d139349b35726d6928425cb7fafff71a2af47089955b1514c5
deleted: sha256:d17cbc79336365aca652050be513f7459f8e7b9b51ea3dffc2921e3b5fed8a8a
deleted: sha256:a662b3874d05b5c531c883a8e9bd05d19552ea2e3ae9582be3df1ec550c198cd
deleted: sha256:2b6fa288084c03fcd620fd2fd416f7348193c40f7d2f84ac264d9b11fa0a23a0
deleted: sha256:1a4034831eebefa8db27e12301cca32e1dbb113c3b5cebb18d781f16a018ab6d
deleted: sha256:625523cb9cce7a1bcc1d03e3038de4843a27af3b92d859004bcefb0027291933
untagged: peppershade/pashword:dev
untagged: peppershade/pashword@sha256:958b99ec36b39cee96d5541e34c4a4f85ffaf6f3009fe58ef7a3b205840e311e
deleted: sha256:de78003385f854afdf1f880f97d83b6772e9c6ccd762725775942ffc99672f41
deleted: sha256:ee5a3eed7f51f34023ec7d2c972211b57272655a268871e77d6c54d7c5b50225
deleted: sha256:ef61ba48c65eb659ad773bb658a7388fd270f7f47d814f53abb7c25c85174a54
deleted: sha256:cc3b7627d8844a501a9b6730994251e825abb21dd8794c0387413eda1b82fbe4
deleted: sha256:31e1b2147f9fca87b04cec713cbc3d253dc5bb41cea2670632c90ea8fdce7086
deleted: sha256:31fcc3bfed118a9cd558a6b3de958d53307a648bbc10fcd4337a6a779b8aa1b4
untagged: rundeck/rundeck:3.4.6
untagged: rundeck/rundeck@sha256:9fa107d48e952ad743c6d2f90b8ee34ce66c73a6528c1c0093ef9300c4389fab
deleted: sha256:4969c5ffbb8b81d45385f977fcfbc972abb0bc51bf8dd02e9a9884af556b4028
deleted: sha256:f40f4028b476a9ea368768b7c15de4283db5276ca8093cc3ecbed0faa62c2888
deleted: sha256:46b1a60aa8073450545f70bb325a62d3cccee2ea9c73f10510b4ff2d3d00c4af
untagged: filebrowser/filebrowser:latest
untagged: filebrowser/filebrowser@sha256:7428a67e0265f1550e2421bdc554d35889921f5ba2285b829693053fb98aab8a
deleted: sha256:8a7d1fc5351a843315ade93e1bd7ba756610ffe9b72ea58153d051690bbf4480
deleted: sha256:704c67ebcc7f8b4f39c4266b50b3b743c39c88407237f05f708242c6fa54ec3a
deleted: sha256:5807fef293a33d106d3e3387b2c8836b933c32e7748bfba97432f6bf428ab1fd
deleted: sha256:aaddf6e0c31c678ac4de345cb08983e0dde3c2b55e3179cd15e952cc13096e7f
deleted: sha256:e7299d483998d741a3e1111b2c7b071ae1c5eee0596e852e816fa24d147e1977
deleted: sha256:d0bf41472db96b345a22ebef425a4239937af87146815538a80dbcf8976b0347
untagged: supabase/storage-api:v0.40.4
untagged: supabase/storage-api@sha256:6646a6cce0eddaad996ed97fdb3f6903aaae6af87dbf19474677e78b1a08e287
deleted: sha256:6b6caa4ca77f604dc12d3b1bfc6326d8c36114975b04779f7f40f385ec3b3a61
deleted: sha256:145e81c9de79ada318c5c77f72c69a6acc8258e653580172b9c89e1ba000f69f
deleted: sha256:97060b1624d1ddb818bb0b26879b5f2a48f89b4ffce662372f38f2022f03529a
deleted: sha256:3c88f3d2e5f94cdb42a45053d4cf19a797696b52325fdcb383322427e3b25c53
deleted: sha256:42aae3a4915b44b1bd1121e49fa3b953376de6cb945f6017b21fe861916d8983
deleted: sha256:fb6eeb89ab4e02f22d14a66ecdbb6635ad2ef6f7b3fd5942b8a9f79a63acba58
deleted: sha256:3a478854156cfade812c6188a487403146f4f9ab43a5c879a4f13e3a69a3ecad
deleted: sha256:ed2d12b32aee2ecd8930fca5edc09f6ac0a1ce90cfe270fb98e7241064adfb75
deleted: sha256:4a341059fea40356ff6cae570df0d087116a607187fd2aa77cd3bc2cc8967e25
deleted: sha256:78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c
untagged: supabase/realtime:v2.10.1
untagged: supabase/realtime@sha256:e217412dabddae0ea838127350e52b71e82e96b24ab171713afcae8dd0bc21ec
deleted: sha256:3737100a14c58bc1d6c1e10e0842dbdb5ffef0b0c3ca8c60ae0b557a8b2ad701
deleted: sha256:ab98b0037ecee0f5ea2bc4ffdc28cc4f99add9d772626dac2dfa18e82fd4bdac
deleted: sha256:5834d5bad799f7e57bb125e2744c336bc1c789cc7e82e9ea9db3159b91aae3b0
deleted: sha256:febe8d1ea65f622f9641ba20b4286990f2643df5e4430fd41a574a1e3d87d442
deleted: sha256:9a8c27dc5e3bc47fb29655dff29d392dba8480f1848353ce321cfd89dddfc6f4
deleted: sha256:00e052784ec434420411ed1d3cdfddfbc1d30a0d0a3909c6e8d80326ed746313
deleted: sha256:ecd3e78035ef54abbdb77c4b3c2cefe5888b30852e475e6afc7a877bc4eaaabf
deleted: sha256:0db60c39e7fc5268731b7a8da2edbb64cf561e42c762ce1a1eede27776ddb7bc
deleted: sha256:e3c83e5212fe2c562c3f1afe0b0170c3874f96f3745c3d120dec32cfe49dbd75
deleted: sha256:a91aca952a99a5070404cefefa2feafaa54779d7823ecdb0ee455d384e6999dd
deleted: sha256:c7e81b76513cc43b1d18fa681e769c65bcee12db1500028b0b917fa989dc96b4
deleted: sha256:da82f98cb45acc952aefbd149be5d47249d9c48104fb89e048d811c025dc0187
deleted: sha256:650abce4b096b06ac8bec2046d821d66d801af34f1f1d4c5e272ad030c7873db
untagged: postgres:latest
untagged: postgres@sha256:8d3be35b184e70d81e54cbcbd3df3c0b47f37d06482c0dd1c140db5dbcc6a808
deleted: sha256:d57ed788c15419329a4d7d4f09f1f4a49bc0e0957c3d4709f31faf21b1e4c6c3
deleted: sha256:a01bcfdb7b1658ff8822a9d2872908edee7aa5a46486b3d455a6400e659d17e4
deleted: sha256:6d7262f11dc059d9d6b7d0ebf97c73746c491a1cd59a832922a0e8668a4dc677
deleted: sha256:a89de7d6180ca81e3a72485c973d23cf8955cbdcdf7d3e477e2e653312b1bc4a
deleted: sha256:93509499b214a36e36db1fe95b36f2b9a9c10ee971ec9590928fb090b7d53e4d
deleted: sha256:f6d573bcc82abacec1a7e03a5fd5fb009ed7b14458e4cdd921f4819a6ee7f2bd
deleted: sha256:62eb9c7a68ee352e300daf909dccd84bf0478cc63ebc0d1192a6bd7f51b6fb0e
deleted: sha256:6799e6e8fe41de51055ab38c916b5bfd53c0cea3917eed9cb7d80354f8e802df
deleted: sha256:2a033762b8bf2c6111367df647028c4ac0dd1a28ff850f6df330e270361485b6
deleted: sha256:53dfc4db7997c4e964b2fc23278104d311de5129da79309edf3b27fa0e4c8d3e
deleted: sha256:9d32e4552eccd213bf9252e3b0df259b781f9bf3c85d99f6140e28e8bb501335
deleted: sha256:2c43363b96364e9ad55eb091ec6480208c1a2dda0e6e18e9b70e32780a6a9b73
deleted: sha256:3b5d69eb59645df66352d3ce4045e45d1c382926d2020f09c7e1d6299de31725
deleted: sha256:4cc98da5e09a7dceb6ad8ea4f16d2914fb591b835b4e9eb0a214b33a31bec668
deleted: sha256:98b5f35ea9d3eca6ed1881b5fe5d1e02024e1450822879e4c13bb48c9386d0ad
untagged: openproject/community:13
untagged: openproject/community@sha256:e63075947ec06c5cb8d7ab8dcdc699e6edce85b7f7e597420defe6b453d21258
deleted: sha256:7c5b9c6059e3ba579f51bfaff1496524110b2d6326c677f0b3e20f76a20b3284
deleted: sha256:88574808ecdeb8d8973f27e101788dcb5695b2517703cf7615613f587e47cb54
deleted: sha256:dcc3795740826fe0dfd6e2c324cd7a1933a066c92154b4e76b27ea798c433bfb
deleted: sha256:47b8aa03e4d7503ae31b40ed255d6f14dd5b26c1e6a66660c2c22950f4efe14f
deleted: sha256:b0298dffe209702b9b0bd90dbc2bca6fec39ff9e45a0ed072abc969f2652c3e9
deleted: sha256:f86b56d792f6609fc8dbd3ddca949f3e148ec55d1ced793916114896124fa5c6
deleted: sha256:d02f10f583789b6ff05ccb8a568db34e2eabde1d3ebbf7beefceaa3ef6745372
deleted: sha256:56b6606c7e3ccb8825df74d456fadc69263e7e630c7873f6ac989027009be46e
deleted: sha256:9f4ff3a64a2cd9edeb865984b174842788cd4c9b81a94518c7e35505cae9c4f9
deleted: sha256:c4ad842c8d1209c451526584e1f2adf62b04c231b5590cbe7a0ff8f7cfb04288
deleted: sha256:6e438230555dd3ef45e92b6cf1cedcde024a593a44b669e75b6ec796aa1424cf
deleted: sha256:bdaca3b9968eb159c2946f31dd998b961014b8e95f2c31da802c452145990259
deleted: sha256:9430bb152837f80ea47b45b1ccbdf980af62438b1c96c42212c298b3e6f0fa2d
deleted: sha256:a2401d07aa03a646ccb206511871f460ed987465e4387f8529a0d2cca87bc101
deleted: sha256:cec5277ac2f99dc8d7e233a6bd91a74a8595f3a8fe9d1ea3b6ce05c1e882739f
deleted: sha256:3b701b6f7febe136daf7dd4129bd6145cb71e767d0a0b591e720af11f4067320
deleted: sha256:d12a1355a8ac793de832fe2ec49acf847d3f0edb2bb15236bd35a0adb244996e
deleted: sha256:e827302a21768f7212a25a7e6b172b114e78e3a0ddf604ab6fe8ff541057fb31
deleted: sha256:ef25f65e31bcc7adcd398fbcb298299157fb566690cd6a0a972bfbbb55fc6403
deleted: sha256:6cf14268e3f5bb5d62fb3fc3c891d38192236a760138a6e73f569543394a6cef
deleted: sha256:30cb5250ecf734e3c199c093fc74de7f6cfa992c27d49680531cf1d3c5c08103
deleted: sha256:5b791c5c9211f6ec8a2d07eb49b887c0b6a1017d8ae5287894cd99a754917f0a
deleted: sha256:e6dd2c5f4912d29bb69fbce8f9eacfaddd3ef78f556114617520cfc5bd40b3fd
deleted: sha256:89ade9f3e716f2e97cf2a1ee6fbcceca8bc9e83b03e7ccaf54798e42c568ebc4
deleted: sha256:e1325a2db1db2260452d6a438db928c7670d8ed925800e99e8919b570174dad0
untagged: node:16-bullseye
untagged: node@sha256:82ce389daf7218617d912a2373082fca88d6b47be8c72bb3e0a0e43daae662ce
deleted: sha256:cf2eae8774a2e6c2a24e4e97eda5230e4c960e36263fe0b2a40a40f5d30d2386
deleted: sha256:51ca99a62a6fe5cff11c9caa0d83f7338172719b9ec29c036b6b5375c9b26c11
deleted: sha256:01b57a21b50c32d701eab1807d5e3234ef697283edb0a93c7df17ab26b0eb0fe
deleted: sha256:7a9c8ce1ff1adbbbe0e2156d183a8b7d7733b90c05c3a4202dd9848ce30fb8f1
deleted: sha256:54fbc6eef840bc0ec2f6d6056d61bf26ddb62ecbc318976dce110c17928d5195
deleted: sha256:2298b1e7fdf1b8fe141ee4d7cd39d58a45cdcd984890627a919cd555bded1d17
deleted: sha256:4c8b85c423d73278fc5ed6ac818f92b0e791cb0c5b7b3c32a946a6112df4b4bf
deleted: sha256:c878afa097c72062fb9ae3872d392e1826a66d5c1f500172ff09fa8a883117ab
deleted: sha256:f370a05d981b3c9d2cbc1253d381ad6a5b0338e2a4bc3f29e8bc888929cedca4
untagged: postgres:13
untagged: postgres@sha256:ef119e5f4d6dd8be3cb83d3c155912e0e564088e957c73d23656726d63cd9300
deleted: sha256:78e992cdc9ff6990dd99867de94cecf894c347408eab791f1bb0905e763708d5
deleted: sha256:21dc1b75fde680b1a5b087090f55f1cdcb37b7bac411575a2836377960c0ca7b
deleted: sha256:3eb82dfefaf58c4f29253dfa50c115574137ddb1b747810fd16c2dc9625b2570
deleted: sha256:4d31dee964d47e316dca0dcc0f80bd290e45ba7e1b1ef5513bdfe0103dbfcdf7
deleted: sha256:5d4fc6a418b0d2c0ec7926f188b4491aa0af0584ff6b649c807dafd09eccf59d
deleted: sha256:3abdb8dd38f8d23b721a9137b8b28ad3208617fb8e24e934ef463d19ad92ec81
deleted: sha256:71100648489a4c89b57c70ec52ee6310ae440e3ee62cda9443b669f94787423c
deleted: sha256:58fe14e05d8ecd608cf21f33cd574e9f302c13c6aab7885be7a35cb16b0e2d0f
deleted: sha256:7e5043b4b82b6986f09472ba2bdfd9de52620640d6f1e021c61dcb0b5cb8c182
deleted: sha256:034d131c229024d8b1f3956a497d9ae83082752f49af076d2f7b992f62bd7875
deleted: sha256:b84cad7fcdf6fda78a04822fe77c314c5dc9cbe886185e1d8c87b1aa66c84e91
deleted: sha256:5a69b7d8e8340b2a34790bfa70527b736841ea91ee6d52bf11f6d4f8ddbac7d0
deleted: sha256:8df6e05f693bfcb7e60a8eb4986488a5c94bd067eb423d32e503b66d1dcf6411
deleted: sha256:a94c0f06b5f0031b9a47fbc2ff85e5e257641acf830b61591050dde2b20d37a7
deleted: sha256:fb1bd2fc52827db4ce719cc1aafd4a035d68bc71183b3bc39014f23e9e5fa256
untagged: harbor.freshbrewed.science/freshbrewedprivate/myghrunner:2.0.0
untagged: harbor.freshbrewed.science/freshbrewedprivate/myghrunner@sha256:3be5a88279854843a5c473d77bac64fc72604db795caaa58f4f13dddfe2d7f7a
untagged: registry.freshbrewed.science:8443/freshbrewedprivate/myghrunner:2.0.0
untagged: registry.freshbrewed.science:8443/freshbrewedprivate/myghrunner@sha256:3be5a88279854843a5c473d77bac64fc72604db795caaa58f4f13dddfe2d7f7a
deleted: sha256:e97fe542f6170edea29fec5b69c589b1a1f0b9f81f3e544adc61123a5ad5dd92
deleted: sha256:2e4f5583e327b7980a7b02332043ae1b2f5175ba88d99c9634a53e576eb7654d
deleted: sha256:1c10373a48b96c32f7a54914dd717ddfe2c3a1cf42ac5197ee80c7956768ac88
deleted: sha256:3f0f4d14353935a4083e6d79c035ca930df99532b9242e545e8ad9dbb0a386b5
deleted: sha256:84113f5e573314f78514a83549436e518adab508658df8742efdad94e91ced73
deleted: sha256:cd0d57ec7ecbd401cc4ef2c36530c08ebf55ea7dce28cadc6bdc717701ef08de
deleted: sha256:c9c421c17daf38b68f3f510fdcfbf56b01338eb027a2a1aa43496a455bf4b0b5
deleted: sha256:60ddb1080f466617c7fc95b7d82729f6e33fd3326aad0f7bbb195e4da26033f6
deleted: sha256:e59ed47ba47c7e18fc7e42f9c153624aed021290baf59dc85832d8fb7b9fbeb6
deleted: sha256:778d53271b03134e605d5bfd48f68dc9159aed09b838dd12310700de01830698
deleted: sha256:5bfee5e5b73e84d525bffff694d0cbed08bdbe1f1a50846f62b271f9525c3465
deleted: sha256:a4c87a6b3fc13c160f931a4a51aa3b28025660988156c69ff124a5b29b8fc722
deleted: sha256:e77d97465a3ab575212f410e8fb25593bdaf57f89b4a0980a09f019b639d9f95
deleted: sha256:16de2472dc576ce17195597c7455d929d1cc21e696a602bfdf2dcc8c8a6564a3
deleted: sha256:299163946c2d1cc24b8cdd050d3130c7ba20d7763ae263ba5cd671c14032e4f3
deleted: sha256:89a5b94d0d58427b8bec6f2e37ebee2160ba08a762747b3ce714dd253c76f58f
deleted: sha256:c5a02647c908c95558f89249685c734520ed71ef2e3834da8a3fa053b76cd77f
deleted: sha256:02c227218a164e893500ad35c775522fcf7765193897a47cee365adde7c1687c
deleted: sha256:ae3df0d0960e32f3b7d6825893c31ebe429893733d6325ecb130a2b78149b723
deleted: sha256:376521583bec8a15333e01b756418e37ae85076fe31e7e0563f23f1b89ec5a16
deleted: sha256:df9721ab6bc256878ad9a60e2ff0e030bcbc08f9cf79eaa087b4307d6512bb53
deleted: sha256:05f89667ecba9422a925279258d2806f2ea174ddba3539cf102da163a6587e1e
deleted: sha256:e9f7fc8a062795ddbafd21689c1ce9af26191a91fc5c04adeb7e7ca98112c862
deleted: sha256:6ed0af48b046af54530a125e5a757b75bcc3e1cb72834d807cc613152e9c0315
deleted: sha256:d101c9453715a978a2a520f553588e77dfb4236762175eba61c5c264a449c75d
untagged: plane-worker:latest
deleted: sha256:e236790392bc79fa85f7fd3ff594fba45ecef3ba60e3dec6e33abee1252ab3d5
untagged: pierrecdn/phpipam:latest
untagged: pierrecdn/phpipam@sha256:f28dc4fd109c15cfef57fd60d17203039c62c66cf095f1928394e88b1a6c2436
deleted: sha256:423e551d0b51298798d702a03e59d3bbb20ad358a68baf379b86a9e5c20ee443
deleted: sha256:e77085dbd78df6af48048a1affae185e68500c6e2e790159bab9865decc36a0f
deleted: sha256:e466e191a8932d8812be6594aedd2335171b76075d7ebba439f38201b4ce2930
deleted: sha256:4a071929cd64663402a8ac0bed5e12771a7b288e5c52818ae6b74d9200adde07
deleted: sha256:d3190e9ba2105e948776c8773b272f0a2542d87357664a9cdb3b71246756e77c
deleted: sha256:c10e70cc32eae7ecf537df3432d1132866d165ad347b7e8411233fb1bed0bffb
deleted: sha256:b188375c71c8953c65fa86af651905243633361c496dd37baa54824d4783f28c
deleted: sha256:ce5f057492536a7c3ba4e4b5e111d405ab513ff4cfe70630826f7eb1cbbd29e4
deleted: sha256:f3907839ffdb8a74bf4b291544af6908d516baa7c3b9d3ae11fd582d607e6717
deleted: sha256:9c95a37a5af1afba29cb416ac4aee7b57c6f482680d488ff278c557136530e74
deleted: sha256:5c61c9eee10aeb069fab0bd1c233da58f03e723b4f93a8f53ee566f976d1a6ce
deleted: sha256:7f8661bae300afb91522490c23d6af47f665dd05875ced85a816aaed77454988
deleted: sha256:befe0332a8ca952f4eff7c4bf05242da06d391af8ad763f095dd624460040a24
deleted: sha256:69e9cd9b1009646601a29579840c60fddbcb06405d953338a331b3f28cc4f5d9
deleted: sha256:42de46564d3c71d5ecd28813357247e4d5496d7853899786c43e63f561306237
deleted: sha256:feed12b1f992d8b02d6ea4e8f40014563f454a8f5c60356a7b89cfc4131127d7
deleted: sha256:9dd11482526128fc57e9f5faff58b4b9136bbabd370d4a82b38c68eef6ded045
deleted: sha256:7c5bf5aff9dbcb15604f3aa7e248da2848d5c9e6590be2e2aa21ca036ab726fb
deleted: sha256:b4a8f29479bb3ba4fef8acae251df87fbbe70234939210ba0d0745f7d1520be5
deleted: sha256:2ee746151beb7b0b3ae453bb83c6d0828ee5671a14f60160da05c663f6b4286a
deleted: sha256:cb7a8356ede0b61bd6f540981c5b1d91b7e5385412e36684f65f904bbf590c92
deleted: sha256:698904aae259d168b38dad3c8d8c6787b722e92f4b9dfcee0ee8a2ab8240f4c5
deleted: sha256:a953d04decb53b836e21dfc15a2b8f9bbecaad1085ef606b0d9db21b09078337
deleted: sha256:8a8f49ca28d2e5ca5aa9455c92331b26ae1e31d2c78a6be0e062429cf3812552
deleted: sha256:db10680b18837fbef39e1e6452ff5448803e1e7b0c13eda5d53b19743e09d947
deleted: sha256:87c8a1d8f54f3aa4e05569e8919397b65056aa71cdf48b7f061432c98475eee9
untagged: pyk8singresssvc:25
untagged: harbor.freshbrewed.science/freshbrewedprivate/pyk8singresssvc:25
untagged: harbor.freshbrewed.science/freshbrewedprivate/pyk8singresssvc@sha256:afb4eca89458adeb3b9626a3528e2b72ad2784813a70d90ba328019670e8be0a
deleted: sha256:2ddf7625d9e101661f32da573665139ec02a28f03e0fd524a4179e5b4758afc2
untagged: chocobozzz/peertube:production-bookworm
untagged: chocobozzz/peertube@sha256:b6733aaa5df8cf778232b6b665cc195c3840d4fffbd12b98bf7796c8bacd8c98
deleted: sha256:88a22412cc3b79339e4e752b3b6d393421d9277b850fa3ac40e801dcbca5a08f
deleted: sha256:209ea286c68520eaadb716395a8532b2818b6e26671dcf936bf13b0fb58d02ff
deleted: sha256:ce8fac178658e16e7dd9adb853885a105ccc28fd19d20d550b71adeed73926b4
deleted: sha256:24c247207f0ad63b666c67e2cfbe8e13f25edd597a612f831801d5a362975a39
deleted: sha256:3dd4c9f30a2d424c9676e5a0f3585cfc30de5cd8e1d9fef26ef9c32de1834bee
deleted: sha256:2bcf267926f270111ff8810fc52874688ae8403360a31ad88f73b763deecd94c
deleted: sha256:785cb974304f700bb70acb4d121d4a2df2f5d0cacd4cea1f66423641070025c3
deleted: sha256:dcc3602dbf7e9a0aae8b19debfb5cab48a0d79fe6660af41f314e53c257800d9
deleted: sha256:7f370e2a8b547faeb6f60f9c600a1148cf7f1106c4131bbc6ee71130e0b31320
deleted: sha256:98fd1c039425b1db06c5fdbb022e47c5c38fe300ca9498cf8873ddb81dfe7471
deleted: sha256:99f56f5df1a60f7367f48ee0ccfc8b11a7701ca96cc0eaa29e12147af648762c
deleted: sha256:db67e5df20ceeb9b423c9ede19fa708318b32ade58e8d5f324396c85b47c9fde
deleted: sha256:d986e034455ce76cbdf329890d9ab14690f00fcae6d70ed2028f0a1e719794b7
untagged: supabase/edge-runtime:v1.22.3
untagged: supabase/edge-runtime@sha256:1fccfd7cc761a00b0795341d0fa2c33a5d9326d86865d31a6ca39a75d72d6e7c
deleted: sha256:eea2af15906f468f4be5bf305372931c54e046d786ceb5c15df0ef565ef1938c
deleted: sha256:6336dd666c6851f41d69bbecd2e531244ee64a66f7e8e83944eeb3c75e5fd528
deleted: sha256:db120de0582c05824bbf85463cace30eaafc80a1040da0a88f3d8dab97ecac0d
deleted: sha256:c7ff5bc506167fa38b982108677f532f63364a6be5567ad2a0cbd1f16b295d52
untagged: ghcr.io/immich-app/immich-machine-learning:release
untagged: ghcr.io/immich-app/immich-machine-learning@sha256:1221bc54568a0779227baeaa4282bb0e0db50831b66f9ac9afa9d78513df36ad
deleted: sha256:cc0c567003a3963e55b918fd257d65a2ee4202f4ceaa201e85f62068cfa73297
deleted: sha256:cfd194bf33630e26e67ce15b022b4b500a502783000e8ffae870b943c6630ecd
deleted: sha256:746d9432fc97899fad54262a042903b417c867bd59e139134644f85b6538ee40
deleted: sha256:48524a1685b8176e58b23c5ab57955e506ad850c72b9f02e30229abfb6c03ecf
deleted: sha256:59b55fa76afd96300267bf65ca486a4e5103c1d555d9011e1a1f51381dba1649
deleted: sha256:6ffa4da690f350d16c9d9cd71fc212d829cd7a8a9ddc2f3b816f782381a5d360
deleted: sha256:92b48d5c8cb42202cfb0c780508f35dc72d086fc599c6678b43c3469dba048f1
deleted: sha256:6590d77ac9e0146189c1292492b3bb7c2ee66bef61cf2591c5b437d6980b33e2
deleted: sha256:05210daad66a0734e5378b779913e56013847b05fe04bdf6109ff2d7a344c876
deleted: sha256:f153facae9d08751f6a9025ff417f008b392a96891cfbccfe925967bf71d04f6
deleted: sha256:ce289265078c198ba3b30d7baad1423b97ebd8ff37cbc6afa40a25b1c03a3182
deleted: sha256:8d853c8add5d1e7b0aafc4b68a3d9fb8e7a0da27970c2acf831fe63be4a0cd2c
untagged: redis:7.2.1
untagged: redis@sha256:4ca2a277f1dc3ddd0da33a258096de9a1cf5b9d9bd96b27ee78763ee2248c28c
deleted: sha256:5b0542ad1e7734b17905e99f80defc1f0a7748dd6d6f1648949eb45583d087de
deleted: sha256:57ddb0b590e783cb033cd152d777e75ad20a2c85c7f85ea7c9f6de020ee30571
deleted: sha256:e933c69e3728b6c09699982b317493292aaaea4f3675a6a8465220a39a15bef7
deleted: sha256:eb773663f379528044162c09cffcdad53bf138d3d809f2712e4b4eaf18dde408
deleted: sha256:f2537fd0ae1a8db0d2377d60090233e6b7582a7694cc48764b77b6867eea888f
deleted: sha256:6509dccc3c3aae4ddd59ded011d26038435261a454f444e2ea10f72167eeb8cd
deleted: sha256:fb96fa2b1dcd8ec2e5b0d4b1dcf6e04b4d12d96a1c4e5e0e6cbddaf6bde56361
deleted: sha256:cb4596cc145400fb1f2aa56d41516b39a366ecdee7bf3f9191116444aacd8c90
untagged: minio/minio:latest
untagged: minio/minio@sha256:62bffc26326ee5c841d7774b1c94712953d315ee5ca603c124206cabc77681d5
deleted: sha256:b9ce366b76485f3ad772bd2417892e9b68f6d020d60b2aa1eb62a403f59e3ffe
deleted: sha256:164fb62df4aafbca9f8f1845eefce1c11164636a62dc531ef661de67b0683b10
deleted: sha256:d1e85853acf90726f8b7e1359d7b137c5b5eeedf192d7f47a41a5dad063a7aeb
deleted: sha256:f3aa011f7d969de6c08b7f06c7a564825da9ae0d49f4aaf75b8af5deeb070504
deleted: sha256:71ec098f0fd655717dce36470dcd56f46b8726353e1d6b113e6912ad49bb24eb
deleted: sha256:0b2cc2860e5a0d85eef5014a836fa05a11258dd14ca5d3d0522cdaec53c9fbb8
deleted: sha256:469819d93189dc7aac5ea5566186f429aefca380e5c0377aed651ae7d96437d7
deleted: sha256:6eeab094aa100ca6e9e926dc9d7dbd4c71ca1315386ca53f140ee88841d75e11
untagged: supabase/gotrue:v2.99.0
untagged: supabase/gotrue@sha256:b503f1fac70544bb5a43d4507e6a0842dc119e4937647b1e7ade34eae7dcdbf0
deleted: sha256:7af77f0382472c7f0c33a0c970998fb9cce214a3d58ef315e126a4ba77c3425f
deleted: sha256:026ef87348cbe4846d832434fbdd25a1a8e89e18e6d0eec8c48f4dcdf0229d2d
deleted: sha256:161764a745dc7f35abdedbcab9f23f7ccc8f52efe142a9e00105ee04d9aee17e
deleted: sha256:9bdf867be629b4e34c4251081af50c54c2c8debe028dc01579c82db6b72c2fe0
deleted: sha256:89dc87c73b3fc38f60681a6ef79002caee84a4d20eb4732e8dc8973ea50140e4
untagged: kong:2.8.1
untagged: kong@sha256:1b53405d8680a09d6f44494b7990bf7da2ea43f84a258c59717d4539abf09f6d
deleted: sha256:3cefb958bcd694597b65adc00053a762b1f9a6df3c6031af528d052d75a64e1a
deleted: sha256:c1ee6d625d988023e424aaf8bc308a6f965ec84b431ef8cf3cf909feb4534150
deleted: sha256:76e66cec8435ad1db5b948618a4d99bb66a837f1e274dedd9dfa2a549422db99
deleted: sha256:c529fd27f1ad09fd92a4dae346e4044be13440b9bd18b994ee6f23c86ea045a4
deleted: sha256:994393dc58e7931862558d06e46aa2bb17487044f670f310dffe1d24e4d1eec7
untagged: hello-world:latest
untagged: hello-world@sha256:c79d06dfdfd3d3eb04cafd0dc2bacab0992ebc243e083cabe208bac4dd7759e0
deleted: sha256:9c7a54a9a43cca047013b82af109fe963fde787f63f9e016fdc3384500c2823d
deleted: sha256:01bb4fce3eb1b56b05adf99504dafd31907a5aadac736e36b27595c8b92f07f1
Total reclaimed space: 48.99GB
At this point we’ve reclaimed nearly 40% of the total drive
$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 766M 4.4M 762M 1% /run
/dev/sda2 234G 106G 117G 48% /
We can see this reflected in the Beszel graph (as well as the CPU impact of my find and remove commands)
NAS Updates
Logging in to my older NAS, I noticed this warning
Going to storage manager, I was worried that I had a drive failure - instead it was just an integrity check failure
I set it to run a Volume check
As the note says, there will be a blip at the end
I saw one more warning
But on confirm, I can now see it is running the check
After the check, it still showed the warning, but the drives are in a good state
even if there must be some errors somewhere
On my other NAS I see an update is ready
I downloaded it but paused before clicking update now
It seems that with 7.2.2, they remove Video Processing and the Video Station. This kind of stinks, imho
I cannot see a good reason to move forward with this - i see it only taking things away.
Cluster maintenance
I noticed a workload stuck in image pull.
$ kubectl get pods -A | grep express
default expressuploader-6d5dcfc675-tjkdq 0/1 ImagePullBackOff 0 (2d19h ago) 156d
This is from my expressuploader project and we can see the containers in releases
Yet, for some reason it seems failed in pulling
$ kubectl describe pod expressuploader-6d5dcfc675-tjkdq | tail -n5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 52m (x787 over 2d19h) kubelet Pulling image "ghcr.io/idjohnson/expressupload:7fe08a213572acc5c17cac60751fb7b5122005d0"
Normal BackOff 2m34s (x17872 over 2d19h) kubelet Back-off pulling image "ghcr.io/idjohnson/expressupload:7fe08a213572acc5c17cac60751fb7b5122005d0"
The first, and most easy test, is to just kill a pod and see if trying again works (the tried and true “turn it off and on again”)
$ kubectl get pods -A | grep express
default expressuploader-6d5dcfc675-tjkdq 0/1 ImagePullBackOff 0 (2d19h ago) 156d
$ kubectl delete pod expressuploader-6d5dcfc675-tjkdq
pod "expressuploader-6d5dcfc675-tjkdq" deleted
$ kubectl get pods -A | grep express
default expressuploader-6d5dcfc675-qbpm9 0/1 ImagePullBackOff 0 26s
I’m wondering if I have an expired GH PAT at play here.
I can see the secret used from the deployment
$ kubectl get deployment expressuploader -o yaml | grep -C 2 imagePullSecrets
name: expressuploader-storage
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: dockerconfigjson-github-com
restartPolicy: Always
I pulled the secret and decoded the values to come to a auth block like
idjohnson:ghp_asdfsadfasdfasfasdfasdfasdf
and it was NOT the active Image Pull secret.
I repacked and re-encoded then just did a live edit of the secret
$ kubectl edit secret dockerconfigjson-github-com
secret/dockerconfigjson-github-com edited
You can also do something like:
kubectl create secret docker-registry my-delete-secret --docker-server=ghcr.io --docker-username=idjohnson --docker-password=ghp_asdfasdasdfasdfasdf
The update to the secret fixed it
$ kubectl get pods -A | grep express
default expressuploader-6d5dcfc675-qbpm9 0/1 ImagePullBackOff 0 8m30s
$ kubectl delete pod expressuploader-6d5dcfc675-qbpm9
pod "expressuploader-6d5dcfc675-qbpm9" deleted
$ kubectl get pods -A | grep express
default expressuploader-6d5dcfc675-sdv8p 1/1 Running 0 13s
The other workload in a constant state of fail is the GCS Fuse CSI Driver.
gcs-fuse-csi-driver gcsfusecsi-node-v6b25 0/2 CrashLoopBackOff 22027 (4m46s ago) 86d
gcs-fuse-csi-driver gcsfusecsi-node-pmz2s 0/2 CrashLoopBackOff 19728 (112s ago) 86d
gcs-fuse-csi-driver gcsfusecsi-node-sr5d7 1/2 ImagePullBackOff 21285 (5m20s ago) 86d
I believe this came from this September post on Fuse and I noted that in testing it only works in GKE.
Since it is contained in a namespace, just deleting the namespace should clear it all out
$ kubectl delete deployment gcs-fuse-csi-driver-webhook -n gcs-fuse-csi-driver
deployment.apps "gcs-fuse-csi-driver-webhook" deleted
$ kubectl get pvc -n gcs-fuse-csi-driver
No resources found in gcs-fuse-csi-driver namespace.
$ kubectl delete ns gcs-fuse-csi-driver
namespace "gcs-fuse-csi-driver" deleted
Now, in my other cluster, I have a bunch of workloads that need addressing
$ kubectl get pods -A | grep -v Running | grep -v Completed
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system svclb-yal-service-778935a1-b4gjp 0/1 Pending 0 234d
grafana alloy-kcr9m 0/2 ContainerCreating 0 107d
kube-system svclb-pdfding-service-11039153-2454z 0/1 Pending 0 105d
grafana alloy-6zxjh 0/2 ContainerCreating 0 80d
grafana alloy-7fzcb 0/2 ContainerCreating 0 80d
grafana alloy-l7bz8 0/2 ContainerCreating 0 80d
kube-system svclb-nginx-ingress-bfbf6b84-d8k6n 0/2 Pending 0 129d
kube-system svclb-pdfding-service-11039153-pvv8r 0/1 Pending 0 105d
kube-system svclb-yal-service-778935a1-8fcpz 0/1 Pending 0 234d
kube-system svclb-yal-service-778935a1-vmwwc 0/1 Pending 0 234d
kube-system svclb-pdfding-service-11039153-szb4w 0/1 Pending 0 105d
kube-system svclb-pdfding-service-11039153-7fnb8 0/1 Pending 0 105d
default siyuan-deployment-test-558dc55b56-7b6nh 0/1 CrashLoopBackOff 56934 (5m1s ago) 202d
valkeytest my-redis-replicas-0 0/1 CrashLoopBackOff 56948 (4m46s ago) 202d
default siyuan-deployment-test-57d48bb46c-6x89r 0/1 CrashLoopBackOff 56946 (4m49s ago) 202d
netbox postgres-deployment-7df44c4688-2nzqx 0/1 CrashLoopBackOff 5659 (4m25s ago) 20d
reduced-to backend-7d6785cbbc-vbwl4 0/1 CrashLoopBackOff 4455 (3m2s ago) 16d
patientsmvc mypatientmvc-deployment-79cc598b7b-48gs6 0/1 CrashLoopBackOff 5662 (41s ago) 20d
First, i’ll ditch the Siyuan test install as I’m not using that presently.
The actual tpk.pw instance is actually in docker
$ kubectl get endpoints | grep 6806
siyuan-service 192.168.1.100:6806 245d
So I can delete the ‘test’ deployment
$ kubectl get deployment | grep -i siyuan
siyuan-deployment-test 0/1 1 0 245d
$ kubectl delete deployment siyuan-deployment-test
deployment.apps "siyuan-deployment-test" deleted
The Valkey instance was fired up in March when it first came up. I had to hack the Redis charts to make it work back then.
Now we have an actual supported helm chart and a matching container in docker hub.
$ helm install valkey-rel -n valkeytest --create-namespace oci://registry-1.docker.io/bitnamicharts/valkey
Pulled: registry-1.docker.io/bitnamicharts/valkey:2.0.2
Digest: sha256:3d1812c0879ef1a359b3f32151550bcdaa65ce55a5643af9692f78037a0d18fe
NAME: valkey-rel
LAST DEPLOYED: Thu Nov 28 14:29:58 2024
NAMESPACE: valkeytest
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: valkey
CHART VERSION: 2.0.2
APP VERSION: 8.0.1
** Please be patient while the chart is being deployed **
Valkey can be accessed on the following DNS names from within your cluster:
valkey-rel-primary.valkeytest.svc.cluster.local for read/write operations (port 6379)
valkey-rel-replicas.valkeytest.svc.cluster.local for read-only operations (port 6379)
To get your password run:
export VALKEY_PASSWORD=$(kubectl get secret --namespace valkeytest valkey-rel -o jsonpath="{.data.valkey-password}" | base64 -d)
To connect to your Valkey server:
1. Run a Valkey pod that you can use as a client:
kubectl run --namespace valkeytest valkey-client --restart='Never' --env VALKEY_PASSWORD=$VALKEY_PASSWORD --image docker.io/bitnami/valkey:8.0.1-debian-12-r1 --command -- sleep infinity
Use the following command to attach to the pod:
kubectl exec --tty -i valkey-client \
--namespace valkeytest -- bash
2. Connect using the Valkey CLI:
REDISCLI_AUTH="$VALKEY_PASSWORD" valkey-cli -h valkey-rel-primary
REDISCLI_AUTH="$VALKEY_PASSWORD" valkey-cli -h valkey-rel-replicas
To connect to your database from outside the cluster execute the following commands:
kubectl port-forward --namespace valkeytest svc/valkey-rel-primary 6379:6379 &
REDISCLI_AUTH="$VALKEY_PASSWORD" valkey-cli -h 127.0.0.1 -p 6379
WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
- replica.resources
- primary.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
I can see it running just fine
$ kubectl get pods -n valkeytest
NAME READY STATUS RESTARTS AGE
my-redis-master-0 1/1 Running 0 236d
my-redis-replicas-0 0/1 CrashLoopBackOff 56951 (3m35s ago) 202d
valkey-rel-primary-0 1/1 Running 0 2m11s
valkey-rel-replicas-0 1/1 Running 0 2m11s
valkey-rel-replicas-1 1/1 Running 0 87s
valkey-rel-replicas-2 1/1 Running 0 55s
I’ll then remove the old deployment
$ helm list -n valkeytest
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
my-redis valkeytest 2 2024-04-06 07:56:05.4996752 -0500 CDT deployed redis-19.0.2 7.2.4
valkey-rel valkeytest 1 2024-11-28 14:29:58.31462213 -0600 CST deployed valkey-2.0.2 8.0.1
$ helm delete my-redis -n valkeytest
release "my-redis" uninstalled
$ kubectl get pods -n valkeytest
NAME READY STATUS RESTARTS AGE
valkey-rel-primary-0 1/1 Running 0 3m30s
valkey-rel-replicas-0 1/1 Running 0 3m30s
valkey-rel-replicas-1 1/1 Running 0 2m46s
valkey-rel-replicas-2 1/1 Running 0 2m14s
Netbox was from a post back in June and it worked then
but there is a broken PSQL container
$ kubectl get pods -n netbox
NAME READY STATUS RESTARTS AGE
redis-deployment-55b654cbc9-rkqmg 1/1 Running 0 176d
netbox-deployment-84956dc6d-gjgqh 1/1 Running 0 20d
postgres-deployment-7df44c4688-2nzqx 0/1 CrashLoopBackOff 5663 (26s ago) 20d
I tried bouncing it
$ kubectl delete pod postgres-deployment-7df44c4688-2nzqx -n netbox
pod "postgres-deployment-7df44c4688-2nzqx" deleted
But it still fails
$ kubectl get pods -n netbox
NAME READY STATUS RESTARTS AGE
redis-deployment-55b654cbc9-rkqmg 1/1 Running 0 176d
netbox-deployment-84956dc6d-gjgqh 1/1 Running 0 20d
postgres-deployment-7df44c4688-6mvtw 0/1 Error 3 (26s ago) 41s
$ kubectl logs postgres-deployment-7df44c4688-6mvtw -n netbox
PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-11-28 20:35:30.880 UTC [1] FATAL: database files are incompatible with server
2024-11-28 20:35:30.880 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.2 (Debian 17.2-1.pgdg120+1).
I’m going to try and fix this.
First, I’ll launch a pod
$ cat ./fixPSQL.yaml
apiVersion: v1
kind: Pod
metadata:
name: postgres
spec:
containers:
- name: postgres
image: postgres:17
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgres-storage
volumes:
- name: postgres-storage
persistentVolumeClaim:
claimName: postgres-pvc
$ kubectl apply -f ./fixPSQL.yaml -n netbox
pod/postgres created
Then I can exec into it
$ kubectl get pods -n netbox
NAME READY STATUS RESTARTS AGE
redis-deployment-55b654cbc9-rkqmg 1/1 Running 0 176d
netbox-deployment-84956dc6d-gjgqh 1/1 Running 0 20d
postgres-deployment-7df44c4688-6mvtw 0/1 CrashLoopBackOff 6 (114s ago) 7m24s
postgres 1/1 Running 0 48s
$ kubectl exec -it postgres -n netbox -- /bin/bash
root@postgres:/#
That didn’t work.
Next, I looked at the deployment
$ kubectl get deployment -n netbox
NAME READY UP-TO-DATE AVAILABLE AGE
redis-deployment 1/1 1 1 176d
netbox-deployment 1/1 1 1 176d
postgres-deployment 0/1 1 0 176d
Then editted it to change from ‘postgres:latest’ to just ‘postgres:16’
$ kubectl edit deployment -n netbox postgres-deployment
deployment.apps/postgres-deployment edited
$ kubectl get pods -n netbox
NAME READY STATUS RESTARTS AGE
redis-deployment-55b654cbc9-rkqmg 1/1 Running 0 176d
netbox-deployment-84956dc6d-gjgqh 1/1 Running 0 20d
postgres-deployment-6dc7cc547b-fc6q8 1/1 Running 0 67s
And I can see it came back to life
The mystery of the Phantom Page
For the second time this week, I was awoken at 3:30am by a webhook error
The first time, I had noticed a blog post errored so I just attributed it to that.
The error seems related to the hosted Cachet instance. Essentially something, like Datadog, is trying to post a status update to cachet (status.freshbrewed.science) but not getting a good response code
The links bring me back to Datadog where I can see some issues
I can see the REST (GET) works just fine for []/api/v1/incidents](https://status.freshbrewed.science/api/v1/incidents)
{"meta":{"pagination":{"total":72,"count":20,"per_page":20,"current_page":1,"total_pages":4,"links":{"next_page":"https:\/\/status.freshbrewed.science\/api\/v1\/incidents?page=2","previous_page":null}}},"data":[{"id":3,"component_id":0,"name":"[Triggered] [TEST]","status":1,"message":"Datadog princessking reports Incident 7685240989616342215 of type event_v2_alert. Last updated 1722338925000","created_at":"2024-07-30 06:28:45","updated_at":"2024-07-30 06:28:45","deleted_at":null,"visible":1,"stickied":false,"occurred_at":"2024-07-30 06:28:45","user_id":1,"notifications":false,"is_resolved":false,"meta":[],"updates":[],"human_status":"Investigating","latest_update_id":null,"latest_status":1,"latest_human_status":"Investigating","latest_icon":"icon ion-flag oranges","permalink":"https:\/\/status.freshbrewed.science\/incidents\/3","duration":0},{"id":4,"component_id":0,"name":"Datadog: [TEST] (error)","status":1,"message":"Datadog princessking reports (events(\"Build service:undefined\").rollup(\"count\").last(\"5m\") > 1). Priority . Last updated 1722339224000. Tags; monitor","created_at":"2024-07-30 06:33:45","updated_at":"2024-07-30 06:33:45","deleted_at":null,"visible":1,"stickied":false,"occurred_at":"2024-07-30 06:33:45","user_id":1,"notifications":false,"is_resolved":false,"meta":[],"updates":[],"human_status":"Investigating","latest_update_id":null,"latest_status":1,"latest_human_status":"Investigating","latest_icon":"icon ion-flag oranges","permalink":"https:\/\/status.freshbrewed.science\/incidents\/4","duration":0},{"id":5,"component_id":0,"name":"Datadog: Passed building jekyll 7686491575535129498 (error)","status":1,"message":"Datadog princessking reports (events(\"Build service:undefined\").rollup(\"count\").last(\"5m\") > 1). Priority . Last updated 1722413579000. Tags; monitor","created_at":"2024-07-31 03:13:00","updated_at":"2024-07-31 03:13:00","deleted_at":null,"visible":1,"stickied":false,"occurred_at":"2024-07-31 03:13:00","user_id":1,"notifications":false,"is_resolved":false,"meta":[],"updates":[],"human_status":"Investigating","latest_update_id":null,"latest_status":1,"latest_human_status":"Investigating","latest_icon":"icon ion-flag oranges","permalink":"https:\/\/status.freshbrewed.science\/incidents\/5","duration":0},{"id":6,"component_id":0,"name":"Datadog: Passed building jekyll 7686491575535129498 (success)","status":1,"message":"Datadog princessking reports (events(\"Build service:undefined\").rollup(\"count\").last(\"5m\") > 1). Priority . Last updated 1722413639000. Tags; monitor","created_at":"2024-07-31 03:14:00","updated_at":"2024-07-31 03:14:00","deleted_at":null,"visible":1,"stickied":false,"occurred_at":"2024-07-31 03:14:00","user_id":1,"notifications":false,"is_resolved":false,"meta":[],"updates":[],"human_status":"Investigating","latest_update_id":null,"latest_status":1,"latest_human_status":"Investigating","latest_icon":"icon ion-flag oranges","permalink":"https:\/\/status.freshbrewed.science\/incidents\/6","duration":0},{"id":7,"component_id":0,"name":"Datadog: [TEST] Cluster 33 not reporting (error)","status":1,"message":"Datadog princessking reports (\"datadog.agent.up\".over(\"cluster_name:int33\").by(\"*\").last(2).count_by_status()). Priority . Last updated 1722426318000. Tags; cluster_name:int33,monitor","created_at":"2024-07-31 06:45:19","updated_at":"2024-07-31 06:45:19","deleted_at":null,"visible":1,"stickied":false,"occurred_at":"2024-07-31 06:45:19","user_id":1,"notifications":false,"is_resolved":false,"meta":[],"updates":[],"human_status":"Investigating","latest_update_id":null,"latest_status":1,"latest_human_status":"Investigating","latest_icon":"icon ion-flag oranges","permalink":"https:\/\/status.freshbrewed.science\/incidents\/7","duration":0},{"id":8,"component_id":0,"name":"Datadog: [TEST] Cluster 33 not reporting (error)","status":1,"message":"Datadog princessking reports (\"datadog.agent.up\".over(\"cluster_name:int33\").by(\"*\").last(2).count_by_status()). Priority . Last updated 1722427105000. Tags; cluster_name:int33,monitor","created_at":"2024-07-31 06:58:26","updated_at":"2024-07-31 06:58:26","deleted_at":null,"visible":1,"stickied":false,"occurred_at":"2024-07-31 06:58:26","user_id":1,"notifications":false,"is_resolved":false,"meta":[],"updates":[],"human_status":"Investigating","latest_update_id":null,"latest_status":1,"latest_human_status":"Investigating","latest_icon":"icon ion-flag oranges","permalink":"https:\/\/status.freshbrewed.science\/incidents\/8","duration":0},{"id":9,"component_id":0,"name":"Datadog: [TEST] Cluster 33 not reporting (success)","status":1,"message":"Datadog princessking reports (\"datadog.agent.up\".over(\"cluster_name:int33\").by(\"*\").last(2).count_by_status()). Priority . Last updated 1722427154000. Tags; cluster_name:int33,monitor","created_at":"2024-07-31 06:59:15","updated_at":"2024-07-31 0
...
I checked the obvious, like bad JSON by putting the payload through jq
$ cat testDD.json | jq
{
"name": "Datadog: Passed building jekyll 7861944572023095403 (error)",
"description": "\n@isaac.johnson@gmail.com\n@webhook-CachetSelfHosted \n\nBranch 2024-servermaint passed build and idjohnson\n\n[2 events triggered this monitor](https://app.datadoghq.com/event/explorer?query=Build+service%3Aundefined&from_ts=1732871008000&to_ts=1732871308000&live=false), here is the last one.\n\n- - -\n\n#### [Passed building jekyll](https://app.datadoghq.com/event/event?id=7861944572023095403&start=1732871008000&end=1732871308000)\n\n\nBranch 2024-servermaint passed build\n\n\n- - -\n\nThe **count** of events matching **[Build service:undefined](https://app.datadoghq.com/event/explorer?query=Build+service%3Aundefined&from_ts=1732871008000&to_ts=1732871308000&live=false)** was **> 1.0** during the **last 5m**.\n\nThe monitor was last triggered at Fri Nov 29 2024 09:08:28 UTC.\n\n- - -\n\n[[Monitor Status](https://app.datadoghq.com/monitors/68055508?from_ts=1732870408000&to_ts=1732871608000&event_id=7861946586324009200&link_source=monitor_notif)] · [[Edit Monitor](https://app.datadoghq.com/monitors/68055508/edit?link_source=monitor_notif)] · [[Related Logs](https://app.datadoghq.com/logs?from_ts=1732871008000&to_ts=1732871308000&live=false&link_source=monitor_notif)]\n",
"message": "Datadog princessking reports (events(\"Build service:undefined\").rollup(\"count\").last(\"5m\") > 1). Priority . Last updated 1732871399000. Tags; monitor",
"status": 1
}
I can go to the Integrations section of Datadog and bring up Webhooks to check the configuration
I can even replicate the same POST locally and it works fine
$ curl -X POST "https://status.freshbrewed.science/api/v1/incidents" -H "Content-Type: application/json" -H "X-Cachet-Token: qNeV3ltHh1eptRI3KlDA" --data-binary @testDD.json -w "\nRESPONSE CODE: %{http_code}\n"
{"data":{"stickied":false,"notifications":false,"user_id":1,"name":"Datadog: Passed building jekyll 7861944572023095403 (error)","status":1,"visible":1,"message":"Datadog princessking reports (events(\"Build service:undefined\").rollup(\"count\").last(\"5m\") > 1). Priority . Last updated 1732871399000. Tags; monitor","occurred_at":"2024-11-29 09:10:51","updated_at":"2024-11-29 09:10:51","created_at":"2024-11-29 09:10:51","id":75,"is_resolved":false,"component":null,"updates":[],"human_status":"Investigating","latest_update_id":null,"latest_status":1,"latest_human_status":"Investigating","latest_icon":"icon ion-flag oranges","permalink":"https:\/\/status.freshbrewed.science\/incidents\/75","duration":0,"meta":[]}}
RESPONSE CODE: 200
So perhaps it is NOT the webhook at fault. Perhaps what is going on is that my build failed, but the message is just a bit confusing
I can see a build error in this history
I was finally able to replicate the failure by test building the Runner Image locally
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ docker build -t atest2 ghRunnerImage
[+] Building 86.6s (11/20) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.69kB 0.0s
=> [internal] load metadata for docker.io/summerwind/actions-runner:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [ 1/17] FROM docker.io/summerwind/actions-runner:latest 0.0s
=> [ 2/17] RUN sudo apt update -y && umask 0002 && sudo apt install -y ca-certificates curl apt-transport-https lsb-release gnupg 12.2s
=> [ 3/17] RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null 0.6s
=> [ 4/17] RUN umask 0002 && echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ focal main" | sudo tee /etc/apt/sources.list.d/azure-cli.list 0.6s
=> [ 5/17] RUN sudo apt update -y && umask 0002 && sudo apt install -y azure-cli awscli ruby-full 44.2s
=> [ 6/17] RUN curl -fsSL https://get.pulumi.com | sh 6.2s
=> [ 7/17] RUN /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" 20.4s
=> ERROR [ 8/17] RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" && brew install go@1.23.1 2.2s
------
> [ 8/17] RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" && brew install go@1.23.1:
2.000 Warning: No available formula with the name "go@1.23.1". Did you mean go@1.21?
2.060 ==> Searching for similarly named formulae...
2.136 ==> Formulae
2.136 go@1.21
2.136
2.136 To install go@1.21, run:
2.136 brew install go@1.21
------
Dockerfile:27
--------------------
26 | # Install Golang 1.23.1
27 | >>> RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" \
28 | >>> && brew install go@1.23.1
29 | #echo 'export PATH="/home/linuxbrew/.linuxbrew/opt/go@1.19/bin:$PATH"'
--------------------
ERROR: failed to solve: process "/bin/sh -c eval \"$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)\" && brew install go@1.23.1" did not complete successfully: exit code: 1
The problem is that I had pinned to a specific bugfix release
# Install Golang 1.23.1
RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" \
&& brew install go@1.23.1
#echo 'export PATH="/home/linuxbrew/.linuxbrew/opt/go@1.19/bin:$PATH"'
When instead, I really should have just used the major.minor as the current version is now “1.23.3” and “1.23.1” is no longer available for homebrew
# Install Golang 1.23
RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" \
&& brew install go@1.23
#echo 'export PATH="/home/linuxbrew/.linuxbrew/opt/go@1.19/bin:$PATH"'
I fixed that only to get another error
=> [15/17] RUN umask 0002 && gem install bundler -v 2.4.22 16.0s
=> ERROR [16/17] RUN umask 0002 && gem install jekyll --version="~> 4.2.0" 35.5s
------
> [16/17] RUN umask 0002 && gem install jekyll --version="~> 4.2.0":
35.35 ERROR: Error installing jekyll:
35.35 The last version of ffi (~> 1.0) to support your Ruby & RubyGems was 1.17.0. Try installing it with `gem install ffi -v 1.17.0` and then running the current command again
35.35 ffi requires RubyGems version >= 3.3.22. The current RubyGems version is 3.1.2. Try 'gem update --system' to update RubyGems itself.
35.37 Successfully installed unicode-display_width-1.8.0
35.37 Successfully installed terminal-table-2.0.0
35.37 Successfully installed safe_yaml-1.0.5
35.37 Successfully installed rouge-3.30.0
35.37 Successfully installed forwardable-extended-2.6.0
35.37 Successfully installed pathutil-0.16.2
35.37 Successfully installed mercenary-0.4.0
35.37 Successfully installed liquid-4.0.4
35.37 Successfully installed rexml-3.3.9
35.37 Successfully installed kramdown-2.5.1
35.37 Successfully installed kramdown-parser-gfm-1.1.0
------
Dockerfile:53
--------------------
52 | # Limitations in newer jekyll
53 | >>> RUN umask 0002 \
54 | >>> && gem install jekyll --version="~> 4.2.0"
55 |
--------------------
ERROR: failed to solve: process "/bin/sh -c umask 0002 && gem install jekyll --version=\"~> 4.2.0\"" did not complete successfully: exit code: 1
It actually took quite a few iterations before I sorted out the Dockerfile. With the age of my Ruby base, I needed to pivot to embedded rbenv. Additionally, while the AI suggested I really could source .bashrc, it never worked. So I pulled in the rbenv command directly:
The working file:
FROM summerwind/actions-runner:latest
RUN sudo apt update -y \
&& umask 0002 \
&& sudo apt install -y ca-certificates curl apt-transport-https lsb-release gnupg \
&& sudo apt install -y build-essential libssl-dev libreadline-dev zlib1g-dev
# Install MS Key
RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null
# Add MS Apt repo
RUN umask 0002 && echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ focal main" | sudo tee /etc/apt/sources.list.d/azure-cli.list
# Install Azure CLI
RUN sudo apt update -y \
&& umask 0002 \
&& sudo apt install -y azure-cli awscli git curl libssl-dev libreadline-dev zlib1g-dev autoconf bison build-essential libyaml-dev libreadline-dev libncurses5-dev libffi-dev libgdbm-dev
RUN curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash
RUN cat ~/.bashrc
# Install Ruby 3.0.0
RUN /bin/bash -c "eval ""$(~/.rbenv/bin/rbenv init - --no-rehash bash)"" && rbenv install 3.0.0 && rbenv global 3.0.0"
# Verify Ruby installation
RUN /bin/bash -c "eval ""$(~/.rbenv/bin/rbenv init - --no-rehash bash)"" && ruby -v"
# Install Pulumi
RUN curl -fsSL https://get.pulumi.com | sh
# Install Homebrew
RUN /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# OpenTF
# Install Golang 1.23.1
RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" \
&& brew install go@1.23
RUN eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)" \
&& brew install opentofu
RUN sudo cp /home/linuxbrew/.linuxbrew/bin/tofu /usr/local/bin/
RUN sudo chown runner /usr/local/bin
#RUN sudo chmod 777 /var/lib/gems/3.0.0
#RUN sudo chown runner /var/lib/gems/3.0.0
# Install Expect and SSHPass
RUN sudo apt update -y \
&& umask 0002 \
&& sudo apt install -y sshpass expect
# Update RubyGems and install ffi, bundler, and jekyll
RUN /bin/bash -c "eval ""$(~/.rbenv/bin/rbenv init - --no-rehash bash)"" && gem update --system && gem install ffi -v 1.17.0 && gem install bundler -v 2.4.22 && gem install jekyll"
RUN sudo rm -rf /var/lib/apt/lists/*
#harbor.freshbrewed.science/freshbrewedprivate/myghrunner:2.1.0
This did solve the build portion of the workflow, but I then discovered my next issue - the internal container registries cert has expired
On the target NAS, I can see that in the Control Panel.
Before I can actually renew this, I’ll need to set port 80 (and likely 443) temporarily to directly point to the NAS
This means for a few minutes, I’ll be redirecting all traffic from my production K8s cluster (as I only have the one IP to use)
I renew and it takes a couple minutes to get the cert
Now that it’s valid again
I set the forwarding rule back to my prod cluster and try the Github workflow again. This time it worked
I can also see it in the Harbor CR
Container Image Backups
Let’s say, however, for safety (since this new version is untested), I want to pull my real live image and save it to a file system - just in case I had a catastrophic event and my harbor went down.
I can pull the ghrunner image locally
$ docker pull harbor.freshbrewed.science/freshbrewedprivate/myghrunner:1.1.16
1.1.16: Pulling from freshbrewedprivate/myghrunner
3c67549075b6: Pull complete
30bbff2a2af9: Pull complete
0380a86574cd: Pull complete
0ae90d602824: Pull complete
722c8e8af01c: Pull complete
d1d7f0ca173f: Pull complete
9fe67899eec3: Pull complete
7529aa908041: Pull complete
b8b295e96a7f: Pull complete
d968d3b07a0f: Pull complete
b87ea8669cd2: Pull complete
c6502b11f2d5: Pull complete
d3890d837bb8: Pull complete
8b1c105f8531: Pull complete
f9986d393a74: Pull complete
d0787bd6ce04: Pull complete
0d9cca41309e: Pull complete
503eb7af1c3a: Pull complete
6bd4ede070f6: Pull complete
c9ea28e3f1aa: Pull complete 157fc3344778: Pull complete
263f010653ae: Pull complete
4b5943f7cce4: Pull complete 4ccac9bd9bf6: Pull complete 35db56b4afa5: Pull complete c650733669ec: Pull complete
9dedae9ae707: Pull complete
2717fa2044cb: Pull complete
cf6aa2c094b5: Pull complete
dade20e5fae7: Pull complete
Digest: sha256:8ece30dba0bb0de72e7022931f754d758744cf594f775d0cdf11f5de4dd4d085
Status: Downloaded newer image for harbor.freshbrewed.science/freshbrewedprivate/myghrunner:1.1.16
harbor.freshbrewed.science/freshbrewedprivate/myghrunner:1.1.16
I can then save it to a local tar file
$ docker save -o myghrunner-1.1.16.tar harbor.freshbrewed.science/freshbrewedprivate/myghrunner:1.1.16
$ ls -ltrah | tail -n2
-rw------- 1 builder builder 5.0G Nov 30 09:29 myghrunner-1.1.16.tar
drwxr-xr-x 21 builder builder 4.0K Nov 30 09:29 .
I can then compress to a gzip
$ tar -czvf myghrunner-1.1.16.tar.gz ./myghrunner-1.1.16.tar
./myghrunner-1.1.16.tar
$ ls -ltrah | tail -n2
drwxr-xr-x 21 builder builder 4.0K Nov 30 09:55 .
-rw-r--r-- 1 builder builder 1.9G Nov 30 09:58 myghrunner-1.1.16.tar.gz
Lastly, I’ll copy it out to a NAS for safe-keeping
$ sudo cp ./myghrunner-1.1.16.tar.gz /mnt/linuxbackups/myghrunner-1.1.16.tar.gz
Which I can confirm is now on the NAS
Later, if I needed, I could extract it and use load to restore
$ docker load --input ./myghrunner-1.1.16.tar
Again
The madness continued last night with another 3am page.
I decided to think a bit more critically about this.
I looked and the reason for this page is it seems the Event Datadog trigger is paging Pagerduty because it cannot post a non-error status to Cachet. This doesn’t even make sense to me.
It’s clearly working because we see build status
But the error is that the posting to that endpoint (which worked) doesn’t respond the way it wants.
For now, since I don’t want non-errors on Cachet (status.freshbrewed.science) anyhow, I’ll remove this webhook post from Datadog
Now the recipient for “misc” events is just my email
That is, the problem was an error posting to the webhook on my first general purpose “event” monitor was then triggering the PD one of the same name
I keep thinking that if I wanted Datadog as my “global event dashboard” instead of Cachet, that would be as easy as just sharing the dashboard as you can see here
Pagerduty to Cachet
I had another thought - Pagerduty has Integrations. What if I tied a PD service to Cachet?
I tried a V3 webhook, but it does not let me configure the payload so it didn’t do anything
However, I can make Beszel get there. I had to work out the syntax of the Shoutrrr push notification scheme
generic://status.freshbrewed.science/api/v1/incidents?name=beszel&description=beszel%20has%20issues&message=beszel%20has%20issues&status=1&@x-cachet-token=asdfsadfsdfasdf
Which you see can at least post a message that Beszel has issues with a Dockerhost
This touches on the other side - non containerized monitoring. I have (and could continue) to just use Uptime Kuma to Matrix for chat room updates
At first, I tried to get Cachet tied to Uptime - this payload should work but gets rejected each time
However, I can use the native Matrix support in Uptime Kuma to post to my notifications channel
which you can see in a test
We can see I have that already working
It might be complicated, but in the end, this is our general notification flow
(if interested, this is the Diagrams code for that)
$ cat pyd_20241205_matrixnotifications.py
from diagrams import Diagram
from diagrams.custom import Custom
from diagrams.onprem.compute import Server
from diagrams.onprem.monitoring import Datadog
from diagrams.saas.chat import Discord
from diagrams.saas.alerting import Pagerduty
with Diagram("Notification Flow", show=True):
datadog = Datadog("Datadog")
servers = [Server(f"Server{i}") for i in range(1, 4)]
uptime = Custom("UptimeKuma","./resources/uptime-kuma.png")
beszel = Custom("Beszel","./resources/beszel.png")
matrix = Custom("Matrix","./resources/matrix-logo.png")
pagerduty = Pagerduty("Pagerduty")
discord = Discord("Discord")
cachet = Custom("Cachet","./resources/cachet.png")
email = Custom("Gmail","./resources/email.png")
for server in servers:
uptime >> server >> datadog
uptime >> matrix
uptime >> pagerduty
uptime >> discord
datadog >> pagerduty
datadog >> email
pagerduty >> email
beszel >> cachet
beszel >> email
Summary
We covered MinIO and DSM updates, inflated laptop batteries, Node Updates and disk space remediation. We also showed how to do container cleanups in Docker.
We did some cleanup on a NAS and almost did an update that would have removed transcoding. We found and fixed some ImagePull issues related to an expired GH token.
I fixed a random build error from building an updated summerwind runner image then fixed an expired Docker registry certificate. Lastly, I showed how to backup a docker image to a filesystem and rework the notification flows for alerting.
I realize this was a bit of a Smörgåsbord of updates, but hopefully you found something of value you can take with you. I know I am mostly thrilled not to get 3am pages (knock on wood).
