Published: Dec 19, 2025 by Isaac Johnson
At first, there was consternation and shaking of fists at the news in October that MinIO would stop distributing binaries. This was only made worse when just this month, they announced that they would end the Open-Source project, moving it to maintenance mode.
Let’s not forget they had already yanked out the popular WebUI in June for the Community edition. It’s the behavior one expects from companies in the death throws trying any avenue to revenue.
The reactions from the developer community were not surprising, first, after ending the binaries and more recently on the maintenance mode news.
From those threads (and a TLDR newsletter), I found two viable alternatives; Garage and RustFS.
I’ll save Garage for another time. Today, I’ll look at RustFS as well as the Open-Source fork of Hashi Vault, OpenBao which has gotten surprisingly mature since I last spoke on it.
I’ll be setting up RustFS and Bao in both Docker and Kubernetes. The connection between those two, of course, is using OpenBao as a KMS backend to provide proper SSE in RustFS (takes a bit, but i do get there in the article below).
Let’s start with RustFS…
RustFS
It was while reading about the latest MinIO news I came across RustFS
RustFS combines the simplicity of MinIO with the memory safety and raw performance of Rust. It offers full S3 compatibility, is completely open-source, and is optimized for data lakes, AI, and big data workloads.
Let’s start with a local install. I’ll create a directory and add folders for data and logs
builder@LuiGi:~$ cd rustfs/
builder@LuiGi:~/rustfs$ mkdir -p logs data
builder@LuiGi:~/rustfs$ sudo chown -R 10001:10001 data logs
[sudo: authenticate] Password:
Then fire up docker in daemon mode:
builder@LuiGi:~/rustfs$ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
Unable to find image 'rustfs/rustfs:latest' locally
latest: Pulling from rustfs/rustfs
2d35ebdb57d9: Already exists
8a5154c4d5d8: Pull complete
a3ba232acba2: Pull complete
c015da94d7be: Pull complete
daf41b756f62: Pull complete
4f4fb700ef54: Pull complete
c3c2e752df29: Pull complete
Digest: sha256:8e467b32af3ff83e70c70dddb0c36b5e611f46e89a3075db8770aea4f30b2fe3
Status: Downloaded newer image for rustfs/rustfs:latest
2a9f2bbdd2ce18752c242d97057875566e1183fe4647b34e5e6ab69dac0f5c24
Now let’s access the admin console on http://localhost:9001/rustfs/console/auth/login
The default login/pass is rustfsadmin/rustfsadmin.
Let’s now create an access key
If you just use aws configure it will ask for the key id and secret, but won’t set the endpoint
builder@LuiGi:~/rustfs$ aws configure
AWS Access Key ID [None]: 8T9kDc46oCuyeKS2nzYj
AWS Secret Access Key [None]: 8T9kDc46oCuyeKS2nzYj8T9kDc46oCuyeKS2nzYj
Default region name [None]:
Default output format [None]: json
It’s not reall a big deal as we can just use AWS_ENDPOINT_URL as an env var when querying for buckets
builder@LuiGi:~/rustfs$ export AWS_ENDPOINT_URL=http://localhost:9000
builder@LuiGi:~/rustfs$ aws s3 ls
builder@LuiGi:~/rustfs$
I can now create a bucket with ease
builder@LuiGi:~/rustfs$ aws s3 mb s3://mytestbucket
make_bucket: mytestbucket
builder@LuiGi:~/rustfs$ aws s3 ls
2025-12-17 18:41:40 mytestbucket
I can copy and list files just as easily
builder@LuiGi:~/rustfs$ aws s3 cp ~/ss2b.sh s3://mytestbucket/
upload: ../ss2b.sh to s3://mytestbucket/ss2b.sh
builder@LuiGi:~/rustfs$ aws s3 ls s3://mytestbucket/
2025-12-17 18:42:31 624 ss2b.sh
We can see that RustFS copied it locally to the data folder
builder@LuiGi:~/rustfs$ ls data/mytestbucket/
ss2b.sh
builder@LuiGi:~/rustfs$ ls -l data/mytestbucket/
total 4
drwxr-xr-x 2 10001 10001 4096 Dec 17 18:42 ss2b.sh
builder@LuiGi:~/rustfs$ ls -l ./
data/ logs/
builder@LuiGi:~/rustfs$ ls -l ./logs/
total 0
I can see the bucket reflected in the Web UI
And if we go in the bucket, we can see the files as well
So far, none of this is encrypted.
What if we wanted to encrypt with OpenBao (Open-Source fork of Hashi Vault)?
That can be added in the KMS section
Now, if I were to encrypt the bucket without setting up a Vault/OpenBao endpoint:
It would just fail when I tried to use it:
builder@LuiGi:~/rustfs$ aws s3 cp ~/ss2b.sh s3://mytestbucket/ss2b.sh.2
upload failed: ../ss2b.sh to s3://mytestbucket/ss2b.sh.2 An error occurred (InternalError) when calling the PutObject operation (reached max retries: 2): Io error: KMS encryption service is not initialized
Helm install
Let’s switch to using the RustFS chart.
I needed to add back Traefik to my test k3s cluster
$ kubectl get ingressclass
NAME CONTROLLER PARAMETERS AGE
traefik traefik.io/ingress-controller <none> 6m4s
What’s quite odd about the Helm documentation is it completely neglects to say where their chart repo lives
So of course it will fail to install
$ helm install rustfs rustfs/rustfs -n rustfs --create-namespace --set ingress.className="traefik"
Error: INSTALLATION FAILED: repo rustfs not found
I looked at some of my cached repos just to guess the name and finally figured it out
$ helm repo add rustfs https://rustfs.github.io/helm/
"rustfs" has been added to your repositories
It bothered me enough to propose a documentation change.
Now I can launch RustFS
$ helm install rustfs rustfs/rustfs -n rustfs --create-namespace --set ingress.className="traefik"
NAME: rustfs
LAST DEPLOYED: Fri Dec 19 07:10:00 2025
NAMESPACE: rustfs
STATUS: deployed
REVISION: 1
NOTES:
1. Watch all pods come up
kubectl get pods -w -l app.kubernetes.io/name=rustfs -n rustfs
2. Visit the dashboard
https://your.rustfs.com/
By default, it wants to make an Ingress with the default URL of your.rustfs.com
$ kubectl get svc -n rustfs
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rustfs-headless ClusterIP None <none> 9000/TCP,9001/TCP 50s
rustfs-svc ClusterIP 10.43.154.49 <none> 9000/TCP,9001/TCP 50s
$ kubectl get ingress -n rustfs
NAME CLASS HOSTS ADDRESS PORTS AGE
rustfs traefik your.rustfs.com 80 60s
For me, the pods started to crash
$ kubectl get po -n rustfs
NAME READY STATUS RESTARTS AGE
rustfs-0 0/1 CrashLoopBackOff 2 (5s ago) 99s
rustfs-1 0/1 PodInitializing 0 99s
rustfs-2 0/1 CrashLoopBackOff 2 (14s ago) 99s
rustfs-3 0/1 Pending 0 99s
with a name resolution error:
$ kubectl logs rustfs-0 -n rustfs
Defaulted container "rustfs" out of: rustfs, init-step (init)
Starting Tokio runtime with configured parameters:
worker_threads: 4, max_blocking_threads: 1024, thread_stack_size: 1048576, thread_keep_alive: 60, global_queue_interval: 31, event_interval: 61, max_io_events_per_tick: 1024, thread_name: rustfs-worker
[2025-12-19 13:11:38.382063 +00:00] ERROR [rustfs] [rustfs/src/main.rs:132] [main:ThreadId(1)] Server encountered an error and is shutting down: failed to lookup address information: Name does not resolve
Error: Custom { kind: Other, error: Custom { kind: Other, error: Custom { kind: Uncategorized, error: "failed to lookup address information: Name does not resolve" } } }
Looking at all the chart parameters I see I can just disable ingress with ingress.enabled.
$ helm upgrade rustfs rustfs/rustfs -n rustfs --create-namespace --set i
ngress.enabled=false
Release "rustfs" has been upgraded. Happy Helming!
NAME: rustfs
LAST DEPLOYED: Fri Dec 19 07:14:09 2025
NAMESPACE: rustfs
STATUS: deployed
REVISION: 2
NOTES:
1. Watch all pods come up
kubectl get pods -w -l app.kubernetes.io/name=rustfs -n rustfs
I still saw them in a crashloop so i forced a cycling
$ kubectl get po -n rustfs
NAME READY STATUS RESTARTS AGE
rustfs-0 0/1 CrashLoopBackOff 5 (6s ago) 4m32s
rustfs-1 0/1 CrashLoopBackOff 4 (56s ago) 4m32s
rustfs-2 0/1 CrashLoopBackOff 5 (27s ago) 4m32s
rustfs-3 0/1 Pending 0 4m32s
$ kubectl delete po rustfs-0 -n rustfs
pod "rustfs-0" deleted
$ kubectl get po -n rustfs
NAME READY STATUS RESTARTS AGE
rustfs-0 0/1 Pending 0 85s
rustfs-1 0/1 CrashLoopBackOff 5 (69s ago) 6m12s
rustfs-2 0/1 CrashLoopBackOff 5 (2m7s ago) 6m12s
rustfs-3 0/1 CrashLoopBackOff 3 (3s ago) 6m12s
I removed the helm chart but did see it made quite a lot of PVCs
I’ll just wipe the namespace to clear them up
$ kubectl delete ns rustfs
namespace "rustfs" deleted
I’m going to try again with the simplest path - no ingress and standalone mode:
$ helm install rustfs rustfs/rustfs -n rustfs --create-namespace --set ingress.enabled=false --set mode.distributed.enabled=false --set mode.standalone.enabled=true
NAME: rustfs
LAST DEPLOYED: Fri Dec 19 07:19:02 2025
NAMESPACE: rustfs
STATUS: deployed
REVISION: 1
NOTES:
1. Watch all pods come up
kubectl get pods -w -l app.kubernetes.io/name=rustfs -n rustfs
This is looking a bit more promising
$ kubectl get po -n rustfs
NAME READY STATUS RESTARTS AGE
rustfs-745b55bdb8-8txll 0/1 Running 0 72s
$ kubectl get svc -n rustfs
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rustfs-svc NodePort 10.43.12.238 <none> 9000:32499/TCP,9001:30193/TCP 81s
Once the pod was up
$ kubectl get po -n rustfs
NAME READY STATUS RESTARTS AGE
rustfs-745b55bdb8-8txll 1/1 Running 0 103s
I can then port-forward to the UI service
$ kubectl port-forward svc/rustfs-svc -n rustfs 9001:9001
Forwarding from 127.0.0.1:9001 -> 9001
Forwarding from [::1]:9001 -> 9001
Handling connection for 9001
Handling connection for 9001
Handling connection for 9001
Handling connection for 9001
Handling connection for 9001
by default, like docker, it uses “rustfsadmin” for the login/pass. Later, we can change that with helm using secret.rustfs.access_key and secret.rustfs.secret_key settings.
OpenBao
Let’s take a moment and fire up OpenBao.
I can install with homebrew:
builder@LuiGi:~/rustfs$ brew install openbao
==> Auto-updating Homebrew...
Adjust how often this is run with `$HOMEBREW_AUTO_UPDATE_SECS` or disable with
`$HOMEBREW_NO_AUTO_UPDATE=1`. Hide these hints with `$HOMEBREW_NO_ENV_HINTS=1` (see `man brew`).
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
astra: Command-Line Interface for DataStax Astra
auto-editor: Efficient media analysis and rendering
bumpp: Interactive CLI that bumps your version numbers and more
carl: Calendar for the command-line
ctre: Compile-time PCRE-compatible regular expression matcher for C++
custom-install: Install CIA files directly to Nintendo 3DS SD card
diffnav: Git diff pager based on delta but with a file tree
docker-language-server: Language server for Dockerfiles, Compose files, and Bake files
faketty: Wrapper to exec a command in a pty, even if redirecting the output
flecs: Fast entity component system for C & C++
fresh-editor: Text editor for your terminal: easy, powerful and fast
git-get: Better way to clone, organize and manage multiple git repositories
git-pages: Scalable static site server for Git forges
git-pages-cli: Tool for publishing a site to a git-pages server
git-xet: Git LFS plugin that uploads and downloads using the Xet protocol
gitlogue: Cinematic Git commit replay tool
goat: General purpose AT Protocol CLI in Go
gtl: Greg's Template Library of useful classes
gup: Update binaries installed by go install
html-to-markdown: Transforms HTML (even entire websites) into clean, readable Markdown
install-nothing: Simulates installing things but doesn't actually install anything
joyce: Emulates the Amstrad PCW on Unix, Windows and macOS
labctl: CLI tool for interacting with iximiuz labs and playgrounds
maigret: Collect a dossier on a person by username from thousands of sites
mapscii: Whole World In Your Console
matcha: Daily digest generator for your RSS feeds
mcp-scan: Constrain, log and scan your MCP connections for security vulnerabilities
mistral-vibe: Minimal CLI coding agent
mq: Jq-like command-line tool for markdown processing
mufetch: Neofetch-style music cli
netshow: Interactive network connection monitor with friendly service names
nkt: TUI for fast and simple interacting with your BibLaTeX database
papeer: Convert websites into eBooks and Markdown
phantom: CLI tool for seamless parallel development with Git worktrees
py7zr: 7-zip in Python
qqqa: Fast, stateless LLM for your shell: qq answers; qa runs commands
rad: Modern CLI scripts made easy
redu: Ncdu for your restic repository
rmrfrs: Filesystem cleaning tool
rockcraft: Tool to create OCI images using the language from Snapcraft and Charmcraft
save3ds_fuse: Extract/Import/FUSE for 3DS save/extdata/database
slicot: Fortran subroutines library for systems and control
svt-vp9: Scalable Video Technology for VP9 Encoder
termshot: Creates screenshots based on terminal command output
tfclean: Remove applied moved block, import block, etc
thorvg: Lightweight portable library used for drawing vector-based scenes and animations
toktop: LLM usage monitor in terminal
tree-sitter@0.25: Incremental parsing library
treemd: TUI and CLI dual pane markdown viewer
ty: Extremely fast Python type checker, written in Rust
wasm-bindgen: Facilitating high-level interactions between Wasm modules and JavaScript
wuchale: Protobuf-like i18n from plain code
You have 6 outdated formulae installed.
==> Fetching downloads for: openbao
✔︎ Bottle Manifest openbao (2.4.4) [Downloaded 7.8KB/ 7.8KB]
✔︎ Bottle openbao (2.4.4) [Downloaded 70.2MB/ 70.2MB]
==> Pouring openbao--2.4.4.x86_64_linux.bottle.tar.gz
==> Caveats
To start openbao now and restart at login:
brew services start openbao
Or, if you don't want/need a background service you can just run:
/home/linuxbrew/.linuxbrew/opt/openbao/bin/bao server -dev
==> Summary
🍺 /home/linuxbrew/.linuxbrew/Cellar/openbao/2.4.4: 9 files, 184.1MB
==> Running `brew cleanup openbao`...
Disable this behaviour by setting `HOMEBREW_NO_INSTALL_CLEANUP=1`.
Hide these hints with `HOMEBREW_NO_ENV_HINTS=1` (see `man brew`).
==> `brew cleanup` has not been run in the last 30 days, running now...
Disable this behaviour by setting `HOMEBREW_NO_INSTALL_CLEANUP=1`.
Hide these hints with `HOMEBREW_NO_ENV_HINTS=1` (see `man brew`).
Removing: /home/builder/.cache/Homebrew/portable-ruby-3.4.7.x86_64_linux.bottle.tar.gz... (13.9MB)
Removing: /home/builder/.cache/Homebrew/bootsnap/8768da12db3cf6b6ad9542ea8361c1427bfdde225172809ac844ae4a66349274... (704 files, 6.7MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/xz... (96 files, 2.7MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/mpdecimal... (21 files, 803.9KB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/readline... (55 files, 3MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/openssl@3... (7,618 files, 44.8MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/zlib... (12 files, 484.2KB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/bzip2... (30 files, 604.3KB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/libedit... (60 files, 1.8MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/expat... (22 files, 889.4KB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/ncurses... (4,035 files, 11.4MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/python@3.13... (3,192 files, 74.3MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/mandoc... (28 files, 2.7MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/unzip... (16 files, 471.7KB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/berkeley-db@5... (5,270 files, 91.8MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/sqlite... (12 files, 6.7MB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/libffi... (18 files, 834.6KB)
Removing: /home/linuxbrew/.linuxbrew/var/homebrew/tmp/.cellar/ca-certificates... (3 files, 259.2KB)
I can then fire up a dev instance
uilder@LuiGi:~/rustfs$ bao server -dev -dev-root-token-id="myweakdevtoken"
==> OpenBao server configuration:
Administrative Namespace:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Environment Variables: AWS_ENDPOINT_URL, COLORTERM, DBUS_SESSION_BUS_ADDRESS, DEBUGINFOD_URLS, DESKTOP_SESSION, DISPLAY, GDMSESSION, GEMINI_API_KEY, GEM_HOME, GNOME_DESKTOP_SESSION_ID, GNOME_SETUP_DISPLAY, GNOME_SHELL_SESSION_MODE, GNOME_TERMINAL_SCREEN, GNOME_TERMINAL_SERVICE, GPG_AGENT_INFO, GTK_MODULES, HOME, HOMEBREW_CELLAR, HOMEBREW_PREFIX, HOMEBREW_REPOSITORY, IM_CONFIG_PHASE, INFOPATH, LANG, LESSCLOSE, LESSOPEN, LOGNAME, LS_COLORS, MEMORY_PRESSURE_WATCH, MEMORY_PRESSURE_WRITE, NVM_BIN, NVM_CD_FLAGS, NVM_DIR, NVM_INC, OLDPWD, PATH, PWD, QT_ACCESSIBILITY, QT_IM_MODULE, QT_IM_MODULES, SHELL, SHLVL, SSH_AUTH_SOCK, SYSTEMD_EXEC_PID, TERM, USER, USERNAME, VTE_VERSION, WAYLAND_DISPLAY, XAUTHORITY, XDG_CONFIG_DIRS, XDG_CURRENT_DESKTOP, XDG_DATA_DIRS, XDG_MENU_PREFIX, XDG_RUNTIME_DIR, XDG_SESSION_CLASS, XDG_SESSION_DESKTOP, XDG_SESSION_TYPE, XMODIFIERS, _
Go Version: go1.25.4
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_json_memory: "34078720", max_request_json_strings: "1000", max_request_size: "33554432", tls: "disabled")
Log Level:
Recovery Mode: false
Storage: inmem
Version: OpenBao v2.0.0-HEAD, built 2025-11-24T14:37:04Z
Version Sha: 4bfd70723d4f9b82be00e87b8c018ac661dd9b99
==> OpenBao server started! Log data will stream in below:
2025-12-17T18:54:43.589-0600 [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2025-12-17T18:54:43.589-0600 [WARN] no `api_addr` value specified in config or in BAO_API_ADDR; falling back to detection if possible, but this value should be manually set
2025-12-17T18:54:43.589-0600 [INFO] core: Initializing version history cache for core
2025-12-17T18:54:43.590-0600 [INFO] core: security barrier not initialized
2025-12-17T18:54:43.590-0600 [INFO] core: security barrier initialized: stored=1 shares=1 threshold=1
2025-12-17T18:54:43.590-0600 [INFO] core: post-unseal setup starting
2025-12-17T18:54:43.594-0600 [INFO] core: loaded wrapping token key
2025-12-17T18:54:43.594-0600 [INFO] core: successfully setup plugin catalog: plugin-directory=""
2025-12-17T18:54:43.594-0600 [INFO] core: reading transactional mount table
2025-12-17T18:54:43.594-0600 [INFO] core: no mounts in transactional mount table; adding default mount table
2025-12-17T18:54:43.594-0600 [INFO] core: successfully mounted: type=cubbyhole version="v2.0.0+builtin.bao" path=cubbyhole/ namespace="ID: root. Path: "
2025-12-17T18:54:43.594-0600 [INFO] core: successfully mounted: type=system version="v2.0.0+builtin.bao" path=sys/ namespace="ID: root. Path: "
2025-12-17T18:54:43.595-0600 [INFO] core: successfully mounted: type=identity version="v2.0.0+builtin.bao" path=identity/ namespace="ID: root. Path: "
2025-12-17T18:54:43.595-0600 [INFO] core: reading transactional auth mount table
2025-12-17T18:54:43.595-0600 [INFO] core: no auth mounts in transactional auth mount table; adding default auth mount table
2025-12-17T18:54:43.595-0600 [INFO] core: successfully mounted: type=token version="v2.0.0+builtin.bao" path=token/ namespace="ID: root. Path: "
2025-12-17T18:54:43.595-0600 [INFO] rollback: Starting the rollback manager with 256 workers
2025-12-17T18:54:43.595-0600 [INFO] rollback: starting rollback manager
2025-12-17T18:54:43.595-0600 [INFO] core: restoring leases
2025-12-17T18:54:43.596-0600 [INFO] expiration: lease restore complete
2025-12-17T18:54:43.596-0600 [INFO] identity: entities restored
2025-12-17T18:54:43.596-0600 [INFO] identity: groups restored
2025-12-17T18:54:43.597-0600 [INFO] core: Recorded vault version: vault version=2.0.0 upgrade time="2025-12-18 00:54:43.596939088 +0000 UTC" build date=2025-11-24T14:37:04Z
2025-12-17T18:54:43.597-0600 [INFO] core: post-unseal setup complete
2025-12-17T18:54:43.598-0600 [INFO] core: root token generated
2025-12-17T18:54:43.598-0600 [INFO] core: pre-seal teardown starting
2025-12-17T18:54:43.598-0600 [INFO] rollback: stopping rollback manager
2025-12-17T18:54:43.598-0600 [INFO] core: pre-seal teardown complete
2025-12-17T18:54:43.598-0600 [INFO] core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2025-12-17T18:54:43.598-0600 [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2025-12-17T18:54:43.598-0600 [INFO] core: post-unseal setup starting
2025-12-17T18:54:43.598-0600 [INFO] core: loaded wrapping token key
2025-12-17T18:54:43.598-0600 [INFO] core: successfully setup plugin catalog: plugin-directory=""
2025-12-17T18:54:43.598-0600 [INFO] core: reading transactional mount table
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=identity version="v2.0.0+builtin.bao" path=identity/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=system version="v2.0.0+builtin.bao" path=sys/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=cubbyhole version="v2.0.0+builtin.bao" path=cubbyhole/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] core: reading transactional auth mount table
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=token version="v2.0.0+builtin.bao" path=token/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] rollback: Starting the rollback manager with 256 workers
2025-12-17T18:54:43.599-0600 [INFO] rollback: starting rollback manager
2025-12-17T18:54:43.599-0600 [INFO] core: restoring leases
2025-12-17T18:54:43.600-0600 [INFO] expiration: lease restore complete
2025-12-17T18:54:43.600-0600 [INFO] identity: entities restored
2025-12-17T18:54:43.600-0600 [INFO] identity: groups restored
2025-12-17T18:54:43.600-0600 [INFO] core: post-unseal setup complete
2025-12-17T18:54:43.600-0600 [INFO] core: vault is unsealed
2025-12-17T18:54:43.601-0600 [INFO] expiration: revoked lease: lease_id=auth/token/root/habcd0ed9fa1b0ddb75ef175989665caf0313aa4cbbf6283009d40ab41aef6b76
2025-12-17T18:54:43.602-0600 [INFO] core: successful mount: namespace="" path=secret/ type=kv version=""
2025-12-17T18:54:43.613-0600 [INFO] secrets.kv.kv_3770cc65: collecting keys to upgrade
2025-12-17T18:54:43.613-0600 [INFO] secrets.kv.kv_3770cc65: done collecting keys: num_keys=1
2025-12-17T18:54:43.613-0600 [INFO] secrets.kv.kv_3770cc65: upgrading keys finished
WARNING! dev mode is enabled! In this mode, OpenBao runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using OpenBao.
You may need to set the following environment variables:
$ export BAO_ADDR='http://127.0.0.1:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: 7xrzsCSGsM3vslebvIEZOKynJ0+CwbFI+uOAoqwuWyE=
Root Token: myweakdevtoken
Development mode should NOT be used in production installations!
Now this is not a production instance and I wouldn’t use it as such, but we can use it to test KMS settings in RustFS
Let’s check the status of the Bao server from another terminal:
builder@LuiGi:~/rustfs$ export VAULT_ADDR='http://localhost:8200'
builder@LuiGi:~/rustfs$ export VAULT_DEV_ROOT_TOKEN_ID=myweakdevtoken
builder@LuiGi:~/rustfs$ bao status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 2.0.0-HEAD
Build Date 2025-11-24T14:37:04Z
Storage Type inmem
Cluster Name vault-cluster-54223770
Cluster ID 525d9c85-37a8-236c-8232-32de8a6f5307
HA Enabled false
Then a quick check we can put and retrieve a key
builder@LuiGi:~/rustfs$ bao kv put secret/funky cold=medina
== Secret Path ==
secret/data/funky
======= Metadata =======
Key Value
--- -----
created_time 2025-12-18T01:08:23.704580441Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
builder@LuiGi:~/rustfs$ bao kv get secret/funky
== Secret Path ==
secret/data/funky
======= Metadata =======
Key Value
--- -----
created_time 2025-12-18T01:08:23.704580441Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
==== Data ====
Key Value
--- -----
cold medina
KMS in RustFS with OpenBao
let’s use that token and the local http://127.0.0.1:8200 Bao (vault) endpoint
I know I enabled transit in OpenBao
builder@LuiGi:~/rustfs$ bao secrets enable transit
Success! Enabled the transit secrets engine at: transit/
builder@LuiGi:~/rustfs$ bao write -f transit/keys/rustfs-master-key
Key Value
--- -----
allow_plaintext_backup false
auto_rotate_period 0s
deletion_allowed false
derived false
exportable false
imported_key false
keys map[1:1766020909]
latest_version 1
min_available_version 0
min_decryption_version 1
min_encryption_version 0
name rustfs-master-key
soft_deleted false
supports_decryption true
supports_derivation true
supports_encryption true
supports_signing false
type aes256-gcm96
and I know I created at least one key (albiet just guessed on a random string for values)
builder@LuiGi:~/rustfs$ bao kv put secret/rustfs/kms/keys rustfs-master=asdfasfdsafasdf
======= Secret Path =======
secret/data/rustfs/kms/keys
======= Metadata =======
Key Value
--- -----
created_time 2025-12-18T01:11:47.496869031Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
But no matter what permutation I tried, I could not seem to start KMS properly
this meant pushes to the encrypted bucket would fail
$ aws s3 cp ~/ss2b.sh s3://mytestbucket/ss2b.sh.3
upload failed: ../../ss2b.sh to s3://mytestbucket/ss2b.sh.3 An error occurred (InternalError) when calling the PutObject operation (reached max retries: 2): Io error: Failed to create data key: Backend error: Failed to read key from Vault: An error occurred with the request
I reviewed the OpenBao docs and watched the logs:
builder@LuiGi:~/rustfs$ bao server -dev -dev-root-token-id="myweakdevtoken"
==> OpenBao server configuration:
Administrative Namespace:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Environment Variables: AWS_ENDPOINT_URL, COLORTERM, DBUS_SESSION_BUS_ADDRESS, DEBUGINFOD_URLS, DESKTOP_SESSION, DISPLAY, GDMSESSION, GEMINI_API_KEY, GEM_HOME, GNOME_DESKTOP_SESSION_ID, GNOME_SETUP_DISPLAY, GNOME_SHELL_SESSION_MODE, GNOME_TERMINAL_SCREEN, GNOME_TERMINAL_SERVICE, GPG_AGENT_INFO, GTK_MODULES, HOME, HOMEBREW_CELLAR, HOMEBREW_PREFIX, HOMEBREW_REPOSITORY, IM_CONFIG_PHASE, INFOPATH, LANG, LESSCLOSE, LESSOPEN, LOGNAME, LS_COLORS, MEMORY_PRESSURE_WATCH, MEMORY_PRESSURE_WRITE, NVM_BIN, NVM_CD_FLAGS, NVM_DIR, NVM_INC, OLDPWD, PATH, PWD, QT_ACCESSIBILITY, QT_IM_MODULE, QT_IM_MODULES, SHELL, SHLVL, SSH_AUTH_SOCK, SYSTEMD_EXEC_PID, TERM, USER, USERNAME, VTE_VERSION, WAYLAND_DISPLAY, XAUTHORITY, XDG_CONFIG_DIRS, XDG_CURRENT_DESKTOP, XDG_DATA_DIRS, XDG_MENU_PREFIX, XDG_RUNTIME_DIR, XDG_SESSION_CLASS, XDG_SESSION_DESKTOP, XDG_SESSION_TYPE, XMODIFIERS, _
Go Version: go1.25.4
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_json_memory: "34078720", max_request_json_strings: "1000", max_request_size: "33554432", tls: "disabled")
Log Level:
Recovery Mode: false
Storage: inmem
Version: OpenBao v2.0.0-HEAD, built 2025-11-24T14:37:04Z
Version Sha: 4bfd70723d4f9b82be00e87b8c018ac661dd9b99
==> OpenBao server started! Log data will stream in below:
2025-12-17T18:54:43.589-0600 [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2025-12-17T18:54:43.589-0600 [WARN] no `api_addr` value specified in config or in BAO_API_ADDR; falling back to detection if possible, but this value should be manually set
2025-12-17T18:54:43.589-0600 [INFO] core: Initializing version history cache for core
2025-12-17T18:54:43.590-0600 [INFO] core: security barrier not initialized
2025-12-17T18:54:43.590-0600 [INFO] core: security barrier initialized: stored=1 shares=1 threshold=1
2025-12-17T18:54:43.590-0600 [INFO] core: post-unseal setup starting
2025-12-17T18:54:43.594-0600 [INFO] core: loaded wrapping token key
2025-12-17T18:54:43.594-0600 [INFO] core: successfully setup plugin catalog: plugin-directory=""
2025-12-17T18:54:43.594-0600 [INFO] core: reading transactional mount table
2025-12-17T18:54:43.594-0600 [INFO] core: no mounts in transactional mount table; adding default mount table
2025-12-17T18:54:43.594-0600 [INFO] core: successfully mounted: type=cubbyhole version="v2.0.0+builtin.bao" path=cubbyhole/ namespace="ID: root. Path: "
2025-12-17T18:54:43.594-0600 [INFO] core: successfully mounted: type=system version="v2.0.0+builtin.bao" path=sys/ namespace="ID: root. Path: "
2025-12-17T18:54:43.595-0600 [INFO] core: successfully mounted: type=identity version="v2.0.0+builtin.bao" path=identity/ namespace="ID: root. Path: "
2025-12-17T18:54:43.595-0600 [INFO] core: reading transactional auth mount table
2025-12-17T18:54:43.595-0600 [INFO] core: no auth mounts in transactional auth mount table; adding default auth mount table
2025-12-17T18:54:43.595-0600 [INFO] core: successfully mounted: type=token version="v2.0.0+builtin.bao" path=token/ namespace="ID: root. Path: "
2025-12-17T18:54:43.595-0600 [INFO] rollback: Starting the rollback manager with 256 workers
2025-12-17T18:54:43.595-0600 [INFO] rollback: starting rollback manager
2025-12-17T18:54:43.595-0600 [INFO] core: restoring leases
2025-12-17T18:54:43.596-0600 [INFO] expiration: lease restore complete
2025-12-17T18:54:43.596-0600 [INFO] identity: entities restored
2025-12-17T18:54:43.596-0600 [INFO] identity: groups restored
2025-12-17T18:54:43.597-0600 [INFO] core: Recorded vault version: vault version=2.0.0 upgrade time="2025-12-18 00:54:43.596939088 +0000 UTC" build date=2025-11-24T14:37:04Z
2025-12-17T18:54:43.597-0600 [INFO] core: post-unseal setup complete
2025-12-17T18:54:43.598-0600 [INFO] core: root token generated
2025-12-17T18:54:43.598-0600 [INFO] core: pre-seal teardown starting
2025-12-17T18:54:43.598-0600 [INFO] rollback: stopping rollback manager
2025-12-17T18:54:43.598-0600 [INFO] core: pre-seal teardown complete
2025-12-17T18:54:43.598-0600 [INFO] core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2025-12-17T18:54:43.598-0600 [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2025-12-17T18:54:43.598-0600 [INFO] core: post-unseal setup starting
2025-12-17T18:54:43.598-0600 [INFO] core: loaded wrapping token key
2025-12-17T18:54:43.598-0600 [INFO] core: successfully setup plugin catalog: plugin-directory=""
2025-12-17T18:54:43.598-0600 [INFO] core: reading transactional mount table
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=identity version="v2.0.0+builtin.bao" path=identity/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=system version="v2.0.0+builtin.bao" path=sys/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=cubbyhole version="v2.0.0+builtin.bao" path=cubbyhole/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] core: reading transactional auth mount table
2025-12-17T18:54:43.599-0600 [INFO] core: successfully mounted: type=token version="v2.0.0+builtin.bao" path=token/ namespace="ID: root. Path: "
2025-12-17T18:54:43.599-0600 [INFO] rollback: Starting the rollback manager with 256 workers
2025-12-17T18:54:43.599-0600 [INFO] rollback: starting rollback manager
2025-12-17T18:54:43.599-0600 [INFO] core: restoring leases
2025-12-17T18:54:43.600-0600 [INFO] expiration: lease restore complete
2025-12-17T18:54:43.600-0600 [INFO] identity: entities restored
2025-12-17T18:54:43.600-0600 [INFO] identity: groups restored
2025-12-17T18:54:43.600-0600 [INFO] core: post-unseal setup complete
2025-12-17T18:54:43.600-0600 [INFO] core: vault is unsealed
2025-12-17T18:54:43.601-0600 [INFO] expiration: revoked lease: lease_id=auth/token/root/habcd0ed9fa1b0ddb75ef175989665caf0313aa4cbbf6283009d40ab41aef6b76
2025-12-17T18:54:43.602-0600 [INFO] core: successful mount: namespace="" path=secret/ type=kv version=""
2025-12-17T18:54:43.613-0600 [INFO] secrets.kv.kv_3770cc65: collecting keys to upgrade
2025-12-17T18:54:43.613-0600 [INFO] secrets.kv.kv_3770cc65: done collecting keys: num_keys=1
2025-12-17T18:54:43.613-0600 [INFO] secrets.kv.kv_3770cc65: upgrading keys finished
WARNING! dev mode is enabled! In this mode, OpenBao runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using OpenBao.
You may need to set the following environment variables:
$ export BAO_ADDR='http://127.0.0.1:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: 7xrzsCSGsM3vslebvIEZOKynJ0+CwbFI+uOAoqwuWyE=
Root Token: myweakdevtoken
Development mode should NOT be used in production installations!
2025-12-17T19:21:36.252-0600 [INFO] core: successful mount: namespace="" path=transit/ type=transit version=""
but I just could not get KMS to work locally.
Also, the docs for rustfs mentioned a few KMS’es we could use like AWS and Azure AKV but I see no such options locally.
Using Helm
To install with Helm into Kubernetes, first add the chart repo:
$ helm repo add openbao https://openbao.github.io/openbao-helm
"openbao" has been added to your repositories
Then we can install with helm
$ helm install openbao openbao/openbao
NAME: openbao
LAST DEPLOYED: Fri Dec 19 07:28:41 2025
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing OpenBao!
Now that you have deployed OpenBao, you should look over the docs on using
OpenBao with Kubernetes available here:
https://openbao.org/docs/
Your release is named openbao. To learn more about the release, try:
$ helm status openbao
$ helm get manifest openbao
however, we may want to enable dev mode to make our testing easier
$ helm uninstall openbao
release "openbao" uninstalled
$ helm install openbao --set "server.dev.enabled=true" openbao/openbao
NAME: openbao
LAST DEPLOYED: Fri Dec 19 07:31:03 2025
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing OpenBao!
Now that you have deployed OpenBao, you should look over the docs on using
OpenBao with Kubernetes available here:
https://openbao.org/docs/
Your release is named openbao. To learn more about the release, try:
$ helm status openbao
$ helm get manifest openbao
I can fetch the token
$ kubectl logs openbao-0 | tail -n 20
2025-12-19T13:31:09.723Z [INFO] core: successful mount: namespace="" path=secret/ type=kv version=""
2025-12-19T13:31:09.733Z [INFO] secrets.kv.kv_ae4eef71: collecting keys to upgrade
2025-12-19T13:31:09.734Z [INFO] secrets.kv.kv_ae4eef71: done collecting keys: num_keys=1
2025-12-19T13:31:09.734Z [INFO] secrets.kv.kv_ae4eef71: upgrading keys finished
WARNING! dev mode is enabled! In this mode, OpenBao runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using OpenBao.
You may need to set the following environment variables:
$ export BAO_ADDR='http://[::]:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: KVcQNTPA6bBbkLjjjrOlNyitSFLx1brrhd+6H2gZFBU=
Root Token: root
Development mode should NOT be used in production installations!
Because it is in development mode, we can login and enable the transit engine and add a transit key or two
$ kubectl exec -it openbao-0 -- /bin/sh
/ $ bao secrets enable transit
Success! Enabled the transit secrets engine at: transit/
/ $ bao write -f transit/keys/rustfs-master-key
Key Value
--- -----
allow_plaintext_backup false
auto_rotate_period 0s
deletion_allowed false
derived false
exportable false
imported_key false
keys map[1:1766151753]
latest_version 1
min_available_version 0
min_decryption_version 1
min_encryption_version 0
name rustfs-master-key
soft_deleted false
supports_decryption true
supports_derivation true
supports_encryption true
supports_signing false
type aes256-gcm96
/ $ bao write -f transit/keys/rustfs-master
Key Value
--- -----
allow_plaintext_backup false
auto_rotate_period 0s
deletion_allowed false
derived false
exportable false
imported_key false
keys map[1:1766151766]
latest_version 1
min_available_version 0
min_decryption_version 1
min_encryption_version 0
name rustfs-master
soft_deleted false
supports_decryption true
supports_derivation true
supports_encryption true
supports_signing false
type aes256-gcm96
I used kubedns to reference the Bao service but still I get a “failed to list keys” error
I tried using a bespoke token with the perms I knew it needed
/ $ cat /tmp/kms-policy.hcl
# Allow RustFS to use the transit engine
path "transit/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Allow RustFS to use the KV store
path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
/ $ bao policy write kmspolicy /tmp/kms-policy.hcl
Success! Uploaded policy: kmspolicy
/ $ vault token create -policy="kmspolicy"
Key Value
--- -----
token s.pmlD7mrRcwhgIxAGLabPrOkY
token_accessor NxDYlA9D9pne6VFlfN2hfvGO
token_duration 768h
token_renewable true
token_policies ["default" "kmspolicy"]
identity_policies []
policies ["default" "kmspolicy"]
I then thought, let’s use NodePort to avoid any KubeDNS issues
uilder@DESKTOP-QADGF36:~$ kubectl apply -f ./openbao.nodeport.yaml
service/openbaonp created
builder@DESKTOP-QADGF36:~$ cat openbao.nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: openbaonp
spec:
ports:
- name: svcobweb
nodePort: 32200
port: 8200
protocol: TCP
targetPort: 8200
- name: svcobprt
nodePort: 32201
port: 8201
protocol: TCP
targetPort: 8201
selector:
app.kubernetes.io/instance: openbao
app.kubernetes.io/name: openbao
component: server
type: NodePort
builder@DESKTOP-QADGF36:~$ kubectl get svc | grep openbao
openbao ClusterIP 10.43.216.22 <none> 8200/TCP,8201/TCP 105m
openbao-agent-injector-svc ClusterIP 10.43.205.17 <none> 443/TCP 105m
openbao-internal ClusterIP None <none> 8200/TCP,8201/TCP 105m
openbaonp NodePort 10.43.199.33 <none> 8200:32200/TCP,8201:32201/TCP 17s
This FINALLY worked. I delayed this article a day because I just was damned determined to sort out my RustFS/OpenBao issues
I can now create a new KMS key
here we can see it in action
Summary
Today we covered a lot - we looked at RustFS, a very capable MinIO replacement for providing S3-compliant object storage. In trying to leverage its built-in Server Side Encryption (SSE) capabilities, we rant into a few challenges using OpenBao. I believe now the fault lies with me or my cluster, not the tooling.
In the end, we setup a simple development mode OpenBao (Open-Source Hashi Vault) with transit enabled and KMS keys used in a RustFS instance both hosted in a newer K3s cluster (running v1.31.9+k3s1 presently). I did end up using a NodePort service which I think may have been the root issue all along.
Most folks setting this up are going to use a production level OpenBao or Vault instance with proper TLS and routing, so really this just would affect dev clusters and local development.
The other approach one could use is to just encrypt the files locally and store them encrypted in the buckets. Then you (or your app) really controls the keys and usage. The only problem here is if your consumer (apps/systems) aren’t tightly controlled (then you would want SSE to own the encryption at rest).
My next steps are likely to setup a proper OpenBao in my production cluster -OR- set it up on a proper docker host with external routing and try all these steps again. I may or may not setup RustFS on my production cluster. I’m more keen to try this on a NAS or physical system serving as a NAS. MinIO, to me, is very much about storage and I often use this when I have tooling that is rather dependent on S3 storage (e.g. Rancher Longhorn).
