Published: Sep 10, 2024 by Isaac Johnson

In our last two posts we explored Cloud-based backups. But what about on-prem? What choices do we have outside the major cloud providers?

Today we will look at using on-prem Synology NASes and how to set them up for NFS shares. We’ll explore Minio containerized and running on our NAS and then using Minio with s3fs for backups and syncing with cron.

Lastly, we’ll look at monitors using Datadog, OpenStatusand using Synology CloudSync to backup our NAS volumes to cloud storage such as GCP Buckets.

NAS

On-prem I keep two active Synology NAS appliances

The newer device is a Synology DS220+ with 10Gb of Memory (which I upgraded myself) and 10Tb of Storage (Raid 1+0).

The older of the two is a DS216Play with 6 Tb, also Raid 1+0. I tend to use WD Red Drives and only recently did this one have some failures that required replacements of both within a month of each other - they had lasted WAY longer than the specs, nearly 8 years of heavy constant use.

Updates are easy

Incurring a minor service blip as it restarts

I just need to watch the page and see when it’s completed

I’m not here to sell hardware, but I’m a fan of these and i think they’re quite excellent. my most important files, like photos are replicated between the two.

And for what it’s worth, I bought my last not from Amazon and instead from Microcenter. If I did it today, I would get the DS1522.

File shares

Knowing that my NAS(es) run the same basic OS, creating fileshares is pretty much identical between the two.

For instance, let’s say I desire a new block of space as a “Shared Folder” in the older NAS.

I can go to Shared Folders and click “Create”

I’ll give it a name such as “linuxbackups”. I do not want soft-delete so I’ll disable the “Recycle Bin”

I could encrypt the folder, but I won’t

Once I confirm settings

I can grant access to specific users

Once created, I’ll edit the share so we can get to the NFS Permissions

There in the “NFS Permissions” tab I’ll grant access to my local CIDR block for machines

Don’t forget to save it to take effect

Let’s say we want this for backups.

All I need to do is create a mount and an entry in fstab. Because my Dockerhost lives in the right CIDR, there is no need for user/pass authentication with NFS

builder@builder-T100:~$ sudo mkdir /mnt/linuxbackups
builder@builder-T100:~$ sudo vi /etc/fstab
builder@builder-T100:~$ tail -n1 /etc/fstab
192.168.1.129:/volume1/linuxbackups /mnt/linuxbackups nfs    auto,nofail,noatime,nolock,intr,tcp,actimeo=1800        0       0
builder@builder-T100:~$ sudo mount -a

Now to test by creating a text file

builder@builder-T100:~$ echo "this is a test" | sudo tee /mnt/linuxbackups/testfile.txt
this is a test

Which I see show up right away

The fact is I’ve used this NAS for PostgreSQL backups for some time

builder@builder-T100:~$ cat /etc/fstab | grep 1.129
192.168.1.129:/volume1/postgres-prod-dbbackups  /mnt/psqlbackups nfs    auto,nofail,noatime,nolock,intr,tcp,actimeo=1800        0       0
192.168.1.129:/volume1/linuxbackups /mnt/linuxbackups nfs    auto,nofail,noatime,nolock,intr,tcp,actimeo=1800        0       0

We can see that writeup from May of 2023 here.

Minio and Docker Compose

The NAS does have the power to run docker

ijohnson@sirnasilot:~$ sudo docker ps
Password:
CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS          PORTS
       NAMES
5fa76248b69d   portainer/portainer-ee:latest   "/portainer"             14 months ago   Up 27 minutes   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp, 9000/tcp   portaineree
83af22fb4b32   registry:latest                 "/entrypoint.sh /etc…"   14 months ago   Up 27 minutes   0.0.0.0:5050->5000/tcp, :::5050->5000/tcp
       registry-1

This gave me the idea of using my newer NAS to actually run Minio in Docker (steps here)

As I want to clone the repo from Github, I need GIT installed.

I can actually use the Synology Package Center to install a GIT Server which includes the GIT binary

I’ll clone the repo

ijohnson@sirnasilot:~$ git clone https://github.com/minio/minio.git
Cloning into 'minio'...
remote: Enumerating objects: 106651, done.
remote: Counting objects: 100% (24644/24644), done.
remote: Compressing objects: 100% (845/845), done.
remote: Total 106651 (delta 23994), reused 23848 (delta 23796), pack-reused 82007 (from 1)
Receiving objects: 100% (106651/106651), 125.70 MiB | 24.33 MiB/s, done.
Resolving deltas: 100% (77976/77976), done.

Then use Docker Compose to pull the images

ijohnson@sirnasilot:~$ cd minio/docs/orchestration/docker-compose/

ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo docker-compose pull
[+] Running 19/19
 ⠿ minio2 Skipped - Image is already being pulled by minio1                                                                                                                                  0.0s
 ⠿ minio3 Skipped - Image is already being pulled by minio1                                                                                                                                  0.0s
 ⠿ minio4 Skipped - Image is already being pulled by minio1                                                                                                                                  0.0s
 ⠿ minio1 Pulled                                                                                                                                                                            22.5s
   ⠿ 5f328c14e09d Pull complete                                                                                                                                                              3.7s
   ⠿ 7c0037c31c26 Pull complete                                                                                                                                                              5.8s
   ⠿ cb8ce3bb8f9a Pull complete                                                                                                                                                              7.3s
   ⠿ f4b17c317d15 Pull complete                                                                                                                                                             10.9s
   ⠿ 4cac341ffb02 Pull complete                                                                                                                                                             12.4s
   ⠿ a08f6ff4d495 Pull complete                                                                                                                                                             13.6s
   ⠿ 429760338d49 Pull complete                                                                                                                                                             14.3s
   ⠿ f5eabdee403e Pull complete                                                                                                                                                             15.2s
   ⠿ b8d89195f89b Pull complete                                                                                                                                                             15.9s
 ⠿ nginx Pulled                                                                                                                                                                             14.4s
   ⠿ df20fa9351a1 Pull complete                                                                                                                                                              0.9s
   ⠿ 3db268b1fe8f Pull complete                                                                                                                                                              3.1s
   ⠿ f682f0660e7a Pull complete                                                                                                                                                              5.4s
   ⠿ 7eb0e8838bc0 Pull complete                                                                                                                                                              7.0s
   ⠿ e8bf1226cc17 Pull complete             

I realized the first mistake when using docker-compose up was that I did not change the drive mounts

So I stopped it

I made some folders for “data1” and “data2”

ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data1-1
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data1-2
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data2-2
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data2-2
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data3-2
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data3-2
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data4-2
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo mkdir /volume1/minio/data4-2

I then gave it a real password and set the volumes correctly

ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ vi docker-compose.yaml
ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ cat docker-compose.yaml
version: '3.7'

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2024-08-29T01-40-52Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
    - "9001"
  environment:
    MINIO_ROOT_USER: admin
    MINIO_ROOT_PASSWORD: nottherealpassword
  healthcheck:
    test: ["CMD", "mc", "ready", "local"]
    interval: 5s
    timeout: 5s
    retries: 5

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - data2-1:/data1
      - data2-2:/data2

  minio3:
    <<: *minio-common
    hostname: minio3
    volumes:
      - data3-1:/data1
      - data3-2:/data2

  minio4:
    <<: *minio-common
    hostname: minio4
    volumes:
      - data4-1:/data1
      - data4-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1-1:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data1-1
  data1-2:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data1-2
  data2-1:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data2-1
  data2-2:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data2-2
  data3-1:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data3-1
  data3-2:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data3-2
  data4-1:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data4-1
  data4-2:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /volume1/minio/data4-2

I’m running interactive to start so I can debug any issues

ijohnson@sirnasilot:~/minio/docs/orchestration/docker-compose$ sudo docker-compose up
[+] Running 5/5
 ⠿ Container docker-compose-minio3-1  Recreated                                                                                                    5.8s
 ⠿ Container docker-compose-minio4-1  Recreated                                                                                                    7.5s
 ⠿ Container docker-compose-minio1-1  Recreated                                                                                                    6.5s
 ⠿ Container docker-compose-minio2-1  Recreated                                                                                                    7.4s
 ⠿ Container docker-compose-nginx-1   Recreated                                                                                                    2.0s
Attaching to docker-compose-minio1-1, docker-compose-minio2-1, docker-compose-minio3-1, docker-compose-minio4-1, docker-compose-nginx-1
docker-compose-minio3-1  | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio3-1  | INFO: Following servers are currently offline or unreachable [http://minio3:9000->http://minio4:9000 is unreachable: remote disconnected http://minio3:9000->http://minio1:9000 is unreachable: remote disconnected http://minio3:9000->http://minio2:9000 is unreachable: remote disconnected]
docker-compose-minio3-1  | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio3-1  | INFO: Following servers are currently offline or unreachable [http://minio3:9000->http://minio4:9000 is unreachable: remote disconnected http://minio3:9000->http://minio1:9000 is unreachable: remote disconnected http://minio3:9000->http://minio2:9000 is unreachable: remote disconnected]
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:05 UTC 09/07/2024
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: no route to host (*net.OpError) Sleeping 795ms (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:05 UTC 09/07/2024
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: connect: no route to host (*net.OpError) Sleeping 275ms (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio3-1  | INFO: Following servers are currently offline or unreachable [http://minio3:9000->http://minio4:9000 is unreachable: remote disconnected http://minio3:9000->http://minio1:9000 is unreachable: remote disconnected http://minio3:9000->http://minio2:9000 is unreachable: remote disconnected]
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:07 UTC 09/07/2024
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: i/o timeout (*net.OpError) Sleeping 0s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:07 UTC 09/07/2024
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: i/o timeout (*net.OpError) Sleeping 0s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:07 UTC 09/07/2024
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: i/o timeout (*net.OpError) Sleeping 0s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:07 UTC 09/07/2024
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: i/o timeout (*net.OpError) Sleeping 0s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  | INFO: Waiting for at least 1 remote servers with valid configuration to be online
docker-compose-minio3-1  | INFO: Following servers are currently offline or unreachable [http://minio3:9000->http://minio4:9000 is unreachable: remote disconnected http://minio3:9000->http://minio2:9000 is unreachable: remote disconnected http://minio3:9000->http://minio1:9000 is unreachable: remote disconnected]
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:08 UTC 09/07/2024
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: connect: connection refused (*net.OpError) Sleeping 1.341s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  | INFO: Unable to use the drive http://minio1:9000/data1: drive not found, will be retried
docker-compose-minio3-1  | INFO: Unable to use the drive http://minio2:9000/data1: drive not found, will be retried
docker-compose-minio3-1  | INFO: Unable to use the drive http://minio1:9000/data2: drive not found, will be retried
docker-compose-minio3-1  | INFO: Unable to use the drive http://minio2:9000/data2: drive not found, will be retried
docker-compose-minio3-1  | INFO: Waiting for quorum READ healthcheck to succeed retrying in 28.326997ms.. possible cause unhealthy sets
docker-compose-minio3-1  | INFO: (Pool: 0 Set: 0 Healthy: false)
docker-compose-minio3-1  | INFO: Waiting for quorum READ healthcheck to succeed retrying in 218.712636ms.. possible cause unhealthy sets
docker-compose-minio3-1  | INFO: (Pool: 0 Set: 0 Healthy: false)
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:09 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: connect: connection refused (*net.OpError) Sleeping 1.592s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio4-1  | INFO: Unable to use the drive http://minio1:9000/data1: drive not found, will be retried
docker-compose-minio4-1  | INFO: Unable to use the drive http://minio2:9000/data1: drive not found, will be retried
docker-compose-minio4-1  | INFO: Unable to use the drive http://minio1:9000/data2: drive not found, will be retried
docker-compose-minio4-1  | INFO: Unable to use the drive http://minio2:9000/data2: drive not found, will be retried
docker-compose-minio4-1  | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio4-1  | INFO: Configured max API requests per node based on available memory: 712
docker-compose-minio4-1  | INFO: All MinIO sub-systems initialized successfully in 2.982574ms
docker-compose-minio4-1  | INFO: IAM load(startup) finished. (duration: 6.466975ms)
docker-compose-minio4-1  | ---------------------------
docker-compose-minio4-1  | MinIO Object Storage Server
docker-compose-minio4-1  | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio4-1  | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio4-1  | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio4-1  |
docker-compose-minio4-1  | API: http://172.18.0.3:9000  http://127.0.0.1:9000
docker-compose-minio4-1  | WebUI: http://172.18.0.3:9001 http://127.0.0.1:9001
docker-compose-minio4-1  |
docker-compose-minio4-1  | Docs: https://docs.min.io
docker-compose-minio4-1  | ---------------------------
docker-compose-minio3-1  | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio3-1  | INFO: Configured max API requests per node based on available memory: 719
docker-compose-minio3-1  | INFO: All MinIO sub-systems initialized successfully in 6.385275ms
docker-compose-minio3-1  | ---------------------------
docker-compose-minio3-1  | MinIO Object Storage Server
docker-compose-minio3-1  | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio3-1  | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio3-1  | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: http://172.18.0.2:9000  http://127.0.0.1:9000
docker-compose-minio3-1  | WebUI: http://172.18.0.2:9001 http://127.0.0.1:9001
docker-compose-minio3-1  |
docker-compose-minio3-1  | Docs: https://docs.min.io
docker-compose-minio3-1  | ---------------------------
docker-compose-minio3-1  | INFO: IAM load(startup) finished. (duration: 8.229168ms)
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:09 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: connect: connection refused (*net.OpError) Sleeping 1.276s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:10 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: i/o timeout (*net.OpError) Sleeping 0s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:10 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.05s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:10 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 605ms (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:10 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.068s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:11 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: connect: connection refused (*net.OpError) Sleeping 1.163s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:11 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio1:9000: dial tcp 172.18.0.4:9000: connect: connection refused (*net.OpError) Sleeping 1.769s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:11 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.423s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:11 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.907s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio1-1  | INFO: Unable to use the drive http://minio2:9000/data1: drive not found, will be retried
docker-compose-minio1-1  | INFO: Unable to use the drive http://minio2:9000/data2: drive not found, will be retried
docker-compose-minio1-1  | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio1-1  | INFO: Configured max API requests per node based on available memory: 705
docker-compose-minio1-1  | INFO: All MinIO sub-systems initialized successfully in 3.732779ms
docker-compose-minio1-1  | ---------------------------
docker-compose-minio1-1  | MinIO Object Storage Server
docker-compose-minio1-1  | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio1-1  | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio1-1  | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio1-1  |
docker-compose-minio1-1  | API: http://172.18.0.4:9000  http://127.0.0.1:9000
docker-compose-minio1-1  | WebUI: http://172.18.0.4:9001 http://127.0.0.1:9001
docker-compose-minio1-1  |
docker-compose-minio1-1  | Docs: https://docs.min.io
docker-compose-minio1-1  | ---------------------------
docker-compose-minio1-1  | INFO: IAM load(startup) finished. (duration: 3.435315ms)
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:12 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.022s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio1-1  |
docker-compose-minio1-1  | API: SYSTEM.grid
docker-compose-minio1-1  | Time: 11:44:13 UTC 09/07/2024
docker-compose-minio1-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio1-1  | Error: grid: http://minio1:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.291s (3) (*fmt.wrapError)
docker-compose-minio1-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio1-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio1-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio1-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio1-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio1-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:13 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.372s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio3-1  |
docker-compose-minio3-1  | API: SYSTEM.grid
docker-compose-minio3-1  | Time: 11:44:13 UTC 09/07/2024
docker-compose-minio3-1  | DeploymentID: 70f64fa5-5f0c-4b67-9d9b-5209ee42cdde
docker-compose-minio3-1  | Error: grid: http://minio3:9000 re-connecting to http://minio2:9000: dial tcp 172.18.0.5:9000: connect: connection refused (*net.OpError) Sleeping 1.246s (3) (*fmt.wrapError)
docker-compose-minio3-1  |        6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
docker-compose-minio3-1  |        5: internal/logger/logonce.go:149:logger.LogOnceIf()
docker-compose-minio3-1  |        4: internal/grid/connection.go:59:grid.gridLogOnceIf()
docker-compose-minio3-1  |        3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
docker-compose-minio3-1  |        2: internal/grid/connection.go:678:grid.(*Connection).connect()
docker-compose-minio3-1  |        1: internal/grid/connection.go:275:grid.newConnection.func3()
docker-compose-minio2-1  | INFO: Unable to use the drive http://minio1:9000/data1: drive not found, will be retried
docker-compose-minio2-1  | INFO: Unable to use the drive http://minio3:9000/data1: drive not found, will be retried
docker-compose-minio2-1  | INFO: Unable to use the drive http://minio1:9000/data2: drive not found, will be retried
docker-compose-minio2-1  | INFO: Unable to use the drive http://minio3:9000/data2: drive not found, will be retried
docker-compose-minio2-1  | INFO: Waiting for all MinIO sub-systems to be initialize...
docker-compose-minio2-1  | INFO: Configured max API requests per node based on available memory: 698
docker-compose-minio2-1  | INFO: All MinIO sub-systems initialized successfully in 6.797672ms
docker-compose-minio2-1  | ---------------------------
docker-compose-minio2-1  | MinIO Object Storage Server
docker-compose-minio2-1  | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio2-1  | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio2-1  | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio2-1  |
docker-compose-minio2-1  | API: http://172.18.0.5:9000  http://127.0.0.1:9000
docker-compose-minio2-1  | WebUI: http://172.18.0.5:9001 http://127.0.0.1:9001
docker-compose-minio2-1  |
docker-compose-minio2-1  | Docs: https://docs.min.io
docker-compose-minio2-1  | ---------------------------
docker-compose-minio2-1  | INFO: IAM load(startup) finished. (duration: 6.230698ms)

When it’s ready, we see at the end

docker-compose-minio2-1  | INFO: All MinIO sub-systems initialized successfully in 6.797672ms
docker-compose-minio2-1  | ---------------------------
docker-compose-minio2-1  | MinIO Object Storage Server
docker-compose-minio2-1  | Copyright: 2015-2024 MinIO, Inc.
docker-compose-minio2-1  | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
docker-compose-minio2-1  | Version: RELEASE.2024-08-29T01-40-52Z (go1.22.6 linux/amd64)
docker-compose-minio2-1  |
docker-compose-minio2-1  | API: http://172.18.0.5:9000  http://127.0.0.1:9000
docker-compose-minio2-1  | WebUI: http://172.18.0.5:9001 http://127.0.0.1:9001
docker-compose-minio2-1  |
docker-compose-minio2-1  | Docs: https://docs.min.io
docker-compose-minio2-1  | ---------------------------
docker-compose-minio2-1  | INFO: IAM load(startup) finished. (duration: 6.230698ms)
docker-compose-nginx-1   | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
docker-compose-nginx-1   | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
docker-compose-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
docker-compose-nginx-1   | 10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
docker-compose-nginx-1   | 10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
docker-compose-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
docker-compose-nginx-1   | /docker-entrypoint.sh: Configuration complete; ready for start up

I can now login

And see the dashboard

Before I go on, I want to stop the instance and then run as a daemon (background)

To use Minio, we need to create an access key

It will propose one, which looks fine to me. We can optionally set an expiry date as well

It then shows them once

and prompts for download as a JSON file.

builder@DESKTOP-QADGF36:~$ cat /mnt/c/Users/isaac/Downloads/credentials\ \(1\).json
{"url":"http://192.168.1.116:9001/api/v1/service-account-credentials","accessKey":"Oc3ZydYnG8fDWvWLPpTu","secretKey":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","api":"s3v4","path":"auto"}

I can now see I have a valid key

Next, to use this with S3 compliant tooling, I’ll need a bucket. We can create a Bucket from the Buckets section under Administrator.

I’ll create a Bucket for databasebackups. I will not enable versioning, but for things you wish to be extra safe on, you could enable it

I now have a “Database Backups” bucket ‘databasebackups”

Usage

I’ll add the secret access key and secret to my ~/.aws/credentials file

builder@DESKTOP-QADGF36:~$ vi ~/.aws/credentials
builder@DESKTOP-QADGF36:~$ cat ~/.aws/credentials | tail -n3
[miniosirnasilot]
aws_access_key_id = Oc3ZydYnG8fDWvWLPpTu
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

We can now use that with the AWS CLI

builder@DESKTOP-QADGF36:~$ aws --profile miniosirnasilot --endpoint-url http://192.168.1.116:9000 s3 ls s3://databasebackups/
builder@DESKTOP-QADGF36:~$ aws --profile miniosirnasilot --endpoint-url http://192.168.1.116:9000 s3 ls
2024-09-07 06:57:55 databasebackups

I’ll admit I did two mistakes first, but in case others do the same:

First, I forgot that this is HTTP not HTTPS and saw SSL errors till I realized i left the ‘s’ in the URL

builder@DESKTOP-QADGF36:~$ aws --endpoint-url https://192.168.1.116:9000 s3 ls

SSL validation failed for https://192.168.1.116:9000/ [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1145)

And I also forgot to add “–profile” so it picked up my default AWS creds

builder@DESKTOP-QADGF36:~$ aws --endpoint-url http://192.168.1.116:9000 s3 ls

An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The Access Key Id you provided does not exist in our records.

Minio with Fuse

We can use Minio just like any S3 bucket in AWS.

For instance, let’s add s3fs to the dockerhost

builder@builder-T100:~$ sudo apt install s3fs
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  apg gnome-control-center-faces gnome-online-accounts libcolord-gtk1 libfreerdp-server2-2 libgnome-bg-4-1 libgsound0
  libgssdp-1.2-0 libgupnp-1.2-1 libgupnp-av-1.0-3 libgupnp-dlna-2.0-4 libntfs-3g89 librygel-core-2.6-2 librygel-db-2.6-2
  librygel-renderer-2.6-2 librygel-server-2.6-2 libvncserver1 mobile-broadband-provider-info network-manager-gnome
  python3-macaroonbakery python3-protobuf python3-pymacaroons python3-rfc3339 python3-tz rygel
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  s3fs
0 upgraded, 1 newly installed, 0 to remove and 166 not upgraded.
Need to get 297 kB of archives.
After this operation, 794 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 s3fs amd64 1.90-1 [297 kB]
Fetched 297 kB in 0s (757 kB/s)
Selecting previously unselected package s3fs.
(Reading database ... 316316 files and directories currently installed.)
Preparing to unpack .../archives/s3fs_1.90-1_amd64.deb ...
Unpacking s3fs (1.90-1) ...
Setting up s3fs (1.90-1) ...
Processing triggers for man-db (2.10.2-1) ...

I’ll now make a mount just for the S3 backend to my NAS

builder@builder-T100:~$ sudo mkdir /mnt/s3fs-dbbackups

I’ll need to make a credentials file with “ACCESSKEY:SECRET”

builder@builder-T100:~$ vi minio.credentials
builder@builder-T100:~$ cat minio.credentials
Oc3ZydYnG8fDWvWLPpTu:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
builder@builder-T100:~$ sudo chmod 600 minio.credentials

I can then add an entry in my fstab file

builder@builder-T100:~$ sudo vi /etc/fstab
builder@builder-T100:~$ cat /etc/fstab | tail -n1
databasebackups        /mnt/s3fs-dbbackups    fuse.s3fs       _netdev,allow_other,url=http://192.168.1.116:9000/,passwd_file=minio.credentials,use_path_request_style,uid=1000,gid=1000,umask=0001 0 0

And mount with sudo mount -a

Let’s create a test file

builder@builder-T100:~$ ls /mnt/s3fs-dbbackups/
builder@builder-T100:~$ echo "This is a test" | sudo tee /mnt/s3fs-dbbackups/testfile.txt
This is a test
builder@builder-T100:~$ ls /mnt/s3fs-dbbackups/
testfile.txt

Even though I do not see the object yet reflected in the Minio UI

I do see it if I hop over to a different host (my WSL) and list contents

builder@DESKTOP-QADGF36:~/Workspaces$ aws --profile miniosirnasilot --endpoint-url http://192.168.1.116:9000 s3 ls s3://databasebackups/
2024-09-07 07:25:05         15 testfile.txt

However, in a few minutes I did see the UI show we had an object

Minio Monitors

We can see some metrics in Monitoring

One useful feature is “Trace” that can identify what is being accessed in Minio

There are many more features such as bitrot scanners and compression settings

As my NAS has pretty good on-device compression, it would be redundant and I don’t want to burden docker with that.

However, I really want to get alerted if there is a problem. Let’s go to Uptime Kuma and add a check

I’ll monitor the Web UI port 9001 and set it to alert in all my ways

However, I had one minor issue recently - my Dockerhost seized and that meant Uptime Kuma stopped running. So fundamentally I’ll need something external to monitor.

Let’s add a Datadog Infrastructure agent to watch this host. I’ll add the APM instrumentation (though it is in Beta)

I can now launch that on my Dockerhost

builder@builder-T100:~$ DD_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxx DD_SITE="datadoghq.com" DD_APM_INSTRUMENTATION_ENABLED=host DD_APM_INSTRUMENTATION_LIBRARIES=java:1,python:2,js:5,dotnet:2,ruby:2 bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 74650  100 74650    0     0   223k      0 --:--:-- --:--:-- --:--:--  224k

* Datadog Agent 7 install script v1.34.0

/usr/bin/systemctl

* Installing apt-transport-https, curl and gnupg

Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Hit:2 https://packages.microsoft.com/repos/azure-cli focal InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:4 https://packages.cloud.google.com/apt gcsfuse-jammy InRelease
Get:5 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Hit:7 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:8 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Err:9 https://packages.cloud.google.com/apt kubernetes-xenial Release
  404  Not Found [IP: 142.250.190.46 443]
Reading package lists...
W: https://packages.cloud.google.com/apt/dists/gcsfuse-jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
E: The repository 'https://apt.kubernetes.io kubernetes-xenial Release' no longer has a Release file.
'apt-get update' failed, the script will not install the latest version of apt-transport-https.
Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.81.0-1ubuntu1.17).
gnupg is already the newest version (2.2.27-3ubuntu2.1).
The following packages were automatically installed and are no longer required:
  apg gnome-control-center-faces gnome-online-accounts libcolord-gtk1
  libfreerdp-server2-2 libgnome-bg-4-1 libgsound0 libgssdp-1.2-0
  libgupnp-1.2-1 libgupnp-av-1.0-3 libgupnp-dlna-2.0-4 libntfs-3g89
  librygel-core-2.6-2 librygel-db-2.6-2 librygel-renderer-2.6-2
  librygel-server-2.6-2 libvncserver1 mobile-broadband-provider-info
  network-manager-gnome python3-macaroonbakery python3-protobuf
  python3-pymacaroons python3-rfc3339 python3-tz rygel
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  apt-transport-https
1 upgraded, 0 newly installed, 0 to remove and 165 not upgraded.
Need to get 1,510 B of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 apt-transport-https all 2.4.12 [1,510 B]
Fetched 1,510 B in 0s (10.1 kB/s)
(Reading database ... 316323 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_2.4.12_all.deb ...
Unpacking apt-transport-https (2.4.12) over (2.4.11) ...
Setting up apt-transport-https (2.4.12) ...

... snip ...

No datadog.yaml file detected, not starting the agent
W: --force-yes is deprecated, use one of the options starting with --allow instead.

* Adding your API key to the Datadog Agent configuration: /etc/datadog-agent/datadog.yaml


* Setting SITE in the Datadog Agent configuration: /etc/datadog-agent/datadog.yaml

/usr/bin/systemctl
* Starting the Datadog Agent...

  Your Datadog Agent is running and functioning properly.
  It will continue to run in the background and submit metrics to Datadog.
  If you ever want to stop the Datadog Agent, run:

      sudo systemctl stop datadog-agent

  And to run it again run:

      sudo systemctl start datadog-agent

  Consider adding dd-agent to the docker group to enable the docker support, run:

      sudo usermod -a -G docker dd-agent

builder@builder-T100:~$

I can now see my host in Datadog

Including stats like traffic, disk, memory and CPU

I’ll next want to configure a host monitor

I set it to the Docker host and increased the time from the default 2 minutes to 10

For notifications, I’ll going to use Email and Pagerduty

I also plan to up the re-notification to every 12hours and stop after 5 occurrences.

Once saved, I can now see my Monitor is active and healthy

I noticed I didn’t see containers for my host

I then added:

container_image:
  enabled: true

To my datadog.yaml file and told it to restart

builder@builder-T100:~$ sudo vi /etc/datadog-agent/datadog.yaml
builder@builder-T100:~$ sudo systemctl restart datadog-agent
builder@builder-T100:~$

Even with the process_config enabled two ways

I never did see processes

Nor containers

OpenStatus

There is another way - and we can stay in the free tier.

I use OpenStatus to monitor my AWX instance today

In the free tier we can have really just one monitor and email

That said, my colleague Tristan could create a simple check for the Uptime endpoint

With the email being our PagerDuty

I can also create a status page

Which we can see here: fbs-uptime.openstatus.dev similar to FB Status.

So now I have UptimeKuma as a containerized monitor for all my systems and then OpenStatus and Datadog to monitor my Dockerhost in case that goes down. All this is in support of ensuring my containerized Minio is alive and well.

Synology Cloud Sync

I can install “Cloud Sync” as a package on the NAS itself

There are a lot of providers listed

I can see what permissions it needs, for instance, to use GCP Cloud Storage

Once I pick a project, I can see the GCP Buckets in the project

I can pick our Fuse test bucket

For instance, I can create a folder an initiate a sync directly from the NAS

Summary

We have covered so many ways to do local and remote backups at this point I should hope one of the patterns would work for most scenarios you may encounter.

Just focusing on GCP, for instance, we looked at Fuse, GSutil with Cron, the Storage Transfer Service and lastly the Cloud Sync app in Synology for On-prem backups to cloud

We also touched on monitors, from Uptime Kuma to OpenStatus to Datadog - with the idea that we can trigger alerts via Pagerduty.

We took a look at Minio finally, this time hosted via Docker on the NAS itself. We then could use the Cloud Sync app on Synology to sync to GCP. This is a wild way we could expose both on-prem storage (Volume in the local NAS) and backups to GCP while using AWS S3 compliant endpoints (Minio).

The reason we do this is sometimes there are solid tools out there from Longhorn storage to GlusterFS that easy can use S3 as backends