Published: Nov 7, 2023 by Isaac Johnson

Matrix is an open protocol for decentralized, secure communications. Basically, a federated method of chatting with rooms and channels - like a marriage of Mastodon and Slack.

It’s separated by a Matrix.org foundation which ensures open federation and a commitment to openness and interoperability. There is a lot on the About Page about the origins and Guardians.

For us, the question is, “Great, how do I install it? How can I use it?” If we look at the Servers page, we can see the ecosystem currently touts 5 primary servers:

Synapse (which we will try), written in Python with Apache 2.0 licensing
Construct, C++ with BSD
Conduit, Rust with Apache 2.0
Dendrite, Go with Apache 2.0
Telodendria, C with MIT licensing

As of this writing, there are 4 Featured clients touted on the site

Cinny (which we will use) - Windows, Mac, Linux and Web
Element X - iOS and Android
FluffyChat - iOS, Android, Linux and Web
Element - all the platforms incl MacOS

(There are a lot more clients than the 4 featured as well)

Installing Matrix Synapse server to test K3s

I’ll first use the demo cluster to see what we are launching with the helm chart.

First, we’ll add the Helm repo and update:

$ helm repo add ananace-charts https://ananace.gitlab.io/charts
"ananace-charts" already exists with the same configuration, skipping
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ananace-charts" chart repository
...Successfully got an update from the "backstage" chart repository
...Successfully got an update from the "frappe" chart repository
...Successfully got an update from the "deliveryhero" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

Now install

$ helm install matrix-synapse ananace-charts/matrix-synapse --set serverName=chosenin.space --set wellknown.enabled=true
NAME: matrix-synapse
LAST DEPLOYED: Wed Nov  1 18:15:25 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
** Note, this chart may take a while to finish setup, please be patient **
** Also, remember to disable the signingkey job (signingkey.job.enabled=false) **

Your Synapse install is now starting, you should soon be able to access it on
the following URL(s);
http://chosenin.space

You can create a user in your new Synapse install by running the following
command; (replacing USERNAME and PASSWORD)

    export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=matrix-synapse,app.kubernetes.io/instance=matrix-synapse,app.kubernetes.io/component=synapse" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec --namespace default $POD_NAME -- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u USERNAME -p PASSWORD --admin http://localhost:8008

You can also specify --no-admin to create a non-admin user.

Let’s try those commands to add a user

$ kubectl get pods --namespace default -l "app.kubernetes.io/name=matrix-synapse,app.kubernetes.io/instance=matrix-synapse,app.kubernetes.io/component=synapse" -o jsonpath="{.items[0].metadata.name}"
matrix-synapse-67db456599-5x2qt

I’ll make an admin and nonadmin user this way

builder@LuiGi17:~/Workspaces/jekyll-blog$ kubectl exec --namespace default m-- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u demouser -p demopass --admin  http://localhost:8008
Sending registration request...
Success!
builder@LuiGi17:~/Workspaces/jekyll-blog$ kubectl exec --namespace default matrix-synapse-67db456599-5x2qt -- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u demosimp -p demopass --no-admin http://localhost:8008
Sending registration request...
Success!

We can see what was created:

$ kubectl get pods
NAME                                                 READY   STATUS    RESTARTS        AGE
matrix-synapse-wellknown-lighttpd-68db77dcd7-75fqx   1/1     Running   0               9m
matrix-synapse-redis-master-75dbc4547-q4fzq          1/1     Running   0               9m
matrix-synapse-postgresql-0                          1/1     Running   0               9m
matrix-synapse-67db456599-5x2qt                      1/1     Running   3 (8m17s ago)   9m

$ kubectl get pvc
NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
matrix-synapse                     Bound     pvc-cfd8d2de-3e8c-45f4-880e-a4e094244853   10Gi       RWO            local-path     9m9s
data-matrix-synapse-postgresql-0   Bound     pvc-5b761de2-10ef-4e6e-9ab9-d75f1559665a   16Gi       RWO            local-path     9m9s

The problem I foresee is that the traefik ingress has a few prefixes. This is going to make it hard to test with just a simple port-forward like I might hope

$ kubectl get ingress matrix-synapse -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: matrix-synapse
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-11-01T23:16:43Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: matrix-synapse
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: matrix-synapse
    app.kubernetes.io/version: 1.95.1
    helm.sh/chart: matrix-synapse-3.7.10
  name: matrix-synapse
  namespace: default
  resourceVersion: "605397"
  uid: 84d67a90-8953-4828-b274-138e24ec2e93
spec:
  ingressClassName: traefik
  rules:
  - host: chosenin.space
    http:
      paths:
      - backend:
          service:
            name: matrix-synapse
            port:
              number: 8008
        path: /_matrix
        pathType: Prefix
      - backend:
          service:
            name: matrix-synapse
            port:
              number: 8008
        path: /_synapse
        pathType: Prefix
      - backend:
          service:
            name: matrix-synapse-wellknown-lighttpd
            port:
              number: 80
        path: /.well-known/matrix
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 192.168.1.13
    - ip: 192.168.1.159
    - ip: 192.168.1.206

So other than the PVCs being a bit heavy, I see no issue with moving on to a live test

Production

First, let’s add a DNS name

$ cat r53-matrix.json
{
    "Comment": "CREATE matrix fb.s A record ",
    "Changes": [
      {
        "Action": "CREATE",
        "ResourceRecordSet": {
          "Name": "matrix.freshbrewed.science",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [
            {
              "Value": "75.73.224.240"
            }
          ]
        }
      }
    ]
  }

$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-matrix.json
{
    "ChangeInfo": {
        "Id": "/change/C10031703U9L7OQPKY3WW",
        "Status": "PENDING",
        "SubmittedAt": "2023-11-01T23:10:58.348Z",
        "Comment": "CREATE matrix fb.s A record "
    }
}

I’ll now install it

$ helm install matrix-synapse ananace-charts/matrix-synapse --set serverName=matrix.freshbrewed.science --set wellknown.enabled=true
NAME: matrix-synapse
LAST DEPLOYED: Wed Nov  1 18:30:11 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
** Note, this chart may take a while to finish setup, please be patient **
** Also, remember to disable the signingkey job (signingkey.job.enabled=false) **

Your Synapse install is now starting, you should soon be able to access it on
the following URL(s);
http://matrix.freshbrewed.science

You can create a user in your new Synapse install by running the following
command; (replacing USERNAME and PASSWORD)

    export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=matrix-synapse,app.kubernetes.io/instance=matrix-synapse,app.kubernetes.io/component=synapse" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec --namespace default $POD_NAME -- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u USERNAME -p PASSWORD --admin http://localhost:8008

You can also specify --no-admin to create a non-admin user.

The ingress as created had no class specified, so it didn’t do too hot

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: matrix-synapse
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-11-01T23:30:42Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: matrix-synapse
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: matrix-synapse
    app.kubernetes.io/version: 1.95.1
    helm.sh/chart: matrix-synapse-3.7.10
  name: matrix-synapse
  namespace: default
  resourceVersion: "257389045"
  uid: d9109677-1eac-42d3-a7f9-5e82ff3d64bc
spec:
  rules:
  - host: matrix.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: matrix-synapse
            port:
              number: 8008
        path: /_matrix
        pathType: Prefix
      - backend:
          service:
            name: matrix-synapse
            port:
              number: 8008
        path: /_synapse
        pathType: Prefix
      - backend:
          service:
            name: matrix-synapse-wellknown-lighttpd
            port:
              number: 80
        path: /.well-known/matrix
        pathType: Prefix
status:
  loadBalancer: {}

I added the annotations myself

    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    meta.helm.sh/release-name: matrix-synapse
    meta.helm.sh/release-namespace: default
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

as well as the TLS secret. This created

$ kubectl get ingress matrix-synapse -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    field.cattle.io/publicEndpoints: '[{"addresses":["192.168.1.215","192.168.1.36","192.168.1.57","192.168.1.78"],"port":443,"protocol":"HTTPS","serviceName":"default:matrix-synapse-wellknown-lighttpd","ingressName":"default:matrix-synapse","hostname":"matrix.freshbrewed.science","path":"/.well-known/matrix","allNodes":false},{"addresses":["192.168.1.215","192.168.1.36","192.168.1.57","192.168.1.78"],"port":443,"protocol":"HTTPS","serviceName":"default:matrix-synapse","ingressName":"default:matrix-synapse","hostname":"matrix.freshbrewed.science","path":"/_matrix","allNodes":false},{"addresses":["192.168.1.215","192.168.1.36","192.168.1.57","192.168.1.78"],"port":443,"protocol":"HTTPS","serviceName":"default:matrix-synapse","ingressName":"default:matrix-synapse","hostname":"matrix.freshbrewed.science","path":"/_synapse","allNodes":false}]'
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    meta.helm.sh/release-name: matrix-synapse
    meta.helm.sh/release-namespace: default
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  creationTimestamp: "2023-11-01T23:30:42Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: matrix-synapse
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: matrix-synapse
    app.kubernetes.io/version: 1.95.1
    helm.sh/chart: matrix-synapse-3.7.10
  name: matrix-synapse
  namespace: default
  resourceVersion: "257391560"
  uid: d9109677-1eac-42d3-a7f9-5e82ff3d64bc
spec:
  rules:
  - host: matrix.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: matrix-synapse
            port:
              number: 8008
        path: /_matrix
        pathType: Prefix
      - backend:
          service:
            name: matrix-synapse
            port:
              number: 8008
        path: /_synapse
        pathType: Prefix
      - backend:
          service:
            name: matrix-synapse-wellknown-lighttpd
            port:
              number: 80
        path: /.well-known/matrix
        pathType: Prefix
  tls:
  - hosts:
    - matrix.freshbrewed.science
    secretName: matrix-tls
status:
  loadBalancer:
    ingress:
    - ip: 192.168.1.215
    - ip: 192.168.1.36
    - ip: 192.168.1.57
    - ip: 192.168.1.78

Make an admin use

builder@LuiGi17:~/Workspaces/jekyll-blog$ export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=matrix-synapse,app.kubernetes.io/instance=matrix-synapse,app.kubernetes.io/component=synapse" -o jsonpath="{.items[0].metadata.name}")

builder@LuiGi17:~/Workspaces/jekyll-blog$ kubectl exec --namespace default $POD_NAME -- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u adminuser -p MYADMINPASSWORD --admin http://localhost:8008
Sending registration request...
Success!

I can now login using something like Cinny. To keep it simple, I’ll just use Cinny web

And while it’s pretty empty right now, I can definitely see things

Here I’ll create a public and private room:

Users

Next, I’ll add a real user for myself. I never like staying in systems as “admin”; it’s like logging to a desktop as Administrator or root.

$ kubectl exec --namespace default $POD_NAME -- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u isaac -p MYPASSWORD  --admin http://localhost:8008
Sending registration request...
Success!

This time I’ll download Cinny for Windows from the downloads page

This experience pretty much matched the web version

I’ll then attempt to join the Public room I created ealier #MyPublicRoom:matrix.freshbrewed.science

I can now test chatting to myself

I decided to test to see if self-registration was enabled by default

which showed it clearly wasnt

This is due to “enable_registration: false” set in homeserver.yaml

I can edit the CM

$ kubectl edit cm matrix-synapse -o yaml

You can see the change:

$ kubectl get cm matrix-synapse -o yaml > before
$ kubectl edit cm matrix-synapse -o yaml
$ kubectl get cm matrix-synapse -o yaml > after
$ diff before after
16c16
<     ##\n\nenable_registration: false\n\n## Metrics ###\n\nenable_metrics: true\n\n##
---
>     ##\n\nenable_registration: true\n\n## Metrics ###\n\nenable_metrics: true\n\n##
55c55
<   resourceVersion: "257388993"
---
>   resourceVersion: "257411145"

Cinny actually crashed the pod over and over after I did this… clearly it does not want open registration without the admin being sure

matrix-synapse-7687454496-wvlxc                          0/1     CrashLoopBackOff   3 (37s ago)       101s



$ kubectl logs matrix-synapse-7687454496-wvlxc
This server is configured to use 'matrix.org' as its trusted key server via the
'trusted_key_servers' config option. 'matrix.org' is a good choice for a key
server since it is long-lived, stable and trusted. However, some admins may
wish to use another server for this purpose.

To suppress this warning and continue using 'matrix.org', admins should set
'suppress_key_server_warning' to 'true' in homeserver.yaml.
--------------------------------------------------------------------------------
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 399, in <module>
    main()
  File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 389, in main
    hs = setup(sys.argv[1:])
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 335, in setup
    raise ConfigError(
synapse.config._base.ConfigError: You have enabled open registration without any verification. This is a known vector for spam and abuse. If you would like to allow public registration, please consider adding email, captcha, or token-based verification. Otherwise this check can be removed by setting the `enable_registration_without_verification` config option to `true`.

My next step is to try federation.

I’ll create an account on one of the main Matrix hosts

Since isaac is taken, I’ll use my primary domain

Now I’m in

And now to test joining the room over on my server

We can see verification I can join and participate in the room

API

We can use the API to post to rooms if we want.

I need to get an API token

$ curl -XPOST -d '{"type":"m.login.password", "user":"adminuser", "password":"xxxxxxxxxxxxx"}' "https://matrix.freshbrewed.science/_matrix/client/r0/login"
{"user_id":"@adminuser:matrix.freshbrewed.science","access_token":"syt_xxxxxxxxxx_xxxxxxxxxxxxxxxx","home_server":"matrix.freshbrewed.science","device_id":"TUDBHPSNKP","well_known":{"m.homeserver":{"base_url":"https://matrix.freshbrewed.science/"}}}

I can now use that to send a message

$ curl 'https://matrix.freshbrewed.science/_matrix/client/r0/rooms/!mCZzGIXZwARenABwkM:matrix.freshbrewed.science/send/m.room.message/?access_token=syt_xxxxxxxxxx_xxxxxxxxxxxxxxxx' -X PUT --data '{"msgtype":"m.text","body":"hello world"}'
{"event_id":"$W_WYtljvVthD1gT3KDMvuGzV8C7hcxfRvdTjVxuT9gw"}

Using with notification systems

I’ll start by adding a Build Notifications room

I want to get the ID for the room (!tFUMciqVMNLKtEKfUh:matrix.freshbrewed.science)

Earlier I experimented with sending a notice via a CURL post and a token. I don’t really want to impersonate myself each time. Instead, I can create a “Pipeline” service account like I had for other local users.

$ kubectl exec --namespace default `kubectl get pods --namespace default -l "app.kubernetes.io/name=matrix-synapse,app.kubernetes.io/instance=matrix-synapse,app.kubernetes.io/component=synapse" -o jsonpath="{.items[0].metadata.name}"` -- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u builder -p Redliub\$1  --no-admin http://localhost:8008
Sending registration request...
Success!

I’ll use a different web browser to test the login

I’ll give it a more proper icon

With icon created, I’ll fetch an API token I can use to post

$ curl -XPOST -d '{"type":"m.login.password", "user":"builder", "password":"xxxxxxxxxxxxxx"}' "https://matrix.freshbrewed.science/_matrix/client/r0/login"
{"user_id":"@builder:matrix.freshbrewed.science","access_token":"syt_XXXXXXXX_XXXXXXXXXXXXXXX","home_server":"matrix.freshbrewed.science","device_id":"FMOHGQPUKB","well_known":{"m.homeserver":{"base_url":"https://matrix.freshbrewed.science/"}}}

Azure DevOps

I’ll now add as a variable to an Azure Pipeline.

I first tried to invite via chat

that resolved to

But nothing happened. I then went to details of the room and chose to invite from there

Now in my “builder” user window, I see a mail notification with an invite

and I can now see the room

I can now test a curl

curl 'https://matrix.freshbrewed.science/_matrix/client/r0/rooms/!tFUMciqVMNLKtEKfUh:matrix.freshbrewed.science/send/m.room.message/?access_token=syt_xxxxxxxxxxxxxx_xxxxx' -X PUT --data '{"msgtype":"m.text","body":"hello world"}'

and I can see it in Cinny as well

I’ll now change that Azure DevOps job to call a notice with success or failed

stages:
- stage: RunAsBinary
  displayName: Running JFRunner binary
  jobs:
  - job: JFRunner
    steps:
    - script: |
        wget https://github.com/jenkinsci/jenkinsfile-runner/releases/download/1.0-beta-29/jenkinsfile-runner-1.0-beta-29.zip
        unzip jenkinsfile-runner-1.0-beta-29.zip
        chmod u+x ./bin/jenkinsfile-runner
      displayName: 'get JFR binary'

    - script: |
        ./bin/jenkinsfile-runner -p ./helloworld/plugins.txt -f ./helloworld/jenkinsfile
      displayName: 'Run JFRunner'
      
    - script: |
        curl 'https://matrix.freshbrewed.science/_matrix/client/r0/rooms/!tFUMciqVMNLKtEKfUh:matrix.freshbrewed.science/send/m.room.message/?access_token=$(BUILDERMATRIXTOKEN)' -X PUT --data '{"msgtype":"m.text","body":"JenkinsBuild $(Build.BuildID) Completed"}'
      displayName: 'Notify Matrix'
      condition: succeeded()

    - script: |
        curl 'https://matrix.freshbrewed.science/_matrix/client/r0/rooms/!tFUMciqVMNLKtEKfUh:matrix.freshbrewed.science/send/m.room.message/?access_token=$(BUILDERMATRIXTOKEN)' -X PUT --data '{"msgtype":"m.text","body":"JenkinsBuild $(Build.BuildID) Failed"}'
      displayName: 'Notify Matrix'
      condition: failed()

My tests failed. It was then I realized these tokens (by default) don’t last too long.

I’ll change to using a user/pass and fetching a token on the fly

    - script: |
        set -x
        curl -XPOST -d '{"type":"m.login.password", "user":"builder", "password":"$(BUILDERPASS)"}' "https://matrix.freshbrewed.science/_matrix/client/r0/login" | jq -r '.access_token' | tr -d '\n' > token
        export MYTOKEN=`cat ./token`
        curl "https://matrix.freshbrewed.science/_matrix/client/r0/rooms/"'!'"tFUMciqVMNLKtEKfUh:matrix.freshbrewed.science/send/m.room.message/?access_token=$MYTOKEN" -X PUT --data '{"msgtype":"m.text","body":"JenkinsBuild $(Build.BuildID) Completed"}'
      displayName: 'Notify Matrix'
      condition: succeeded()

    - script: |
        set -x
        curl -XPOST -d '{"type":"m.login.password", "user":"builder", "password":"$(BUILDERPASS)"}' "https://matrix.freshbrewed.science/_matrix/client/r0/login" | jq -r '.access_token' | tr -d '\n' > token
        export MYTOKEN=`cat ./token`
        curl "https://matrix.freshbrewed.science/_matrix/client/r0/rooms/"'!'"tFUMciqVMNLKtEKfUh:matrix.freshbrewed.science/send/m.room.message/?access_token=$MYTOKEN" -X PUT --data '{"msgtype":"m.text","body":"JenkinsBuild $(Build.BuildID) Failed"}'
      displayName: 'Notify Matrix'
      condition: failed()

This worked great

Github Actions

I’ll set a password in a Github Secret

I added steps in TEST, Main and Failed

    - name: Update Matrix Channel
      run: |
          set -x
          curl -XPOST -d '{"type":"m.login.password", "user":"builder", "password":"$"}' "https://matrix.freshbrewed.science/_matrix/client/r0/login" | jq -r '.access_token' | tr -d '\n' > token
          export MYTOKEN=`cat ./token`
          curl "https://matrix.freshbrewed.science/_matrix/client/r0/rooms/"'!'"tFUMciqVMNLKtEKfUh:matrix.freshbrewed.science/send/m.room.message/?access_token=$MYTOKEN" -X PUT --data '{"msgtype":"m.text","body":"Jekyll Blog - Finished TEST $"}'
      env:
        GHTOKEN: $

The build kicked off for my current PR (build 692)

I could see the notice in Cinny

Summary

We created a Matrix Synapse server in a test Kubernetes cluster and quickly moved on to prod. There I setup TLS ingress via Nginx and worked out some kinks in the Helm chart. I then created several accounts, some local and some remote to test rooms (public and private) and how to join. We looked into curl commands to get API tokens and post to rooms. Lastly, we setup several build notifications with Azure DevOps and Github Actions.