Fresh/Brewed

YOURLS in production

Published: Nov 14, 2024 by Isaac Johnson

Ealier this week we looked at three link shorteners and trackers; YOURLS, shlink and Reduced.to. We narrowed it to YOURLS and shlink as our go forward.

YOURLS

Despite the great CLI and REST backend of shlink, I think I’ll try going forward with YOURLS.

The first test was to see if I could POST to the API

$ curl -X POST -H "Content-Type: application/json" --user "user:asdfasdf" https://yourls.tpk.pw/yourls-api.php -d '{"url":"https://freshbrewed.science/2024/11/12/urlshortner.html","keyword":"shorten","title":"Blog: Open-source URL shorteners","format":"json","action":"shorturl","username":"user","password":"asdfasdf"}'
<?xml version="1.0" encoding="UTF-8"?>
<root><message>Please log in</message><errorCode>403</errorCode><callback></callback></root>

$ curl -X POST -H "Content-Type: application/json" https://yourls.tpk.pw/yourls-api.php -d '{"url":"https://freshbrewed.science/2024/11/12/urlshortner.html","keyword":"shorten","title":"Blog: Open-source URL shorteners","format":"json","action":"shorturl","username":"user","password":"asdfasdf","signature":"asdfasdfasdf"}'
<?xml version="1.0" encoding="UTF-8"?>
<root><message>Please log in</message><errorCode>403</errorCode><callback></callback></root>

Despite finding old code suggesting that would work, it clearly didnt. I saw in the latest docs they use GET with a signature you can get from your admin page:

/content/images/2024/11/yourls2-01.png

I tried that (still including my user pass for the test)

$ curl -G \
ta-urlencode "ur>   --data-urlencode "url=https://freshbrewed.science/2024/11/12/urlshortner.html" \
>   --data-urlencode "keyword=shorten" \
>   --data-urlencode "title=Blog: Open-source URL shorteners" \
>   --data-urlencode "format=json" \
>   --data-urlencode "action=shorturl" \
>   --data-urlencode "username=user" \
>   --data-urlencode "password=asdfasdfasfd" \
>   --data-urlencode "signature=asdfasdfasdf" \
>   https://yourls.tpk.pw/yourls-api.php
{"status":"success","code":"","message":"https:\/\/freshbrewed.science\/2024\/11\/12\/urlshortner.html added to database","errorCode":"","statusCode":200,"url":{"keyword":"shorten","url":"https:\/\/freshbrewed.science\/2024\/11\/12\/urlshortner.html","title":"Blog: Open-source URL shorteners","date":"2024-11-13 12:26:16","ip":"192.168.1.1"},"title":"Blog: Open-source URL shorteners","shorturl":"https:\/\/yourls.tpk.pw\/shorten"}

That clearly works, with the new links showing up in the interface

/content/images/2024/11/yourls2-02.png

Production install

Let’s use an even shorter URL for this final prod setup.

$ az account set --subscription "Pay-As-You-Go" && az network dns record-set a add-record -g idjdnsrg -z tpk.pw -a 75.73.224.240 -n go
{
  "ARecords": [
    {
      "ipv4Address": "75.73.224.240"
    }
  ],
  "TTL": 3600,
  "etag": "07f9737a-922a-4ade-8c70-aada66aab373",
  "fqdn": "go.tpk.pw.",
  "id": "/subscriptions/d955c0ba-13dc-44cf-a29a-8fed74cbb22d/resourceGroups/idjdnsrg/providers/Microsoft.Network/dnszones/tpk.pw/A/go",
  "name": "go",
  "provisioningState": "Succeeded",
  "resourceGroup": "idjdnsrg",
  "targetResource": {},
  "trafficManagementProfile": {},
  "type": "Microsoft.Network/dnszones/A"
}

Before I move on, I need to create a MariaDB (MySQL) database and user we can use for deploy

ijohnson@SassyNassy:~$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4197377
Server version: 10.3.32-MariaDB Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database yourls;
Query OK, 1 row affected (0.046 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON yourls.* TO 'yourls'@'%' IDENTIFIED BY 'YourLs#Pass333444';
Query OK, 0 rows affected (0.463 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.089 sec)

We can now launch the instance with helm using the database values 

$ helm install yourls --create-namespace -n yourls --set mariadb.enabled=false,externalDatabase.host=192.168.1.129,externalDatabase.user=yourls,externalDatabase.password=asdfasdf,externalDatabase.database=yourls,yourls.domain=go.tpk.pw,yourls.username=builder yourls/yourls
NAME: yourls
LAST DEPLOYED: Wed Nov 13 06:48:19 2024
NAMESPACE: yourls
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: yourls
CHART VERSION: 6.1.14
APP VERSION: 1.9.2

** Please be patient while the chart is being deployed **

Your YOURLS site can be accessed through the following DNS name from within your cluster:

    yourls.yourls.svc.cluster.local (port 80)

To access your YOURLS site from outside the cluster follow the steps below:

1. Get the YOURLS URL by running these commands:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace yourls -w yourls'

    export SERVICE_IP=$(kubectl get svc --namespace yourls yourls --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
    echo "YOURLS URL: http://$SERVICE_IP/"
    echo "YOURLS Admin URL: http://$SERVICE_IP/admin"

2. Open a browser and access YOURLS using the obtained URL.

3. Login with the following credentials below to see your app:

    echo Username: $(kubectl get secret --namespace yourls yourls -o jsonpath="{.data.username}" | base64 --decode)
    echo Password: $(kubectl get secret --namespace yourls yourls -o jsonpath="{.data.password}" | base64 --decode)

We can see it launched, but is awaiting our admin install

builder@DESKTOP-QADGF36:~$ kubectl get pods -n yourls
NAME                      READY   STATUS    RESTARTS   AGE
yourls-7cf8df7659-gkrv4   0/1     Running   0          61s
builder@DESKTOP-QADGF36:~$ kubectl logs yourls-7cf8df7659-gkrv4 -n yourls
YOURLS not found in /var/www/html - copying now...
Complete! YOURLS has been successfully copied to /var/www/html
Running custom script /docker-entrypoint-init.d/*
... ignoring /docker-entrypoint-init.d/*
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.42.2.183. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.42.2.183. Set the 'ServerName' directive globally to suppress this message
[Wed Nov 13 12:48:44.829469 2024] [mpm_prefork:notice] [pid 1:tid 1] AH00163: Apache/2.4.62 (Debian) PHP/8.3.13 configured -- resuming normal operations
[Wed Nov 13 12:48:44.829663 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
10.42.2.1 - - [13/Nov/2024:12:49:19 +0000] "GET /admin/install.php HTTP/1.1" 503 3134 "-" "kube-probe/1.26"
10.42.2.1 - - [13/Nov/2024:12:49:29 +0000] "GET /admin/install.php HTTP/1.1" 503 3134 "-" "kube-probe/1.26"

I tried to port-forward

$ kubectl port-forward yourls-7cf8df7659-gkrv4 -n yourls 8084:80
Forwarding from 127.0.0.1:8084 -> 80
Forwarding from [::1]:8084 -> 80
Handling connection for 8084
Handling connection for 8084

But the install says we cannot connect to the DB

/content/images/2024/11/yourls2-03.png

It dawned on me that on this NAS, I have the port off from default (using 3307 instead of the standard MySQL 3306).

I updated with a helm update adding the port to the values

$ helm upgrade yourls --create-namespace -n yourls --set mariadb.enabled=false,externalDatabase.host=192.168.1.129,externalDatabase.user=yourls,externalDatabase.password=asdfasdfasdfs,externalDatabase.database=yourls,externalDatabase.port=3307,yourls.domain
=go.tpk.pw,yourls.username=builder yourls/yourls
...

I tested locally

$ mysql --host=192.168.1.129 --user=yourls --password=YourLs#Pass333444 --port=3307 yourls
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4197913
Server version: 5.5.5-10.3.32-MariaDB Source distribution

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> \dt;
mysql>

I’m going to try MariaDB 10 on my other NAS which runs on port 3306 (the standard). Perhaps it isn’t respecting the passed in port.

I tested my connection locally after creating the DB and user

builder@DESKTOP-QADGF36:~$ mysql --host=192.168.1.116 --user=yourls --password=asdfasdfasdf yourls
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 62
Server version: 5.5.5-10.11.6-MariaDB Source distribution

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> \q
Bye

Then a helm delete and install

$ helm delete yourls -n yourls
release "yourls" uninstalled

$ helm install yourls --create-namespace -n yourls --set mariadb.enabled=false,externalDatabase.host=192.168.1.116,externalDatabase.user=yourls,externalDatabase.password=MyYourls_334488,externalDatabase.database=yourls,yourls.domain=go.tpk.pw,yourls.username=builder yourls/yourls
NAME: yourls
LAST DEPLOYED: Wed Nov 13 07:22:42 2024
NAMESPACE: yourls
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: yourls
CHART VERSION: 6.1.14
APP VERSION: 1.9.2

** Please be patient while the chart is being deployed **

Your YOURLS site can be accessed through the following DNS name from within your cluster:

    yourls.yourls.svc.cluster.local (port 80)

To access your YOURLS site from outside the cluster follow the steps below:

1. Get the YOURLS URL by running these commands:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace yourls -w yourls'

    export SERVICE_IP=$(kubectl get svc --namespace yourls yourls --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
    echo "YOURLS URL: http://$SERVICE_IP/"
    echo "YOURLS Admin URL: http://$SERVICE_IP/admin"

2. Open a browser and access YOURLS using the obtained URL.

3. Login with the following credentials below to see your app:

    echo Username: $(kubectl get secret --namespace yourls yourls -o jsonpath="{.data.username}" | base64 --decode)
    echo Password: $(kubectl get secret --namespace yourls yourls -o jsonpath="{.data.password}" | base64 --decode)

This time it seems to be working

$ kubectl port-forward yourls-866ffb8cf9-wsrf6 -n yourls 8084:80
Forwarding from 127.0.0.1:8084 -> 80
Forwarding from [::1]:8084 -> 80
Handling connection for 8084
Handling connection for 8084
Handling connection for 8084

/content/images/2024/11/yourls2-04.png

And I see it complete

/content/images/2024/11/yourls2-05.png

I can go ahead and make the ingress now that we’ve sorted out our DB connectivity issues.

Actually, im noticing it tried to make a LoadBalancer type of ingress. I’de much rather keep it as a ClusterIP as we plan to route our ingress there.

$ kubectl get svc -n yourls
NAME     TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
yourls   LoadBalancer   10.43.64.38   <pending>     80:31833/TCP,443:30527/TCP   4m15s

I should be able to do a helm upgrade to fix that with service.type=ClusterIP

$ helm upgrade yourls --create-namespace -n yourls --set mariadb.enabled=false,externalDatabase.host=192.168.1.116,externalDatabase.user=yourls,externalDatabase.password=asdfsadfasdf,externalDatabase.database=yourls,yourls.domain=go.tpk.pw,yourls.username=builder,service.type=ClusterIP yourls/yourls
Release "yourls" has been upgraded. Happy Helming!
NAME: yourls
LAST DEPLOYED: Wed Nov 13 07:29:14 2024
NAMESPACE: yourls
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
CHART NAME: yourls
CHART VERSION: 6.1.14
APP VERSION: 1.9.2
... snip ...

That looks better

$ kubectl get svc -n yourls
NAME     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
yourls   ClusterIP   10.43.64.38   <none>        80/TCP,443/TCP   7m5s

Let’s make that Ingress now

$ cat ./ingress.yourlsnew.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: azuredns-tpkpw
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.org/websocket-services: yourls
  name: yourlsingress
spec:
  rules:
  - host: go.tpk.pw
    http:
      paths:
      - backend:
          service:
            name: yourls
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - go.tpk.pw
    secretName: mygo-tls

$ kubectl apply -f ./ingress.yourlsnew.yaml -n yourls
ingress.networking.k8s.io/yourlsingress created

Which started the cert process

$ kubectl get cert -n yourls
NAME       READY   SECRET     AGE
mygo-tls   False   mygo-tls   32s

$ kubectl get ingress -n yourls
NAME            CLASS    HOSTS       ADDRESS   PORTS     AGE
yourlsingress   <none>   go.tpk.pw             80, 443   40s

Once the cert was validated

$ kubectl get certificate -n yourls
NAME       READY   SECRET     AGE
mygo-tls   True    mygo-tls   49s

I could login

/content/images/2024/11/yourls2-06.png

I can now see the basic setup

/content/images/2024/11/yourls2-07.png

Using in Github Actions

My next thought was about modifying my flow to put in a link for Socials in my Mastodon step (I will not use Twitter for obvious reasons).

      - name: Post to Mastodon
        run: | 
            export LATESTFILE=`ls -l _posts/ | grep ".* 20[0-9][0-9]-[0-9][0-9].*markdown" | tail -n1 | sed 's/.* \(20[0-9][0-9]-[0-9][0-9].*markdown\)/\1/'`
            export LATESTURL=`ls -l _posts/ | grep ".* 20[0-9][0-9]-[0-9][0-9].*markdown" | tail -n1 | sed 's/.* \(20[0-9][0-9]-[0-9][0-9]-[0-9][0-9]-\)\(.*\)\.markdown/\2.html/'`
            echo '{"status":"' | tr -d '\n' > payload.json
            cat _posts/$LATESTFILE | grep "^social: " | head -n 1 | sed 's/^social: "\(.*\)"/\1/' | tr -d '\n'>> payload.json
            cat _posts/$LATESTFILE | grep "^date: " | head -n 1 | sed "s/^date: .\([0-9]*\)-\([0-9]*\)-\([0-9]*\) .*/ https:\/\/freshbrewed.science\/\1\/\2\/\3\/$LATESTURL/g" | sed 's/.markdown/.html/' | tr -d '\n' >> payload.json
            echo '"}' >> payload.json

            cat payload.json | base64

            # allow a way to not repost on image updates and hotfixes
            log=$(git log -n 1)

            if [[ $log != *"SKIPSOCIAL"* ]]; then
              echo "CHECK-OK: posting to social"
              curl -X POST -H "Authorization: Bearer $MASTODONAPI" -H 'Content-Type: application/json' -d @payload.json https://noc.social/api/v1/statuses
            else 
              echo "CHECK-SKIP: Skip Posting To Social.. would have posted:"
              cat payload.json
            fi
        env:
          MASTODONAPI: $
          GHTOKEN: $

That Payload basically translates to a slug for noc.social, e.g.

{"status":"I needed to find a decent URL shortener and optionally link tracker and realized there are quite a few good Open-Source options for self-hosting.  I the blog writeup i look at YOURLS, SHLink and reduced.to.  I'll set all three up in Docker and Kubernetes and compare and contrast their features and usage.  I'll also touch on the SaaS offering from reduced.to https://freshbrewed.science/2024/11/12/urlshortner.html"}

I found myself needing to work out the syntax for the bash locally so i can up with a quick test file

#!/bin/bash
set -x

# clean
rm ./title.txt || true
rm ./payload.json || true
rm ./target.url || true
rm ./yourls.output.json || true

export LATESTFILE=`ls -l _posts/ | grep ".* 20[0-9][0-9]-[0-9][0-9].*markdown" | tail -n1 | sed 's/.* \(20[0-9][0-9]-[0-9][0-9].*markdown\)/\1/'`
export LATESTURL=`ls -l _posts/ | grep ".* 20[0-9][0-9]-[0-9][0-9].*markdown" | tail -n1 | sed 's/.* \(20[0-9][0-9]-[0-9][0-9]-[0-9][0-9]-\)\(.*\)\.markdown/\2.html/'`
echo '{"status":"' | tr -d '\n' > payload.json
cat _posts/$LATESTFILE | grep "^title: " | head -n 1 | sed 's/^title: "\(.*\)"/\1/' | tr -d '\n'> title.txt
cat _posts/$LATESTFILE | grep "^social: " | head -n 1 | sed 's/^social: "\(.*\)"/\1/' | tr -d '\n'>> payload.json
cat _posts/$LATESTFILE | grep "^date: " | head -n 1 | sed "s/^date: .\([0-9]*\)-\([0-9]*\)-\([0-9]*\) .*/https:\/\/freshbrewed.science\/\1\/\2\/\3\/$LATESTURL/g" | sed 's/.markdown/.html/' | tr -d '\n' > target.url

curl --data-urlencode "url=`cat target.url`" --data-urlencode "title=`cat title.txt`" --data-urlencode 'format=json' --data-urlencode 'action=shorturl' --data-urlencode 'signature=asdfasdf' "https://go.tpk.pw/yourls-api.php" | tee yourls.output.json

echo " " | tr -d '\n' >> payload.json
cat yourls.output.json | jq -r '.shorturl' | tr -d '\n' >> payload.json
echo '"}' >> payload.json

This gave a nice little slug (and would get a 400 back from an existing link but still deliver the same YOURLS payload)

$ cat yourls.output.json
{"status":"fail","code":"error:url","message":"https:\/\/freshbrewed.science\/2024\/11\/14\/links.html already exists in database (short URL: go.tpk.pw\/1)","errorCode":"400","statusCode":"400","url":{"keyword":"1","url":"https:\/\/freshbrewed.science\/2024\/11\/14\/links.html","title":"YOURLS in production","date":"2024-11-14 01:08:56","ip":"172.56.13.36","clicks":1},"title":"YOURLS in production","shorturl":"https:\/\/go.tpk.pw\/1"}

$ cat payload.json
{"status":"As I mentioned in the last post, I would move either shlink to yourls into production including a shorter url and real HA database backend.  In the post, I do that as well as tie into Gthuba actions. https://go.tpk.pw/1"}

The redirect works, but obviously at the time of this writing the file doesnt exist just yet.

I added to the github action, but I’ll have to wait to test when this is published.

Plugins

Let’s assume we want random strings in the URLs instead of just a number that iterates. That’s actually a pre-installed plugin we can use OOTB.

We just need to enable it

/content/images/2024/11/yourls2-09.png

Now if I delete the current link

/content/images/2024/11/yourls2-10.png

and try my test script again, I get a random short string for the URL

{"status":"As I mentioned in the last post, I would move either shlink to yourls into production including a shorter url and real HA database backend.  In the post, I do that as well as tie into Gthuba actions. https://go.tpk.pw/9g375 "}

Summary

This was not a tremendously long post but not every one needs to be. Our goal here was to take a URL shortener, in our case YOURLS and deploy it to a production cluster. We wanted to use an HA MariaDB backend which I did on my newer NAS. I then wanted to create a nicer short URL (go.tpk.pw) and lastly tie it in to my CICD, which is Github Actions.

If all goes well, this post should be the first to use the shortener. One consequence is that if my YOURLS ever junks out and people read my old posts, they won’t have direct links. I, of course, did all I could to ensure this service is restorable and the database is backed up on durable storage with dedicated battery backups (which they are). In fact the NASes (only them) have the USB sensors to power down on low battery from the APCs to make them even safer.

OpenSource Links shorteners yourls github

Have something to add? Feedback? You can use the feedback form

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes