Published: Jul 12, 2022 by Isaac Johnson

On multiple occasions I have had the need to setup up Ansible Tower, or it’s Open Source cousin AWX, as part of cloud migrations or to replace dated setups on-prem.

It is most often installed directly or as a docker compose based install on a linux host. However, we can install it using a Helm chart instead for a robust Kubernetes driven implementation.

Installing

There are quite a few charts we can use. A simple Google search found the following:

Datatree: https://www.datree.io/helm-chart/awx-lifen
Adwerx: https://artifacthub.io/packages/helm/adwerx/awx
ArtifactHub / Arthur-C: https://artifacthub.io/packages/helm/lifen/awx
Rfyio: https://github.com/arthur-c/ansible-awx-helm-chart/blob/master/README.md
Novum: https://novumrgi.github.io/helm/charts/awx/

I opted to use the Adwerx chart which will install AWX 17.1.0 with Ansible 2.9.18. We’ll look at two more later in this article.

We just need to add the repo and install with helm

$ helm repo add adwerx https://adwerx.github.io/charts
"adwerx" has been added to your repositories


$ helm install adwerxawx -n adwerx --create-namespace adwerx/awx --set defaultAdminUser=admin --set defaultAdminPassword=MyPassword --set
 postgresql.postgresqlPassword=password --set secretKey=myawxsecret
NAME: adwerxawx
LAST DEPLOYED: Sat Jul  9 21:48:48 2022
NAMESPACE: adwerx
STATUS: deployed
REVISION: 1

We can then port-forward to the service

$ kubectl port-forward svc/adwerxawx -n adwerx 8080:8080
Forwarding from 127.0.0.1:8080 -> 8052
Forwarding from [::1]:8080 -> 8052

Our first step is to create an Organization

Now that we created “HomeLab”

We can add an Inventory. Inventories are what hold Hosts

In order to connect, we’ll need to add a credential.

We have a lot of options. Perhaps the simplest is a simple SSH username/password.

Choose “Machine” for the type

Then add your username and password.

To test, we need a playbook. There are a few Test Ansible repos out there. I opted for https://github.com/ansible/test-playbooks

We can then add a Template from the Project in Templates

If you are able to connect to the GIT Repo you will see the list of Playbooks on the right

Once added, we can click Launch to test the Template

which turns into a Run

Lastly, if we can connect to the host and execute the playbook, we should see output in the details of the run.

Testing Resiliency

Let’s rotate the pods like might happen in a scaling or update event

$ kubectl get pods -n adwerx
NAME                         READY   STATUS    RESTARTS   AGE
adwerxawx-postgresql-0       1/1     Running   0          22h
adwerxawx-68889fdd67-c7dsc   3/3     Running   0          22h
builder@DESKTOP-72D2D9T:~$ kubectl delete pod adwerxawx-postgresql-0 -n adwerx & kubectl delete pod adwerxawx-68889fdd67-c7dsc -n adwerx &
[1] 12899
[2] 12900

We can see the pods terminating

$ kubectl get pods -n adwerx
NAME                         READY   STATUS              RESTARTS   AGE
adwerxawx-postgresql-0       1/1     Terminating         0          22h
adwerxawx-68889fdd67-c7dsc   3/3     Terminating         0          22h
adwerxawx-68889fdd67-lvg5d   0/3     ContainerCreating   0          3s

$ kubectl get pods -n adwerx
NAME                         READY   STATUS    RESTARTS   AGE
adwerxawx-postgresql-0       1/1     Running   0          41s
adwerxawx-68889fdd67-lvg5d   3/3     Running   0          72s

Then to check it all still works, I’ll port-forward back

$ kubectl port-forward svc/adwerxawx -n adwerx 8080:8080
Forwarding from 127.0.0.1:8080 -> 8052
Forwarding from [::1]:8080 -> 8052
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080

I can login

And we can see all our history and details are there

Ingress

First, I need to create Route53

$ cat r53-awx.json
{
  "Comment": "CREATE awx fb.s A record ",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "awx.freshbrewed.science",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "73.242.50.46"
          }
        ]
      }
    }
  ]
}

$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-awx.json
{
    "ChangeInfo": {
        "Id": "/change/C0046530W9QFMQ2ULX7J",
        "Status": "PENDING",
        "SubmittedAt": "2022-07-11T02:15:06.921Z",
        "Comment": "CREATE awx fb.s A record "
    }
}

We can then add an Ingress

$ cat awx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 32k
    nginx.ingress.kubernetes.io/proxy-buffers-number: 8 32k
    nginx.ingress.kubernetes.io/proxy-read-timeout: "43200"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "43200"
  generation: 1
  labels:
    app: awx
  name: awx-ingress
  namespace: adwerx
spec:
  ingressClassName: nginx
  rules:
  - host: awx.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: adwerxawx
            port:
              number: 8080
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - awx.freshbrewed.science
    secretName: tls-awx

$ kubectl apply -f awx-ingress.yaml -n adwerx
ingress.networking.k8s.io/awx-ingress created

As soon as the cert is satisfied, we can reach the AWX externally without port-forward

builder@DESKTOP-72D2D9T:~/Workspaces/jekyll-blog$ kubectl get cert -n adwerx
NAME      READY   SECRET    AGE
tls-awx   False   tls-awx   103s
builder@DESKTOP-72D2D9T:~/Workspaces/jekyll-blog$ kubectl get cert -n adwerx
NAME      READY   SECRET    AGE
tls-awx   True    tls-awx   117s

Other Charts: Lifen / Datree AWX

I thought it wise to try some of the other charts I found.

The first was from Datree.io which is a Kubernetes configuration checking tool provider.

$ helm repo add lifen-charts http://honestica.github.io/lifen-charts/
"lifen-charts" has been added to your repositories
builder@DESKTOP-72D2D9T:~/Workspaces/jekyll-blog$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "lifen-charts" chart repository
...Successfully got an update from the "cribl" chart repository
...Successfully got an update from the "metallb" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "adwerx" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "jenkins" chart repository
...Successfully got an update from the "argo-cd" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "bitnami" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm install -n datreeawx --create-namespace myawx lifen-charts/awx --version 10.0.3 --set default_admin_password=MyPassword
W0710 21:53:06.324770    1867 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0710 21:53:07.116774    1867 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
NAME: myawx
LAST DEPLOYED: Sun Jul 10 21:53:04 2022
NAMESPACE: datreeawx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:

I then checked the pods

$ kubectl get pods -n datreeawx
NAME                          READY   STATUS              RESTARTS   AGE
myawx-55dc6b97c-7xjqr         0/3     ContainerCreating   0          29s
myawx-memcached-0             0/1     ContainerCreating   0          29s
myawx-postgresql-0            0/1     ContainerCreating   0          28s
myawx-rabbitmq-0              0/1     Running             0          28s
myawx-web-88c9d89f4-kt7td     0/1     ImagePullBackOff    0          29s
myawx-task-78c64c6965-lrs2l   0/1     ImagePullBackOff    0          29s


$ kubectl get pods -n datreeawx
NAME                          READY   STATUS             RESTARTS   AGE
myawx-rabbitmq-0              1/1     Running            0          2m36s
myawx-web-88c9d89f4-kt7td     0/1     ImagePullBackOff   0          2m37s
myawx-memcached-0             1/1     Running            0          2m37s
myawx-task-78c64c6965-lrs2l   0/1     ImagePullBackOff   0          2m37s
myawx-memcached-1             1/1     Running            0          36s
myawx-memcached-2             1/1     Running            0          26s
myawx-postgresql-0            1/1     Running            0          2m36s
myawx-55dc6b97c-7xjqr         1/3     ErrImagePull       0          2m37s


$ kubectl get pods -n datreeawx
NAME                          READY   STATUS             RESTARTS   AGE
myawx-rabbitmq-0              1/1     Running            0          14m
myawx-memcached-0             1/1     Running            0          14m
myawx-memcached-1             1/1     Running            0          12m
myawx-memcached-2             1/1     Running            0          12m
myawx-postgresql-0            1/1     Running            0          14m
myawx-web-88c9d89f4-kt7td     0/1     ImagePullBackOff   0          14m
myawx-task-78c64c6965-lrs2l   0/1     ImagePullBackOff   0          14m
myawx-55dc6b97c-7xjqr         1/3     ImagePullBackOff   0          14m

Several of them used a lable long since removed out of github

I moved to the latest label I saw on their AWX images

$ helm upgrade -n datreeawx myawx lifen-charts/awx --set awx_task.image.tag=11.2.0 --set awx_web.image.tag=11.2.0 --set
 default_admin_password=MyPassword
W0710 22:14:19.599212    2786 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0710 22:14:19.817989    2786 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Release "myawx" has been upgraded. Happy Helming!
NAME: myawx
LAST DEPLOYED: Sun Jul 10 22:14:17 2022
NAMESPACE: datreeawx
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:

However, this did not solve the issue

$ kubectl get pods -n datreeawx
NAME                          READY   STATUS             RESTARTS          AGE
myawx-memcached-0             1/1     Running            0                 7h55m
myawx-memcached-1             1/1     Running            0                 7h53m
myawx-memcached-2             1/1     Running            0                 7h53m
myawx-postgresql-0            1/1     Running            0                 7h55m
myawx-task-589676bdf9-wpd6j   1/1     Running            0                 7h34m
myawx-web-88c9d89f4-kt7td     0/1     ImagePullBackOff   0                 7h55m
myawx-55dc6b97c-7xjqr         1/3     ImagePullBackOff   0                 7h55m
myawx-web-9f8dbc576-b5tcl     0/1     CrashLoopBackOff   153 (2m42s ago)   7h34m
myawx-rabbitmq-0              0/1     Running            80 (3m58s ago)    7h33m
myawx-58cb777df-m4r29         2/3     Running            128 (6m11s ago)   7h34m

Even rotating the pod doesn’t help.

$ kubectl get pods -n datreeawx
NAME                          READY   STATUS             RESTARTS          AGE
myawx-memcached-0             1/1     Running            0                 7h59m
myawx-memcached-1             1/1     Running            0                 7h57m
myawx-memcached-2             1/1     Running            0                 7h57m
myawx-postgresql-0            1/1     Running            0                 7h59m
myawx-task-589676bdf9-wpd6j   1/1     Running            0                 7h38m
myawx-58cb777df-m4r29         2/3     CrashLoopBackOff   129 (3m17s ago)   7h38m
myawx-web-88c9d89f4-kt7td     0/1     ImagePullBackOff   0                 7h59m
myawx-55dc6b97c-7xjqr         1/3     ImagePullBackOff   0                 7h59m
myawx-rabbitmq-0              0/1     Running            81 (3m13s ago)    7h38m
myawx-web-9f8dbc576-q8c75     0/1     CrashLoopBackOff   4 (29s ago)       2m59s

$ kubectl delete pod myawx-web-9f8dbc576-q8c75 -n datreeawx &
[1] 23059

$ kubectl get pods -n datreeawx | grep web
myawx-web-88c9d89f4-kt7td     0/1     ImagePullBackOff   0                8h
myawx-web-9f8dbc576-79pfk     0/1     CrashLoopBackOff   4 (17s ago)      2m47s

$ kubectl logs myawx-web-9f8dbc576-79pfk -n datreeawx
Using /etc/ansible/ansible.cfg as config file

$ kubectl get pods -n datreeawx | grep web
myawx-web-88c9d89f4-kt7td     0/1     ImagePullBackOff   0               8h
myawx-web-9f8dbc576-79pfk     0/1     Running            17 (17s ago)    38m

Lastly, just in case there was a newer chart, I checked that as well

$ helm search repo lifen-charts
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
lifen-charts/awx                        10.0.3          v10.0.0         A Helm chart for Kubernetes
lifen-charts/kube-iptables-tailer       0.2.3           v0.1.0          A Helm chart for Kubernetes
lifen-charts/looker                     0.1.2           2.16.0          A Looker Helm chart for Kubernetes.
lifen-charts/neuvector                  1.5.2           3.2.1           NeuVector Full Lifecycle Container Security Pla...
lifen-charts/op-scim-bridge             1.0.3           v2.3.1          The 1Password SCIM bridge
lifen-charts/squid                      0.4.3           v0.1.0          Squid proxy helm chart
lifen-charts/teleport                   0.1.1                           Teleport Enterprise

but 10.0.3 (which I used) is the latest.

I’ll have to call this path a bust.

I then cleaned up so I didn’t leave running broken charts

$ helm list -n datreeawx
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
myawx   datreeawx       2               2022-07-10 22:14:17.366489 -0500 CDT    deployed        awx-10.0.3      v10.0.0
$ helm delete myawx -n datreeawx
release "myawx" uninstalled

Other Charts: Gitea

The next chart comes from a Consulting firm, Novum-RGI.

First, we add the Helm repo

$ helm repo add novum-rgi-helm https://novumrgi.github.io/helm/
"novum-rgi-helm" has been added to your repositories

Then install

$ helm install -n giteaawx --create-namespace gitea novum-rgi-helm/awx
NAME: gitea
LAST DEPLOYED: Sun Jul 10 22:06:49 2022
NAMESPACE: giteaawx
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace giteaawx -l "app.kubernetes.io/name=awx,app.kubernetes.io/instance=gitea" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8052 to use your application"
  kubectl --namespace giteaawx port-forward $POD_NAME 8052:8052

Note: you can use the defaults (adminPassword:awxPassword123). Or override (as you should) with --set awx.adminPassword=mypasword --set awx.adminUser=myadminuser --set awx.adminMail=myemailaddress

We can just deal with the defaults for now:

$ helm get values gitea -n giteaawx --all
COMPUTED VALUES:
awx:
  adminMail: admin@awx.com
  adminPassword: awxPassword123
  adminUser: admin
  ...

Let’s get the running pods

$ kubectl get pods -n giteaawx
NAME                         READY   STATUS    RESTARTS   AGE
gitea-postgresql-0           1/1     Running   0          9m16s
gitea-awx-66ff757c75-f62ws   3/3     Running   0          9m16s

And port-forward

$ kubectl port-forward svc/gitea-awx -n giteaawx 8052:8052
Forwarding from 127.0.0.1:8052 -> 8052
Forwarding from [::1]:8052 -> 8052
Handling connection for 8052
Handling connection for 8052
Handling connection for 8052
Handling connection for 8052
Handling connection for 8052
Handling connection for 8052
Handling connection for 8052

Here we see the older “winged” AWX login

And once logged in, we can see it’s the newer version (AWX 15.0.0 , Ansible 2.9.13).

One thing that caught my attention was that this instance had some demo content already there (I did not add the Demo Job Template or any projects)

Ingress

Now let’s assume I wanted to add Ingress to this instance.

As before, we could create an R53 Route

$ cat r53-awx2.json
{
  "Comment": "CREATE awx2 fb.s A record ",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "awx2.freshbrewed.science",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "73.242.50.46"
          }
        ]
      }
    }
  ]
}

$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-awx2.j
son
{
    "ChangeInfo": {
        "Id": "/change/C08657872IO7KCP8AI9S7",
        "Status": "PENDING",
        "SubmittedAt": "2022-07-11T11:47:47.187Z",
        "Comment": "CREATE awx2 fb.s A record "
    }
}

Then add the Ingress definition

$ cat awx2-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 32k
    nginx.ingress.kubernetes.io/proxy-buffers-number: 8 32k
    nginx.ingress.kubernetes.io/proxy-read-timeout: "43200"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "43200"
  generation: 1
  labels:
    app: awx2
  name: awx2-ingress
  namespace: giteaawx
spec:
  ingressClassName: nginx
  rules:
  - host: awx2.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: gitea-awx
            port:
              number: 8052
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - awx2.freshbrewed.science
    secretName: tls-awx2

$ kubectl apply -f awx2-ingress.yaml
ingress.networking.k8s.io/awx2-ingress created

Clearly, I’ll want to set the password now that it’s externally exposed.

$ helm upgrade -n giteaawx --create-namespace gitea novum-rgi-helm/awx --set awx.adminPassword=myrealpassword --set awx.adminUser=myrealuser --set awx.adminMail=isaac.johnson@gmail.com
Release "gitea" has been upgraded. Happy Helming!
NAME: gitea
LAST DEPLOYED: Mon Jul 11 06:53:33 2022
NAMESPACE: giteaawx
STATUS: deployed
REVISION: 2
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace giteaawx -l "app.kubernetes.io/name=awx,app.kubernetes.io/instance=gitea" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8052 to use your application"
  kubectl --namespace giteaawx port-forward $POD_NAME 8052:8052

As soon as we see the Cert is valid, we can test

$ kubectl get cert -n giteaawx
NAME       READY   SECRET     AGE
tls-awx2   False   tls-awx2   43s

$ kubectl get cert -n giteaawx
NAME       READY   SECRET     AGE
tls-awx2   True    tls-awx2   3m13s

Interestingly enough, I did not find the upgrade actually changed the password.

However, I changed in the UI and it worked. To be certain it would persist, I also rotated the pods and tried again

$ kubectl delete pod gitea-postgresql-0 -n giteaawx & kubectl delete pod gitea-awx-66ff757c75-f62ws -n giteaawx &
[1] 25222
[2] 25223

$ kubectl get pods -n giteaawx
NAME                         READY   STATUS    RESTARTS   AGE
gitea-postgresql-0           1/1     Running   0          70s
gitea-awx-66ff757c75-4mjzt   2/3     Running   0          83s

Summary

Ansible as a provisioner solves the problem of how do we update non-containerized infrastructure in a reliable repeatable way.

Ansible is one of the most popular infrastructure updating platforms thus there are plenty of example playbooks and sample code. And as it is Open Source, plenty have expanded Ansibles functionality with plugins and tasks.

We can also use Ansible in conjunction with Packer to build and update VM Images and AMIs.