Fresh/Brewed

Gitlab Revisit: Part 3

Published: Jun 7, 2022 by Isaac Johnson

Thus far we’ve covered quite a lot in Gitlab. In Part 1 we covered Setup, CI/CD using both Private and Shared Runners. In our last post, Part 2 we covered Issue Management, JIRA integration, MR process, Service Desk, Milestones, Package and Container Registries.

Today we will wrap the series by covering Wiki Pages, Snippets (Gists), and dig into GitOps one more time.

Wiki pages

Gitlab includes a Wiki system that makes it easy to create markdown driven documentation.

/content/images/2022/05/gitlab-95.png

We can choose to create a wiki page or enable the confluence wiki integration (for Atlassian Confluence).

Let’s start the with that.

First, I’ll need to enable Confluence in my Atlassian portal:

/content/images/2022/05/gitlab-96.png

Clicking enable starts the process

/content/images/2022/05/gitlab-97.png

TThis brings us to the setup wizard

/content/images/2022/05/gitlab-98.png

We need to provide a name for our “Space”

/content/images/2022/05/gitlab-99.png

We now have a Confluence wiki here

/content/images/2022/05/gitlab-100.png

Back in Gitlab, wwe now can save the URL

/content/images/2022/05/gitlab-101.png

But note, if we create a page back in Gitlab, it will not show up right now

/content/images/2022/05/gitlab-102.png

What ends up happening is we just get a link over to our Confluence

/content/images/2022/05/gitlab-103.png

This seems rather useless to me, but I suppose it’s something.

If we want to remove that link, we can go to Settings/Integrations and set the Confluence Workspace to inactive

/content/images/2022/05/gitlab-104.png

After saving changes, our local wiki (the page we created above) is available again

/content/images/2022/05/gitlab-105.png

Snippets

There often is a need to send a quick snippet of code to someone. This is similar to Github’s Gists.

We can go to Snippets and create a new snippet by clicking “New Snippet”

/content/images/2022/05/gitlab-106.png

Let’s first create a private snippet

/content/images/2022/05/gitlab-107.png

I now have a private-snippet-1.

/content/images/2022/05/gitlab-108.png

In an anonymous window, i won’t be able to see it

/content/images/2022/05/gitlab-109.png

We can create a public snippet

/content/images/2022/05/gitlab-110.png

We now have a snippet that should be visible anywhere

/content/images/2022/05/gitlab-111.png

We can compare that with a public gist

/content/images/2022/05/gitlab-112.png

GitOps

I’ll start with a fresh AKS cluster.

Create a resource group and create/reset the SP credential:

$ az account set --subscription "Pay-As-You-Go"
builder@DESKTOP-QADGF36:~/Workspaces/newtest$ az ad sp create-for-rbac --name "idjknsp01" --skip-assignment --output json > mysp.json
WARNING: Found an existing application instance of "7a60e530-6d09-4e03-84d2-699943aa0933". We will patch it
WARNING: The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
WARNING: 'name' property in the output is deprecated and will be removed in the future. Use 'appId' instead.

$ export SP_PASS=`cat mysp.json | jq -r .password`
$ export SP_ID=`cat mysp.json | jq -r .appId`

$ az group create -n aksgitlabrg --location centralus
{
  "id": "/subscriptions/d955c0ba-13dc-44cf-a29a-8fed74cbb22d/resourceGroups/aksgitlabrg",
  "location": "centralus",
  "managedBy": null,
  "name": "aksgitlabrg",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

Now we can create the cluster

$ az aks create -n idjaksgl1 -g aksgitlabrg --location centralus --node-count 3 --enable-cluster-autoscaler --min-count 2 --max-count 4 --generate-ssh-keys --network-plugin azure --network-policy azure --service-principal $SP_ID --client-secret $SP_PASS
 \ Running ..

Next, I’ll pull the kubeconfig and test.

$ az aks list -o table
Name       Location    ResourceGroup    KubernetesVersion    ProvisioningState    Fqdn
---------  ----------  ---------------  -------------------  -------------------  -------------------------------------------------------------
idjaksgl1  centralus   aksgitlabrg      1.22.6               Succeeded            idjaksgl1-aksgitlabrg-d955c0-a080d420.hcp.centralus.azmk8s.io

$ rm -f ~/.kube/config
$ az aks get-credentials -n idjaksgl1 -g aksgitlabrg --admin
Merged "idjaksgl1-admin" as current context in /home/builder/.kube/config

$ kubectl get nodes
NAME                                STATUS   ROLES   AGE     VERSION
aks-nodepool1-45381683-vmss000000   Ready    agent   3m21s   v1.22.6
aks-nodepool1-45381683-vmss000001   Ready    agent   3m26s   v1.22.6
aks-nodepool1-45381683-vmss000002   Ready    agent   3m33s   v1.22.6

Under Infrastructure/Kubernetes clusters I’ll click “Connect a Cluster” and type a new name (myaksagent)

/content/images/2022/05/gitlab-113.png

then click “Register”

/content/images/2022/05/gitlab-114.png

Then we can use helm to install

/content/images/2022/05/gitlab-115.png

We’ll follow the steps and install the agent

$ helm repo add gitlab https://charts.gitlab.io
"gitlab" already exists with the same configuration, skipping
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Unable to get an update from the "myharbor" chart repository (https://harbor.freshbrewed.science/chartrepo/library):
        failed to fetch https://harbor.freshbrewed.science/chartrepo/library/index.yaml : 502 Bad Gateway
...Successfully got an update from the "sonarqube" chart repository
...Successfully got an update from the "dapr" chart repository
...Successfully got an update from the "kubecost" chart repository
...Successfully got an update from the "uptime-kuma" chart repository
...Successfully got an update from the "epsagon" chart repository
...Successfully got an update from the "sumologic" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "rancher-latest" chart repository
...Successfully got an update from the "newrelic" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm upgrade --install gitlab-agent gitlab/gitlab-agent --namespace gitlab-agent --create-namespace --set config.token=F*******************************g --set config.kasAddress=wss://kas.gitlab.com

I can now see the agent is deployed

$ helm list -n gitlab-agent
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
gitlab-agent    gitlab-agent    1               2022-05-30 16:12:49.592431791 -0500 CDT deployed        gitlab-agent-1.1.0      v15.0.0

$ kubectl get pods -n gitlab-agent
NAME                            READY   STATUS    RESTARTS   AGE
gitlab-agent-7ffc86ff4f-7vc6x   1/1     Running   0          55s

I tried many variants of the agent config. using the Porject ID, name, paths with and without slashes.

gitops:
  manifest_projects:
  - id: 'isaac.johnson/dockerWithTests2'
    default_namespace: gitlab-agent
    paths:
    - glob: '**/*.yaml'
ci_access:
  projects:
    - id: 'isaac.johnson/dockerWithTests2'

/content/images/2022/05/gitlab-116.png

In all cases, the logs showed an RPC error

{"level":"error","time":"2022-05-30T21:40:28.268Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"36439577","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BEKHEDX6ZKP20VX3RERT06"}
{"level":"error","time":"2022-05-30T21:40:55.425Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"36439577","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BEMBZ3K9PRMFRW9J9KP4CW"}
{"level":"info","time":"2022-05-30T21:41:30.424Z","msg":"Stopping synchronization worker","mod_name":"gitops","project_id":"36439577"}
{"level":"info","time":"2022-05-30T21:41:30.424Z","msg":"Waiting for synchronization worker to stop","mod_name":"gitops","project_id":"36439577"}
{"level":"info","time":"2022-05-30T21:41:30.424Z","msg":"Starting synchronization worker","mod_name":"gitops","project_id":"/isaac.johnson/dockerWithTests2"}
{"level":"error","time":"2022-05-30T21:41:30.521Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"/isaac.johnson/dockerWithTests2","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BENE6H5P1R9VBR1B30BWXR"}
{"level":"error","time":"2022-05-30T21:41:42.844Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"/isaac.johnson/dockerWithTests2","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BENT8Y74EDSYZZPEWZG3KR"}
{"level":"error","time":"2022-05-30T21:42:15.048Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"/isaac.johnson/dockerWithTests2","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BEPSQAE8J8DYD1K2KZGDWD"}
{"level":"info","time":"2022-05-30T21:42:30.501Z","msg":"Stopping synchronization worker","mod_name":"gitops","project_id":"/isaac.johnson/dockerWithTests2"}
{"level":"info","time":"2022-05-30T21:42:30.502Z","msg":"Waiting for synchronization worker to stop","mod_name":"gitops","project_id":"/isaac.johnson/dockerWithTests2"}
{"level":"info","time":"2022-05-30T21:42:30.502Z","msg":"Starting synchronization worker","mod_name":"gitops","project_id":"isaac.johnson/dockerWithTests2"}
{"level":"error","time":"2022-05-30T21:42:30.607Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"isaac.johnson/dockerWithTests2","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BEQ8VX3BJGZRMQRX77ASSV"}
{"level":"error","time":"2022-05-30T21:42:46.756Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"isaac.johnson/dockerWithTests2","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BEQRP6FMGC3XS754HPBV31"}
{"level":"error","time":"2022-05-30T21:43:22.331Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"isaac.johnson/dockerWithTests2","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BERVDWV21S2F11T14PHCNN"}

I checked the projects API (https://gitlab.com/api/v4/projects?id_after=36439575&id_before=36439579) and found the project ID should be 36439577 but clearly this does not work.

gitops:
  manifest_projects:
  - id: '36439577'
    default_namespace: gitlab-agent
    paths:
    - glob: '**/*.yaml'

logs:

$ kubectl logs gitlab-agent-7ffc86ff4f-7vc6x -n gitlab-agent | tail -n10
{"level":"info","time":"2022-05-30T23:06:30.109Z","msg":"Waiting for synchronization worker to stop","mod_name":"gitops","project_id":"idjohnson/dockerWithTests2"}
{"level":"info","time":"2022-05-30T23:06:30.110Z","msg":"Starting synchronization worker","mod_name":"gitops","project_id":"36439577/cluster_agents"}
{"level":"error","time":"2022-05-30T23:06:30.676Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"36439577/cluster_agents","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BKH2F7MV8EDC280GW6ZZDR"}
{"level":"error","time":"2022-05-30T23:06:44.180Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"36439577/cluster_agents","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BKHG1N78NQXWKWBB26A9TF"}
{"level":"info","time":"2022-05-30T23:07:10.197Z","msg":"Stopping synchronization worker","mod_name":"gitops","project_id":"36439577/cluster_agents"}
{"level":"info","time":"2022-05-30T23:07:10.197Z","msg":"Waiting for synchronization worker to stop","mod_name":"gitops","project_id":"36439577/cluster_agents"}
{"level":"info","time":"2022-05-30T23:07:10.197Z","msg":"Starting synchronization worker","mod_name":"gitops","project_id":"36439577"}
{"level":"error","time":"2022-05-30T23:07:10.310Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"36439577","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BKJ9GGZ2AV0HJ0AVNF80WT"}
{"level":"error","time":"2022-05-30T23:07:24.854Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"36439577","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BKJQRRGNA020SKADQZCNEF"}
{"level":"error","time":"2022-05-30T23:07:55.463Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"36439577","agent_id":20302,"error":"rpc error: code = NotFound desc = project not found","correlation_id":"01G4BKKNN99H8RY7E50XPGYNK3"}

I tried and built out a terraform example from their Nov-2021 guide here

But upon trying, I got errors about missing modules

builder@DESKTOP-QADGF36:~/Workspaces/newtest/terraform/gitlab-agent$ terraform init
Initializing modules...
╷
│ Error: Module has no versions
│
│ Module "gitlab.com/gitlab-org/kubernetes-agent-terraform-register-agent/local" (main.tf:17) has no versions available on
│ gitlab.com.
╵

╷
│ Error: Module has no versions
│
│ Module "gitlab.com/gitlab-org/kubernetes-agent-terraform-register-agent/local" (main.tf:17) has no versions available on
│ gitlab.com.
╵

Then the guide pointed to the source; https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent-terraform-register-agent which shows a 404 page.

I also tried forcing a high privileged PAT as the config.token and rotating the agent pod. but that failed even worse

$ kubectl logs -n gitlab-agent gitlab-agent-7ffc86ff4f-ln6fv
{"level":"info","time":"2022-05-30T23:28:37.312Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
{"level":"error","time":"2022-05-30T23:28:37.506Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01G4BMSJHZKVX8GYCYF3Q2YFF5"}
{"level":"error","time":"2022-05-30T23:28:37.506Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01G4BMSJHZKVX8GYCYEYK0W89W"}
{"level":"warn","time":"2022-05-30T23:28:37.506Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01G4BMSJHZKVX8GYCYF1RMTDXW"}
{"level":"error","time":"2022-05-30T23:28:48.302Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01G4BMSX4R7TBGX9G71DMKJ65A"}
{"level":"warn","time":"2022-05-30T23:28:48.978Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01G4BMSXSVXPMBX03W5MVP3107"}

Some of the many guides I tried;

  • getting Project IDs from the Graph: https://gitlab.com/api/v4/projects?id_after=36439575&id_before=36439579
    • https://docs.gitlab.com/ee/api/projects.html
  • https://docs.gitlab.com/ee/user/clusters/agent/install/
  • https://about.gitlab.com/blog/2021/11/04/gitops-with-gitlab-infrastructure-provisioning/
  • https://about.gitlab.com/blog/2021/11/18/gitops-with-gitlab-connecting-the-cluster/
  • https://docs.gitlab.com/ee/user/clusters/agent/install/index.html
  • https://docs.gitlab.com/ee/user/clusters/agent/gitops.html#gitops-configuration-reference
  • https://docs.gitlab.com/ee/user/clusters/agent/ci_cd_workflow.html
  • https://docs.gitlab.com/ee/user/clusters/agent/install/
  • https://docs.gitlab.com/ee/user/clusters/agent/work_with_agent.html

I verified the Project ID in the GraphQL as well

/content/images/2022/05/gitlab-117.png

Using Argo

I didn’t want to end a write-up without some path. We could use Flux via Azure AKS/Arc or GCP Anthos. I went back to my on-prem ArgoCD and added it as an app

We can add the cluster

$ argocd cluster add idjaksgl1-admin
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `idjaksgl1-admin` with full cluster level admin privileges. Do you want to continue [y/N]? y
INFO[0002] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0002] ClusterRole "argocd-manager-role" created
INFO[0002] ClusterRoleBinding "argocd-manager-role-binding" created
Cluster 'https://idjaksgl1-aksgitlabrg-d955c0-e625977b.hcp.centralus.azmk8s.io:443' added

I’ll add the Gitlab repo

/content/images/2022/05/gitlab-119.png

then add an application

/content/images/2022/05/gitlab-120.png

If I was on a standard ArgoCD, it might look like this

/content/images/2022/05/gitlab-118.png

This will then sync and install the app

/content/images/2022/05/gitlab-121.png

I initially got an error due to a missing “test” namespace. I quickly corrected

$ kubectl create ns test
namespace/test created

and retried the sync

/content/images/2022/05/gitlab-122.png

When done (and i had to make a small correction to my .gitlab-ci), We could see it launch and serve traffic

image: docker:20.10.16

variables:
  DOCKER_TLS_CERTDIR: "/certs"

services:
  - docker:20.10.16-dind

stages:
  - build

# Build MR
docker_build_all:
  stage: build
  image: node:latest
  script:
    - echo "@nodewithtests:registry=https://gitlab.com/api/v4/projects/36439577/packages/npm/">.npmrc
    - npm config set -- '//gitlab.com/api/v4/projects/36439577/packages/npm/:_authToken' "${CI_JOB_TOKEN}"
    - npm config set always-auth true
    - npm publish --registry https://gitlab.com/api/v4/projects/36439577/packages/npm/

docker_build:
  stage: build
  script:
    - export
    - docker build --target test -t registry.gitlab.com/isaac.johnson/dockerwithtests2 .
  rules:
    - if: $CI_COMMIT_BRANCH != 'main'

# Build prod
docker_build_main:
  stage: build
  script:
    - set +x
    - docker login registry.gitlab.com -u $GITLAB_REGISTRY_USER -p $GITLAB_REGISTRY_PASSWORD
    - set -x
    - docker build -t registry.gitlab.com/isaac.johnson/dockerwithtests2:latest .
    - docker push registry.gitlab.com/isaac.johnson/dockerwithtests2:latest
  rules:
    - if: $CI_COMMIT_BRANCH == 'main'

/content/images/2022/05/gitlab-123.png

Summary

Gitlab offers quite a lot of features. It’s a well-rounded suite that goes toe-to-toe with offerings from Microsoft (Github and Azure DevOps) and Atlassian. I found the Issue Management was far more impressive than that found in Github, but I still felt JIRA and Azure DevOps are the best for Work Items/Agile. I found the runners worked as well as those found elsewhere, but generally I prefer the more selective nature of Azure DevOps on agent pools.

I also had lunch recently with a colleague whose company is all-in on Gitlab and loves it. Evidently, they get cutting edge features and Gitlab works hard to support them on the latest rollouts. I also think they bundle a fair number of features in the free tier. I was surprised I got as much as I did in Issue Management, Runners and Configurable features.

The disappointing bits really came down to GitOps. I really wanted to make that work and just couldn’t. Perhaps in the future they have a more robust method to do GitOps, but until then, I would just stick with ArgoCD or Flux. For what it’s worth, I did post to their forum seeking some help, but to no avail.

gitlab kubernetes

Have something to add? Feedback? You can use the feedback form

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes