Revisiting Kubernetes and KEDA

Published: Feb 26, 2021 by Isaac Johnson

Let’s revisit KEDA (Kubernetes Event Drive Autoscaling) and Azure Functions. We looked into this last March and did a hello world sample using Dockerhub and a simple function. Since then, Azure Function Core Tools 3 has been released and there is now even more you can do with Azure Functions and KEDA.

Setup a cluster

If you don’t have one already, create an AKS cluster

az aks account set --subscription "Pay-As-You-Go"
az group create --name idjaks04rg --location centralus
{
  "id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1asdf/resourceGroups/idjaks04rg",
  "location": "centralus",
  "managedBy": null,
  "name": "idjaks04rg",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}
az ad sp create-for-rbac -n idjaks04sp --skip-assignment --output json > my_sp.json
Changing "idjaks04sp" to a valid URI of "http://idjaks04sp", which is the required format used for service principal names
Found an existing application instance of "4d3b5455-asdf-asdf-asdf-39daed1casdf". We will patch it
cat my_sp.json | jq -r .appId
4d3b5455-8ab1-4dc3-954b-39daed1cd954

az aks create --resource-group idjaks04rg --name idjaks04 --location centralus --node-count 3 --enable-cluster-autoscaler --min-count 2 --max-count 4 --generate-ssh-keys --network-plugin azure --network-policy azure --service-principal 4d3b5455-8ab1-4dc3-954b-39daed1cd954 --client-secret f2ab6d29-d489-4f9d-8a53-57ea48b4eb74
 - Running ..                

Add KEDA Chart Repo

We need to add the Kedacore Helm repo

$ helm repo add kedacore https://kedacore.github.io/charts
"kedacore" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kedacore" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "openfaas" chart repository
...Successfully got an update from the "bitnami" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈

Installing KEDA

$ kubectl create ns keda
namespace/keda created
builder@DESKTOP-JBA79RT:~$ helm install keda kedacore/keda --version 1.4.2 --namespace keda
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: keda
LAST DEPLOYED: Sun Feb 21 11:37:27 2021
NAMESPACE: keda
STATUS: deployed
REVISION: 1
TEST SUITE: None

Installing Azure Functions Core 3

We need to first set up apt to pull from the Microsoft Apt repo

$ curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
$ sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
$ sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-$(lsb_release -cs)-prod $(lsb_release -cs) main" > /etc/apt/sources.list.d/dotnetdev.list'
$ sudo apt-get update
Get:1 https://packages.microsoft.com/repos/azure-cli bionic InRelease [3965 B]
Get:2 https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod bionic InRelease [4003 B]
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 https://apt.releases.hashicorp.com bionic InRelease [4421 B]
Get:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease [4003 B]
Hit:6 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:7 https://packages.microsoft.com/repos/azure-cli bionic/main amd64 Packages [12.8 kB]
Get:8 https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod bionic/main amd64 Packages [164 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:10 https://apt.releases.hashicorp.com bionic/main amd64 Packages [19.3 kB]
Get:11 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 Packages [164 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1544 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1885 kB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1109 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1718 kB]
Get:19 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB]
Fetched 8183 kB in 3s (3204 kB/s)
Reading package lists... Done

Now install the Dotnetcore function runtime

$ sudo apt-get install -y azure-functions-core-tools-3
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  azure-functions-core-tools-3
0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
Need to get 209 MB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod bionic/main amd64 azure-functions-core-tools-3 amd64 3.0.3284-1 [209 MB]
Fetched 209 MB in 8s (27.3 MB/s)
Selecting previously unselected package azure-functions-core-tools-3.
(Reading database ... 164029 files and directories currently installed.)
Preparing to unpack .../azure-functions-core-tools-3_3.0.3284-1_amd64.deb ...
Unpacking azure-functions-core-tools-3 (3.0.3284-1) ...
Setting up azure-functions-core-tools-3 (3.0.3284-1) ...

Telemetry
---------
The Azure Functions Core tools collect usage data in order to help us improve your experience.
The data is anonymous and doesn't include any user specific or personal information. The data is collected by Microsoft.

You can opt-out of telemetry by setting the FUNCTIONS_CORE_TOOLS_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Create a function

Let’s create a dir for our work and use func init to create the structure

$ func init --docker --worker-runtime node --language javascript
Writing package.json
Writing .gitignore
Writing host.json
Writing local.settings.json
Writing /home/builder/Workspaces/azureFunction01/.vscode
Writing Dockerfile
Writing .dockerignore

checking what was created

$ ls -ltra
total 36
drwxr-xr-x 43 builder builder 4096 Feb 21 11:42 ..
-rw-r--r-- 1 builder builder 130 Feb 21 11:44 package.json
-rw-r--r-- 1 builder builder 395 Feb 21 11:44 .gitignore
-rw-r--r-- 1 builder builder 288 Feb 21 11:44 host.json
-rw-r--r-- 1 builder builder 115 Feb 21 11:44 local.settings.json
drwxr-xr-x 2 builder builder 4096 Feb 21 11:44 .vscode
-rw-r--r-- 1 builder builder 381 Feb 21 11:44 Dockerfile
-rw-r--r-- 1 builder builder 19 Feb 21 11:44 .dockerignore
drwxr-xr-x 3 builder builder 4096 Feb 21 11:44 .

At this point we have the shell for making and testing Azure Functions, but none are defined yet. In fact our package.json is pretty much empty:

We can create a function, however, using the func tool:

$ func new
Select a number for template:
1. Azure Blob Storage trigger
2. Azure Cosmos DB trigger
3. Durable Functions activity
4. Durable Functions HTTP starter
5. Durable Functions orchestrator
6. Azure Event Grid trigger
7. Azure Event Hub trigger
8. HTTP trigger
9. IoT Hub (Event Hub)
10. Azure Queue Storage trigger
11. RabbitMQ trigger
12. SendGrid
13. Azure Service Bus Queue trigger
14. Azure Service Bus Topic trigger
15. SignalR negotiate HTTP trigger
16. Timer trigger
Choose option: 8
HTTP trigger
Function name: [HttpTrigger] SampleHttpTrigger
Writing /home/builder/Workspaces/azureFunction01/SampleHttpTrigger/index.js
Writing /home/builder/Workspaces/azureFunction01/SampleHttpTrigger/function.json
The function "SampleHttpTrigger" was created successfully from the "HTTP trigger" template.

We now have a “SampleHttpTrigger” function.

Testing

First, i added a block to avoid proxies on localhost

$ cat local.settings.json
{
  "IsEncrypted": false,
  "Values": {
    "NO_PROXY": "localhost,127.0.0.1",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "AzureWebJobsStorage": ""
  }
}

Since I’m using WSL, i set node to use 10.22.1

$ nvm list
         v9.3.0
       v10.22.1
-> system
node -> stable (-> v10.22.1) (default)
nvm ustable -> 10.22 (-> v10.22.1) (default)
se unstable -> 9.3 (-> v9.3.0) (default)
iojs -> iojs- (-> N/A) (default)
builder@DESKTOP-JBA79RT:~/Workspaces/azureFunction01$ nvm use 10.22.1
Now using node v10.22.1 (npm v6.14.6)

Then launched

$ func host start

Azure Functions Core Tools
Core Tools Version: 3.0.3284 Commit hash: 98bc25e668274edd175a1647fe5a9bc4ffb6887d
Function Runtime Version: 3.0.15371.0


Functions:

        SampleHttpTrigger: [GET,POST] http://localhost:7071/api/SampleHttpTrigger

For detailed output, run func with --verbose flag.
[2021-02-21T17:57:10.495Z] Worker process started and initialized.
[2021-02-21T17:57:15.316Z] Host lock lease acquired by instance ID '0000000000000000000000002D380465'.

Testing the URL:

Was also reflected locally:

[2021-02-21T17:58:38.166Z] Executing 'Functions.SampleHttpTrigger' (Reason='This function was programmatically called via the host APIs.', Id=18997f03-87d2-4048-a814-29c9ccd8bc9b)
[2021-02-21T17:58:38.231Z] JavaScript HTTP trigger function processed a request.
[2021-02-21T17:58:38.288Z] Executed 'Functions.SampleHttpTrigger' (Succeeded, Id=18997f03-87d2-4048-a814-29c9ccd8bc9b, Duration=146ms)

Now if we try and just publish a function app here, it wont work:

$ func azure functionapp publish SampleHttpTrigger
Can't find app with name "SampleHttpTrigger"

Before we deploy to k8s, we better setup an ACR

$ az acr create -n idjacr04cr -g idjaks04rg --sku Basic --admin-enabled true
{- Finished ..
  "adminUserEnabled": true,
  "creationDate": "2021-02-21T18:05:20.313633+00:00",
  "dataEndpointEnabled": false,
  "dataEndpointHostNames": [],
  "encryption": {
    "keyVaultProperties": null,
    "status": "disabled"
  },
...

Then login

$ az acr login --name idjacr04cr
Login Succeeded

Build and deploy to k8s

Let’s now build a container and push to k8s

$ func kubernetes deploy --name samplehttptrigger --registry idjacr04cr.azurecr.io
Running 'docker build -t idjacr04cr.azurecr.io/samplehttptrigger /home/builder/Workspaces/azureFunction01'...done
Running 'docker push idjacr04cr.azurecr.io/samplehttptrigger'....................................................................................................................................................................................................................done
secret/samplehttptrigger created
secret/func-keys-kube-secret-samplehttptrigger created
serviceaccount/samplehttptrigger-function-keys-identity-svc-act created
role.rbac.authorization.k8s.io/functions-keys-manager-role created
rolebinding.rbac.authorization.k8s.io/samplehttptrigger-function-keys-identity-svc-act-functions-keys-manager-rolebinding created
service/samplehttptrigger-http created
deployment.apps/samplehttptrigger-http created
Waiting for deployment "samplehttptrigger-http" rollout to finish: 0 of 1 updated replicas are available...

We can check our deployment, but soon realize we neglected to setup a secret

$ kubectl describe pod samplehttptrigger-http-54cd894697-zh8dk | tail -n5
  Normal BackOff 3m (x6 over 4m20s) kubelet Back-off pulling image "idjacr04cr.azurecr.io/samplehttptrigger"
  Normal Pulling 2m45s (x4 over 4m21s) kubelet Pulling image "idjacr04cr.azurecr.io/samplehttptrigger"
  Warning Failed 2m45s (x4 over 4m20s) kubelet Failed to pull image "idjacr04cr.azurecr.io/samplehttptrigger": rpc error: code = Unknown desc = Error response from daemon: Get https://idjacr04cr.azurecr.io/v2/samplehttptrigger/manifests/latest: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.
  Warning Failed 2m45s (x4 over 4m20s) kubelet Error: ErrImagePull
  Warning Failed 2m34s (x7 over 4m20s) kubelet Error: ImagePullBackOff

We need an image pull secret applied.

Let’s get the username/password from ACR:

Create an image pull secret

$ kubectl create secret docker-registry idjacr04cred --docker-server=idjacr04cr.azurecr.io --docker-username=idjacr04cr --docker-password=pOjOGT+8WvL853AzvHQpHQU7ybTapuo7 --docker-email=johnsi10@medtronic.com
secret/idjacr04cred created

_ Note: I found I needed to delete the former deployment in order for it to change to use the imagepull secret_

$ kubectl delete deployment samplehttptrigger-http
deployment.apps "samplehttptrigger-http" deleted

$ func kubernetes deploy --pull-secret idjacr04cred --name samplehttptrigger --registry idjacr04cr.azurecr.io
Running 'docker build -t idjacr04cr.azurecr.io/samplehttptrigger /home/builder/Workspaces/azureFunction01'...done
Running 'docker push idjacr04cr.azurecr.io/samplehttptrigger'.........done
secret/samplehttptrigger unchanged
secret/func-keys-kube-secret-samplehttptrigger unchanged
serviceaccount/samplehttptrigger-function-keys-identity-svc-act unchanged
role.rbac.authorization.k8s.io/functions-keys-manager-role unchanged
rolebinding.rbac.authorization.k8s.io/samplehttptrigger-function-keys-identity-svc-act-functions-keys-manager-rolebinding unchanged
service/samplehttptrigger-http unchanged
deployment.apps/samplehttptrigger-http created
Waiting for deployment "samplehttptrigger-http" rollout to finish: 0 of 1 updated replicas are available...
deployment "samplehttptrigger-http" successfully rolled out
        SampleHttpTrigger - [httpTrigger]
        Invoke url: http://13.86.102.216/api/samplehttptrigger?code=nPbviiIE8fiFzUiMhh0wOp3HLOaNdamHPaf9aXoDiYTLxvdQVVMfGw==

        Master key: dnpCHO0HFcR/1g8WqzmFChNwThrjJSBiDnd2zhoFs/UPZ3oBIa2Hsw==

We now have it running in AKS

Setting up a Pipeline

Let’s create a repo and push our local contents

We will also need to create a Kubernetes and Docker Registry service connection in Azure DevOps (often in our project)

Create an Azure Pipeline. You’ll want to reference that ACR and AKS in the steps below in the first two inputs.

trigger:
- main
 
pool:
  vmImage: ubuntu-latest
 
steps:
- task: FuncToolsInstaller@0
  inputs:
    version: 'latest'
 
- task: AzureFunctionOnKubernetes@0
  inputs:
    dockerRegistryServiceConnection: 'idjacr04cr'
    kubernetesServiceConnection: 'idjaks04'
    namespace: 'default'
    secretName: 'mypipelinesecret2'
    appName: 'idjtestfunctionapp'
    waitForStability: false
    arguments: '--javascript'

We can now see it in action:

builder@DESKTOP-JBA79RT:~$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
samplehttptrigger-http 1/1 1 1 18m
builder@DESKTOP-JBA79RT:~$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
idjtestfunctionapp-http 0/1 1 0 12s
samplehttptrigger-http 1/1 1 1 19m

In order to finish the deployment, I did need a secret created in the namespace

$ cat samplehttptrigger.secret.yaml
apiVersion: v1
data:
  AzureWebJobsStorage: ""
  FUNCTIONS_WORKER_RUNTIME: bm9kZQ==
  NO_PROXY: bG9jYWxob3N0LDEyNy4wLjAuMQ==
kind: Secret
metadata:
  name: mypipelinesecret
  namespace: default
type: Opaque

Now we can implement the secret create in the YAML pipeline and test

trigger:
- main
 
pool:
  vmImage: ubuntu-latest
 
steps:
 
- task: FuncToolsInstaller@0
  inputs:
    version: 'latest'

- task: Kubernetes@1
  inputs:
    connectionType: 'Kubernetes Service Connection'
    kubernetesServiceEndpoint: 'idjaks04'
    namespace: 'default'
    secretType: 'generic'
    secretArguments: |
      --from-literal=NO_PROXY=localhost,127.0.0.1
      --from-literal=FUNCTIONS_WORKER_RUNTIME=node
      --from-literal=AzureWebJobsStorage=''
    secretName: 'mypipelinesecret2'
 
- task: AzureFunctionOnKubernetes@0
  inputs:
    dockerRegistryServiceConnection: 'idjacr04cr'
    kubernetesServiceConnection: 'idjaks04'
    namespace: 'default'
    secretName: 'mypipelinesecret2'
    appName: 'idjtestfunctionapp'
    waitForStability: false
    arguments: '--javascript'