Automating AKS Deployments like a Boss: Part 4 (EFK)
In the case of those who have a cross cloud or hybrid cloud environment, or prefer to use cloud-agnostic tooling, there are ways to monitor using several other monitoring frameworks.
One of the more common is the Elastic-Stack, often referred to as ELK (Elasticsearch, Logstash and Kibana). More recent variants often replace Logstash with things like FluentD or FluentBit (EFK).
Unlike prior blog posts, we're going to continue on from Part 3 - so if you haven't set up you AKS cluster yet, please head back and create your cluster first.
Before we dig in, log into you Cluster:
$ az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all the subscriptions to which you have access...
[
{
"cloudName": "AzureCloud",
"id": "abc123a-1234-1234-1234-abcd1234ab",
"isDefault": true,
"name": "Pay-As-You-Go",
"state": "Enabled",
"tenantId": "1234567-8901-2345-6789-0123456789",
"user": {
"name": "isaac.johnson@cdc.com",
"type": "user"
}
}
]
$ az aks get-credentials --resource-group idj-aks-monitoring --name idj-aks-monitoring-aks1
Merged "idj-aks-monitoring-aks1" as current context in /Users/isaac.johnson/.kube/config
Next, following this guide we add the repo and install the chart for the elasticsearch operator.
$ helm repo add akomljen-charts https://raw.githubusercontent.com/komljen/helm-charts/master/charts/
"akomljen-charts" has been added to your repositories
AHD-MBP13-048:~ isaac.johnson$ helm install --name es-operator --namespace logging akomljen-charts/elasticsearch-operator
NAME: es-operator
LAST DEPLOYED: Sun Mar 31 19:27:19 2019
NAMESPACE: logging
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
es-operator-elasticsearch-operator 1s
==> v1/ClusterRoleBinding
NAME AGE
es-operator-elasticsearch-operator 1s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
es-operator-elasticsearch-operator 0/1 1 0 1s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
es-operator-elasticsearch-operator-7c4c6f9cff-zthvt 0/1 ContainerCreating 0 1s
==> v1/ServiceAccount
NAME SECRETS AGE
es-operator-elasticsearch-operator 1 1s
AHD-MBP13-048:~ isaac.johnson$ kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
es-operator-elasticsearch-operator-7c4c6f9cff-zthvt 1/1 Running 0 27s
Then install the EFK stack
$ helm install --name efk --namespace logging akomljen-charts/efk
NAME: efk
LAST DEPLOYED: Sun Mar 31 21:10:13 2019
NAMESPACE: logging
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
efk-fluent-bit 2s
==> v1/ClusterRoleBinding
NAME AGE
efk-fluent-bit 2s
==> v1/ConfigMap
NAME DATA AGE
efk-elasticsearch-curator-config 2 2s
efk-fluent-bit-config 6 2s
efk-kibana 1 2s
==> v1/ElasticsearchCluster
NAME AGE
efk-cluster 2s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
efk-kibana-bc68c74d7-fvlsm 0/1 ContainerCreating 0 2s
fluent-bit-7vk52 0/1 ContainerCreating 0 2s
fluent-bit-tjsjq 0/1 ContainerCreating 0 2s
==> v1/Secret
NAME TYPE DATA AGE
efk-fluent-bit-es-tls-secret Opaque 1 2s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
efk-kibana ClusterIP 10.0.61.238 <none> 443/TCP 2s
==> v1/ServiceAccount
NAME SECRETS AGE
efk-fluent-bit 1 2s
==> v1beta1/CronJob
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
efk-elasticsearch-curator 0 1 * * * False 0 <none> 2s
==> v1beta1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluent-bit 2 2 0 2 0 <none> 2s
==> v1beta1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
efk-kibana 0/1 1 0 2s
We can look up the Kibana pod and port-forward to check it out after the pods come up:
$ kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
efk-kibana-bc68c74d7-fvlsm 0/1 ContainerCreating 0 31s
es-client-efk-cluster-767bcdf64d-mjxgm 0/1 ContainerCreating 0 30s
es-data-efk-cluster-default-0 0/1 ContainerCreating 0 30s
es-master-efk-cluster-default-0 0/1 ContainerCreating 0 30s
es-operator-elasticsearch-operator-7c4c6f9cff-hgxw4 1/1 Running 0 89s
fluent-bit-7vk52 1/1 Running 0 31s
fluent-bit-tjsjq 1/1 Running 0 31s
AHD-MBP13-048:~ isaac.johnson$ kubectl port-forward -n logging efk-kibana-bc68c74d7-fvlsm 5601:5601
Forwarding from 127.0.0.1:5601 -> 5601
Forwarding from [::1]:5601 -> 5601
Handling connection for 5601
Using Kibana
Go to management and create an index pattern on “kubernetes_cluster*”. Then use @timestamp as a filter:
That’s it! Now you just need to go to “Discover” to checkout your cluster logs:
We can now replicate the same search we did in our last blog entry:
If you want to do alerting, you’ll need to install x-pack on es and kibana. You can fine guides here:
- https://www.elastic.co/guide/en/kibana/6.0/installing-xpack-kb.html
- https://github.com/Azure/helm-elasticstack
Installing with x-pack
I want to preface by saying we won't get all the way there. So read ahead if you want to follow this whole guide. I'll cover more in the wrap up of commercial Elastic - but the summary is that support for containerized Elastic is tenuous at best.
Create a keygroup and vault:
$ az group create -l eastus -n MyELKStack
{
"id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack",
"location": "eastus",
"managedBy": null,
"name": "MyELKStack",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": null
}
$ az keyvault create --name ELKKeyVault --resource-group MyELKStack
{
"id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.KeyVault/vaults/ELKKeyVault",
"location": "eastus",
Get your UPN and set the policy on the Vault. I'm not mangling my UPN email as usual since i found the format strange and want you to be able to see the format.
az ad user list | grep userPrincipalName
"userPrincipalName": "cli@isaacjohnsongmail.onmicrosoft.com",
"userPrincipalName": "isaac.johnson_gmail.com#EXT#@isaacjohnsongmail.onmicrosoft.com",
"userPrincipalName": "testuser@isaacjohnsongmail.onmicrosoft.com",
$ az keyvault set-policy --upn cli@isaacjohnsongmail.onmicrosoft.com --name ELKKeyVault --resource-group MyELKStack --certificate-permissions create delete get import list update --key-permissions create delete get import list update --secret-permissions get delete list set
{
"id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.KeyVault/vaults/ELKKeyVault",
"location": "eastus",
"name": "ELKKeyVault",
..
$ az keyvault set-policy --upn isaac.johnson_gmail.com#EXT#@isaacjohnsongmail.onmicrosoft.com --name ELKKeyVault --resource-group MyELKStack --certificate-permissions create delete get import list update --key-permissions create delete get import list update --secret-permissions get delete list set
{
"id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.KeyVault/vaults/ELKKeyVault",
"location": "eastus",
Next create a public IP and a redis instance
$ az network public-ip create -n MyIpName --resource-group=MyELKStack --allocation-method=static
{
"publicIp": {
"ddosSettings": null,
"dnsSettings": null,
"etag": "W/\"51259ba3-d155-4241-bd0d-798e9077db54\"",
"id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.Network/publicIPAddresses/MyIpName",
"idleTimeoutInMinutes": 4,
"ipAddress": "40.121.67.134",
$ az redis create --name idj-dev-logscache --location eastus --resource-group MyELKStack --sku Standard --vm-size C1
- Running ..
Next, launch an nginx to front our service to the public IP. there are two ways you can do that:
$ cat nginx-values.yaml | grep loadBalancerIP
loadBalancerIP: "40.121.67.134"
$ helm install -f nginx-values.yaml stable/nginx-ingress
NAME: orbiting-bobcat
LAST DEPLOYED: Sun Mar 31 22:11:38 2019
$ kubectl get svc | grep ingress-controller
orbiting-bobcat-nginx-ingress-controller LoadBalancer 10.0.213.164 <pending> 80:32083/TCP,443:30758/TCP 4m43s
Or you can assume it will create it's own IP
helm install stable/nginx-ingress --namespace kube-system --set controller.replicaCount=2
$ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
geared-otter-nginx-ingress-controller LoadBalancer 10.0.160.14 23.96.87.38 80:31391/TCP,443:30973/TCP 61s
Lastly clone and install from git the Elastic help chart
$ git clone https://github.com/Azure/helm-elasticstack.git
Cloning into 'helm-elasticstack'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 734 (delta 4), reused 0 (delta 0), pack-reused 721
Receiving objects: 100% (734/734), 207.78 KiB | 2.08 MiB/s, done.
Resolving deltas: 100% (376/376), done.
$ cd helm-elasticstack/charts/elasticsearch/
AHD-MBP13-048:elasticsearch isaac.johnson$ ls
Chart.yaml README.md deploy.sh environments templates values.yaml
AHD-MBP13-048:elasticsearch isaac.johnson$ ./deploy.sh -e acs -n elk
Checking kubectl command
Checking helm command
Checking az command
Ideally we would install with x-pack, but that requires a licensed product and here is where we really run into a wall.
You can generate a 30 day sample license, but the socials suggest the lowest tier paid license (Gold) that might get you alerting runs $13k a year for 3 nodes - and that is for the most basic gold package. They won’t list a price on their forums or sub page (https://www.elastic.co/subscriptions).
The end result is that for all this work, we end up with a Kibana roughly identical to what we already fired with a simple helm command at the top using akomljen-charts/efk.
Cleanup
Make sure to delete all your helm deployments
$ helm list
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
geared-otter 1 Sun Mar 31 22:18:32 2019 DEPLOYED nginx-ingress-1.3.1 0.22.0 kube-system
mothy-goat 1 Sun Mar 31 13:50:21 2019 DEPLOYED guestbook-0.2.0 default
orbiting-bobcat 1 Sun Mar 31 22:11:38 2019 DEPLOYED nginx-ingress-1.3.1 0.22.0 default
AHD-MBP13-048:elasticsearch isaac.johnson$ helm delete geared-otter --purge
release "geared-otter" deleted
AHD-MBP13-048:elasticsearch isaac.johnson$ helm delete mothy-goat --purge
release "mothy-goat" deleted
AHD-MBP13-048:elasticsearch isaac.johnson$ helm delete orbiting-bobcat --purge
'release "orbiting-bobcat" deleted
Then delete the resource groups from the portal.
Summary
In this guide we explored installing the Elastic stack two different ways into our Cluster and looked at how Kibana can help diagnose issues in our cluster. We stopped short of getting x-pack to run so we did not dig into EFK alerting. In our next post, we will look into AppDynamics.