In the case of those who have a cross cloud or hybrid cloud environment, or prefer to use cloud-agnostic tooling, there are ways to monitor using several other monitoring frameworks.

One of the more common is the Elastic-Stack, often referred to as ELK (Elasticsearch, Logstash and Kibana). More recent variants often replace Logstash with things like FluentD or FluentBit (EFK).

Unlike prior blog posts, we're going to continue on from Part 3 - so if you haven't set up you AKS cluster yet, please head back and create your cluster first.

Before we dig in, log into you Cluster:

$ az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all the subscriptions to which you have access...
[
  {
    "cloudName": "AzureCloud",
    "id": "abc123a-1234-1234-1234-abcd1234ab",
    "isDefault": true,
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "1234567-8901-2345-6789-0123456789",
    "user": {
      "name": "isaac.johnson@cdc.com",
      "type": "user"
    }
  }
]
$ az aks get-credentials --resource-group idj-aks-monitoring --name idj-aks-monitoring-aks1
Merged "idj-aks-monitoring-aks1" as current context in /Users/isaac.johnson/.kube/config

Next, following this guide we add the repo and install the chart for the elasticsearch operator.

$ helm repo add akomljen-charts https://raw.githubusercontent.com/komljen/helm-charts/master/charts/
"akomljen-charts" has been added to your repositories
AHD-MBP13-048:~ isaac.johnson$ helm install --name es-operator --namespace logging akomljen-charts/elasticsearch-operator
NAME:   es-operator
LAST DEPLOYED: Sun Mar 31 19:27:19 2019
NAMESPACE: logging
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                                AGE
es-operator-elasticsearch-operator  1s

==> v1/ClusterRoleBinding
NAME                                AGE
es-operator-elasticsearch-operator  1s

==> v1/Deployment
NAME                                READY  UP-TO-DATE  AVAILABLE  AGE
es-operator-elasticsearch-operator  0/1    1           0          1s

==> v1/Pod(related)
NAME                                                 READY  STATUS             RESTARTS  AGE
es-operator-elasticsearch-operator-7c4c6f9cff-zthvt  0/1    ContainerCreating  0         1s

==> v1/ServiceAccount
NAME                                SECRETS  AGE
es-operator-elasticsearch-operator  1        1s


AHD-MBP13-048:~ isaac.johnson$ kubectl get pods -n logging
NAME                                                  READY   STATUS    RESTARTS   AGE
es-operator-elasticsearch-operator-7c4c6f9cff-zthvt   1/1     Running   0          27s

Then install the EFK stack


$ helm install --name efk --namespace logging akomljen-charts/efk
NAME:   efk
LAST DEPLOYED: Sun Mar 31 21:10:13 2019
NAMESPACE: logging
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME            AGE
efk-fluent-bit  2s

==> v1/ClusterRoleBinding
NAME            AGE
efk-fluent-bit  2s

==> v1/ConfigMap
NAME                              DATA  AGE
efk-elasticsearch-curator-config  2     2s
efk-fluent-bit-config             6     2s
efk-kibana                        1     2s

==> v1/ElasticsearchCluster
NAME         AGE
efk-cluster  2s

==> v1/Pod(related)
NAME                        READY  STATUS             RESTARTS  AGE
efk-kibana-bc68c74d7-fvlsm  0/1    ContainerCreating  0         2s
fluent-bit-7vk52            0/1    ContainerCreating  0         2s
fluent-bit-tjsjq            0/1    ContainerCreating  0         2s

==> v1/Secret
NAME                          TYPE    DATA  AGE
efk-fluent-bit-es-tls-secret  Opaque  1     2s

==> v1/Service
NAME        TYPE       CLUSTER-IP   EXTERNAL-IP  PORT(S)  AGE
efk-kibana  ClusterIP  10.0.61.238  <none>       443/TCP  2s

==> v1/ServiceAccount
NAME            SECRETS  AGE
efk-fluent-bit  1        2s

==> v1beta1/CronJob
NAME                       SCHEDULE   SUSPEND  ACTIVE  LAST SCHEDULE  AGE
efk-elasticsearch-curator  0 1 * * *  False    0       <none>         2s

==> v1beta1/DaemonSet
NAME        DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
fluent-bit  2        2        0      2           0          <none>         2s

==> v1beta1/Deployment
NAME        READY  UP-TO-DATE  AVAILABLE  AGE
efk-kibana  0/1    1           0          2s

We can look up the Kibana pod and port-forward to check it out after the pods come up:

$ kubectl get pods -n logging
NAME                                                  READY   STATUS              RESTARTS   AGE
efk-kibana-bc68c74d7-fvlsm                            0/1     ContainerCreating   0          31s
es-client-efk-cluster-767bcdf64d-mjxgm                0/1     ContainerCreating   0          30s
es-data-efk-cluster-default-0                         0/1     ContainerCreating   0          30s
es-master-efk-cluster-default-0                       0/1     ContainerCreating   0          30s
es-operator-elasticsearch-operator-7c4c6f9cff-hgxw4   1/1     Running             0          89s
fluent-bit-7vk52                                      1/1     Running             0          31s
fluent-bit-tjsjq                                      1/1     Running             0          31s
AHD-MBP13-048:~ isaac.johnson$ kubectl port-forward -n logging efk-kibana-bc68c74d7-fvlsm 5601:5601
Forwarding from 127.0.0.1:5601 -> 5601
Forwarding from [::1]:5601 -> 5601
Handling connection for 5601
Kibana accessed via port-forward

Using Kibana

Go to management and create an index pattern on “kubernetes_cluster*”. Then use @timestamp as a filter:

That’s it!  Now you just need to go to “Discover” to checkout your cluster logs:

We can now replicate the same search we did in our last blog entry:

If you want to do alerting, you’ll need to install x-pack on es and kibana.  You can fine guides here:

Installing with x-pack

I want to preface by saying we won't get all the way there. So read ahead if you want to follow this whole guide.  I'll cover more in the wrap up of commercial Elastic - but the summary is that support for containerized Elastic is tenuous at best.

Create a keygroup and vault:

$ az group create -l eastus -n MyELKStack
{
  "id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack",
  "location": "eastus",
  "managedBy": null,
  "name": "MyELKStack",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": null
}

$ az keyvault create --name ELKKeyVault --resource-group MyELKStack
{
  "id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.KeyVault/vaults/ELKKeyVault",
  "location": "eastus",

Get your UPN and set the policy on the Vault.  I'm not mangling my UPN email as usual since i found the format strange and want you to be able to see the format.

az ad user list | grep userPrincipalName
    "userPrincipalName": "cli@isaacjohnsongmail.onmicrosoft.com",
    "userPrincipalName": "isaac.johnson_gmail.com#EXT#@isaacjohnsongmail.onmicrosoft.com",
    "userPrincipalName": "testuser@isaacjohnsongmail.onmicrosoft.com",

$ az keyvault set-policy --upn cli@isaacjohnsongmail.onmicrosoft.com --name ELKKeyVault --resource-group MyELKStack --certificate-permissions create delete get import list update --key-permissions create delete get import list update --secret-permissions get delete list set
{
  "id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.KeyVault/vaults/ELKKeyVault",
  "location": "eastus",
  "name": "ELKKeyVault",
..

$ az keyvault set-policy --upn isaac.johnson_gmail.com#EXT#@isaacjohnsongmail.onmicrosoft.com --name ELKKeyVault --resource-group MyELKStack --certificate-permissions create delete get import list update --key-permissions create delete get import list update --secret-permissions get delete list set
{
  "id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.KeyVault/vaults/ELKKeyVault",
  "location": "eastus",

Next create a public IP and a redis instance

$ az network public-ip create -n MyIpName --resource-group=MyELKStack --allocation-method=static
{
  "publicIp": {
    "ddosSettings": null,
    "dnsSettings": null,
    "etag": "W/\"51259ba3-d155-4241-bd0d-798e9077db54\"",
    "id": "/subscriptions/abc123a-1234-1234-1234-abcd1234ab/resourceGroups/MyELKStack/providers/Microsoft.Network/publicIPAddresses/MyIpName",
    "idleTimeoutInMinutes": 4,
    "ipAddress": "40.121.67.134",



$ az redis create --name idj-dev-logscache --location eastus --resource-group MyELKStack --sku Standard --vm-size C1
 - Running ..

Next, launch an nginx to front our service to the public IP. there are two ways you can do that:

$ cat nginx-values.yaml | grep loadBalancerIP
    loadBalancerIP: "40.121.67.134"


$ helm install -f nginx-values.yaml stable/nginx-ingress
NAME:   orbiting-bobcat
LAST DEPLOYED: Sun Mar 31 22:11:38 2019

$ kubectl get svc | grep ingress-controller
orbiting-bobcat-nginx-ingress-controller        LoadBalancer   10.0.213.164   <pending>        80:32083/TCP,443:30758/TCP   4m43s

Or you can assume it will create it's own IP

helm install stable/nginx-ingress --namespace kube-system --set controller.replicaCount=2

$ kubectl get svc -n kube-system
NAME                                         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
geared-otter-nginx-ingress-controller        LoadBalancer   10.0.160.14    23.96.87.38   80:31391/TCP,443:30973/TCP   61s

Lastly clone and install from git the Elastic help chart

$ git clone https://github.com/Azure/helm-elasticstack.git
Cloning into 'helm-elasticstack'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 734 (delta 4), reused 0 (delta 0), pack-reused 721
Receiving objects: 100% (734/734), 207.78 KiB | 2.08 MiB/s, done.
Resolving deltas: 100% (376/376), done.

$ cd helm-elasticstack/charts/elasticsearch/
AHD-MBP13-048:elasticsearch isaac.johnson$ ls
Chart.yaml	README.md	deploy.sh	environments	templates	values.yaml
AHD-MBP13-048:elasticsearch isaac.johnson$ ./deploy.sh -e acs -n elk
Checking kubectl command
Checking helm command
Checking az command

Ideally we would install with x-pack, but that requires a licensed product and here is where we really run into a wall.  

You can generate a 30 day sample license, but the socials suggest the lowest tier paid license (Gold) that might get you alerting runs $13k a year for 3 nodes - and that is for the most basic gold package.  They won’t list a price on their forums or sub page (https://www.elastic.co/subscriptions).

The end result is that for all this work, we end up with a Kibana roughly identical to what we already fired with a simple helm command at the top using akomljen-charts/efk.

Cleanup

Make sure to delete all your helm deployments

$ helm list
NAME           	REVISION	UPDATED                 	STATUS  	CHART              	APP VERSION	NAMESPACE  
geared-otter   	1       	Sun Mar 31 22:18:32 2019	DEPLOYED	nginx-ingress-1.3.1	0.22.0     	kube-system
mothy-goat     	1       	Sun Mar 31 13:50:21 2019	DEPLOYED	guestbook-0.2.0    	           	default    
orbiting-bobcat	1       	Sun Mar 31 22:11:38 2019	DEPLOYED	nginx-ingress-1.3.1	0.22.0     	default    
AHD-MBP13-048:elasticsearch isaac.johnson$ helm delete geared-otter --purge
release "geared-otter" deleted
AHD-MBP13-048:elasticsearch isaac.johnson$ helm delete mothy-goat --purge
release "mothy-goat" deleted
AHD-MBP13-048:elasticsearch isaac.johnson$ helm delete orbiting-bobcat --purge
'release "orbiting-bobcat" deleted

Then delete the resource groups from the portal.

Summary

In this guide we explored installing the Elastic stack two different ways into our Cluster and looked at how Kibana can help diagnose issues in our cluster. We stopped short of getting x-pack to run so we did not dig into EFK alerting.  In our next post, we will look into AppDynamics.