Automating AKS Deployments like a Boss: Part 7
In our last article we covered deploying AKS with the Azure CLI and Kubespray. Today we will dig into two more ways; Hashicorp Terraform and ARM Templates.
Terraform
Terraform is Hashicorp's solution for Infrastructure as Code (IaC). Unlike ARM Templates or Cloudformation files, it's platform independent. Written in an easy to read markdown (HCL), Terraform (TF) leverages providers (like Azure) with modules (like Kubernetes) to create recipes to reliably deploy Infrastucture to a variety of targets. A key point to note is that TF will need to store what it does in a "tfstate" file or files. It's usually best to store these in a HADR way - be it S3 or AFS. One can even use dropbox or git repos.
Getting started
First, we can get a lot of this done via the Azure Cloudshell which happens to have the Hashi suite of tools preloaded. Just click the cloudshell link in the toolbar at top (and in our case, i chose a bash shell)
Next create the directory and main.tf file we’ll need:
isaac@Azure:~$ cd clouddrive
isaac@Azure:~/clouddrive$ mkdir terraform-aks-k8s
isaac@Azure:~/clouddrive$ cd terraform-aks-k8s
isaac@Azure:~/clouddrive/terraform-aks-k8s$ echo -e 'provider "azurerm" {\n\tversion = "~>1.5"\n}\n\nterraform {\n\tbackend "azurerm" {}\n}\n' > main.tf
isaac@Azure:~/clouddrive/terraform-aks-k8s$ cat main.tf
provider "azurerm" {
version = "~>1.5"
}
terraform {
backend "azurerm" {}
}
Next we need to create k8s.tf - this defines the resource group, workspace and analytics, and cluster settings we're going to need.
vi k8s.tf
:set paste
resource "azurerm_resource_group" "k8s" {
name = "${var.resource_group_name}"
location = "${var.location}"
}
resource "azurerm_log_analytics_workspace" "test" {
name = "${var.log_analytics_workspace_name}"
location = "${var.log_analytics_workspace_location}"
resource_group_name = "${azurerm_resource_group.k8s.name}"
sku = "${var.log_analytics_workspace_sku}"
}
resource "azurerm_log_analytics_solution" "test" {
solution_name = "ContainerInsights"
location = "${azurerm_log_analytics_workspace.test.location}"
resource_group_name = "${azurerm_resource_group.k8s.name}"
workspace_resource_id = "${azurerm_log_analytics_workspace.test.id}"
workspace_name = "${azurerm_log_analytics_workspace.test.name}"
plan {
publisher = "Microsoft"
product = "OMSGallery/ContainerInsights"
}
}
resource "azurerm_kubernetes_cluster" "k8s" {
name = "${var.cluster_name}"
location = "${azurerm_resource_group.k8s.location}"
resource_group_name = "${azurerm_resource_group.k8s.name}"
dns_prefix = "${var.dns_prefix}"
linux_profile {
admin_username = "ubuntu"
ssh_key {
key_data = "${file("${var.ssh_public_key}")}"
}
}
agent_pool_profile {
name = "agentpool"
count = "${var.agent_count}"
vm_size = "Standard_DS1_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
service_principal {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}
addon_profile {
oms_agent {
enabled = true
log_analytics_workspace_id = "${azurerm_log_analytics_workspace.test.id}"
}
}
tags {
Environment = "Development"
}
}
ZZ (to exit and save)
Create a variables.tf in much the same way:
variable "client_id" {}
variable "client_secret" {}
variable "agent_count" {
default = 3
}
variable "ssh_public_key" {
default = "~/.ssh/id_rsa.pub"
}
variable "dns_prefix" {
default = "idj-k8stest"
}
variable cluster_name {
default = "idj-k8stest"
}
variable resource_group_name {
default = "azure-k8stest"
}
variable location {
default = "Central US"
}
variable log_analytics_workspace_name {
default = "testLogAnalyticsWorkspaceName"
}
# refer https://azure.microsoft.com/global-infrastructure/services/?products=monitor for log analytics available regions
variable log_analytics_workspace_location {
default = "eastus"
}
# refer https://azure.microsoft.com/pricing/details/monitor/ for log analytics pricing
variable log_analytics_workspace_sku {
default = "PerGB2018"
}
note: above i changed the cluster_name and resource_group_name to something of my own (and as we'll see later, i should have also change the log_analytics_workspace_name... oooo.. foreshadowing)
Lastly, create the output.tf file:
output "client_key" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.client_key}"
}
output "client_certificate" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate}"
}
output "cluster_ca_certificate" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate}"
}
output "cluster_username" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.username}"
}
output "cluster_password" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.password}"
}
output "kube_config" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config_raw}"
}
output "host" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.host}"
}
Before we run this though, we need to create the resource group and storage account:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ az group create --location centralus --name idj-azure-k8stest
{
"id": "/subscriptions/abcd1234-abab-abab-dcdc-dcdc123456/resourceGroups/idj-azure-k8stest",
"location": "centralus",
"managedBy": null,
"name": "idj-azure-k8stest",
"properties": {
"provisioningState": "Succeeded"
},
"tags": {
"Exception": "no",
"Owner": "missing",
"StopResources": "yes"
},
"type": null
}
isaac@Azure:~/clouddrive/terraform-aks-k8s$ az storage account create -n idjk8stesttfstate -g idj-azure-k8stest -l centralus --sku Standard_LRS
{
"accessTier": null,
"creationTime": "2019-04-29T20:56:13.981373+00:00",
"customDomain": null,
"enableAzureFilesAadIntegration": null,
"enableHttpsTrafficOnly": false,
"encryption": {
"keySource": "Microsoft.Storage",
"keyVaultProperties": null,
"services": {
"blob": {
"enabled": true,
"lastEnabledTime": "2019-04-29T20:56:14.106346+00:00"
},
"file": {
"enabled": true,
"lastEnabledTime": "2019-04-29T20:56:14.106346+00:00"
},
"queue": null,
"table": null
}
},
"failoverInProgress": null,
"geoReplicationStats": null,
"id": "/subscriptions/abcd1234-abab-abab-dcdc-dcdc123456/resourceGroups/idj-azure-k8stest/providers/Microsoft.Storage/storageAccounts/idjk8stesttfstate",
"identity": null,
"isHnsEnabled": null,
"kind": "Storage",
"lastGeoFailoverTime": null,
"location": "centralus",
"name": "idjk8stesttfstate",
"networkRuleSet": {
"bypass": "AzureServices",
"defaultAction": "Allow",
"ipRules": [],
"virtualNetworkRules": []
},
"primaryEndpoints": {
"blob": "https://idjk8stesttfstate.blob.core.windows.net/",
"dfs": null,
"file": "https://idjk8stesttfstate.file.core.windows.net/",
"queue": "https://idjk8stesttfstate.queue.core.windows.net/",
"table": "https://idjk8stesttfstate.table.core.windows.net/",
"web": null
},
"primaryLocation": "centralus",
"provisioningState": "Succeeded",
"resourceGroup": "idj-azure-k8stest",
"secondaryEndpoints": null,
"secondaryLocation": null,
"sku": {
"capabilities": null,
"kind": null,
"locations": null,
"name": "Standard_LRS",
"resourceType": null,
"restrictions": null,
"tier": "Standard"
},
"statusOfPrimary": "available",
"statusOfSecondary": null,
"tags": {},
"type": "Microsoft.Storage/storageAccounts"
}
isaac@Azure:~/clouddrive/terraform-aks-k8s$
Now we need to snag the keys for the account:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ az storage account keys list --account-name idjk8stesttfstate --resource-group idj-azure-k8stest -o json
[
{
"keyName": "key1",
"permissions": "Full",
"value": "1mzOV7nV31UpbzP3XQW4ceKMGS1i2TyzRQTs+Gh7W4Yuxpgzy5txPTzdMdonXTgF4/Y1Iv3C9OeTLc52Mqd64Q=="
},
{
"keyName": "key2",
"permissions": "Full",
"value": "1N0ZMJH3RVWTrWQXgysZFOP1BUWxMq4obKzcLZx3xMwpBFCENwpMpY4mbytjawo0/FNg1Dm6/Ehv01DGUEmnmQ=="
}
]
Lastly, create a container to store our tfstate files:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ az storage container create -n tfstate --account-name idjk8stesttfstate --account-key 1mzOV7nV31UpbzP3XQW4ceKMGS1i2TyzRQTs+Gh7W4Yuxpgzy5txPTzdMdonXTgF4/Y1Iv3C9OeTLc52Mqd64Q==
{
"created": true
}
Terraform Init
Finally, let’s TF init.. Noting the storage account (so terraform knows were to store state files):
isaac@Azure:~/clouddrive/terraform-aks-k8s$ terraform init -backend-config="storage_account_name=idjk8stesttfstate" -backend-config="container_name=tfstate" -backend-config="access_key=1mzOV7nV31UpbzP3XQW4ceKMGS1i2TyzRQTs+Gh7W4Yuxpgzy5txPTzdMdonXTgF4/Y1Iv3C9OeTLc52Mqd64Q==" -backend-config="key=codelab.microsoft.tfstate"
Initializing the backend...
Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "azurerm" (1.27.1)...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Previously we covered creating a service user.. We’ll use that same Service principal again (idj-k8sdemo) - you’ll find it in AAD under App Registrations (legacy):
We need the Client ID and secret
Follow guides to get the key. Set TF_VAR_client_id and TF_VAR_client_secret to the SP ID and Secret, respectively
And lastly, let’s just print it all out for sanity before we plan:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ find . -type f -name \*.tf -exec cat {} \; -print
resource "azurerm_resource_group" "k8s" {
name = "${var.resource_group_name}"
location = "${var.location}"
}
resource "azurerm_log_analytics_workspace" "test" {
name = "${var.log_analytics_workspace_name}"
location = "${var.log_analytics_workspace_location}"
resource_group_name = "${azurerm_resource_group.k8s.name}"
sku = "${var.log_analytics_workspace_sku}"
}
resource "azurerm_log_analytics_solution" "test" {
solution_name = "ContainerInsights"
location = "${azurerm_log_analytics_workspace.test.location}"
resource_group_name = "${azurerm_resource_group.k8s.name}"
workspace_resource_id = "${azurerm_log_analytics_workspace.test.id}"
workspace_name = "${azurerm_log_analytics_workspace.test.name}"
plan {
publisher = "Microsoft"
product = "OMSGallery/ContainerInsights"
}
}
resource "azurerm_kubernetes_cluster" "k8s" {
name = "${var.cluster_name}"
location = "${azurerm_resource_group.k8s.location}"
resource_group_name = "${azurerm_resource_group.k8s.name}"
dns_prefix = "${var.dns_prefix}"
linux_profile {
admin_username = "ubuntu"
ssh_key {
key_data = "${file("${var.ssh_public_key}")}"
}
}
agent_pool_profile {
name = "agentpool"
count = "${var.agent_count}"
vm_size = "Standard_DS1_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
service_principal {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}
addon_profile {
oms_agent {
enabled = true
log_analytics_workspace_id = "${azurerm_log_analytics_workspace.test.id}"
}
}
tags {
Environment = "Development"
}
}
./k8s.tf
provider "azurerm" {
version = "~>1.5"
}
terraform {
backend "azurerm" {}
}
./main.tf
output "client_key" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.client_key}"
}
output "client_certificate" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate}"
}
output "cluster_ca_certificate" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate}"
}
output "cluster_username" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.username}"
}
output "cluster_password" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.password}"
}
output "kube_config" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config_raw}"
}
output "host" {
value = "${azurerm_kubernetes_cluster.k8s.kube_config.0.host}"
}
./output.tf
variable "client_id" {}
variable "client_secret" {}
variable "agent_count" {
default = 3
}
variable "ssh_public_key" {
default = "~/.ssh/id_rsa.pub"
}
variable "dns_prefix" {
default = "k8stest"
}
variable cluster_name {
default = "idj-k8stest"
}
variable resource_group_name {
default = "idj-azure-k8stest"
}
variable location {
default = "Central US"
}
variable log_analytics_workspace_name {
default = "testLogAnalyticsWorkspaceName"
}
# refer https://azure.microsoft.com/global-infrastructure/services/?products=monitor for log analytics available regions
variable log_analytics_workspace_location {
default = "eastus"
}
# refer https://azure.microsoft.com/pricing/details/monitor/ for log analytics pricing
variable log_analytics_workspace_sku {
default = "PerGB2018"
}
./variables.tf
If you haven't already, take a moment and create an ssh-key pair in your cloud shell:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ ssh-keygen -t rsa -b 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/home/isaac/.ssh/id_rsa):
Created directory '/home/isaac/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/isaac/.ssh/id_rsa.
Your public key has been saved in /home/isaac/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:ZsUPdzw525clzhaPglfO6EVS9w7x9Wmi9+y56rbU5w8 isaac@cc-b83e3c54-79998bc5bc-jgxt6
The key's randomart image is:
+---[RSA 2048]----+
| o o|
| . o ==|
| + o.%o=|
| . =.@o%o|
| S ..=.X.=|
| o o.++ .|
| ..E+.|
| ...oo|
| o+o+=|
+----[SHA256]-----+
And since our local root is ephemeral and we may need them later, save them to clouddrive:
tar -czvf ~/clouddrive/ssh-for-safety.tgz ~/.ssh
Terraform Plan
This tests our whole plan and saves the plan.out
isaac@Azure:~/clouddrive/terraform-aks-k8s$ terraform plan -out out.plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ azurerm_kubernetes_cluster.k8s
id: <computed>
addon_profile.#: "1"
addon_profile.0.oms_agent.#: "1"
addon_profile.0.oms_agent.0.enabled: "true"
addon_profile.0.oms_agent.0.log_analytics_workspace_id: "${azurerm_log_analytics_workspace.test.id}"
agent_pool_profile.#: "1"
agent_pool_profile.0.count: "3"
agent_pool_profile.0.dns_prefix: <computed>
agent_pool_profile.0.fqdn: <computed>
agent_pool_profile.0.max_pods: <computed>
agent_pool_profile.0.name: "agentpool"
agent_pool_profile.0.os_disk_size_gb: "30"
agent_pool_profile.0.os_type: "Linux"
agent_pool_profile.0.vm_size: "Standard_DS1_v2"
dns_prefix: "k8stest"
fqdn: <computed>
kube_admin_config.#: <computed>
kube_admin_config_raw: <computed>
kube_config.#: <computed>
kube_config_raw: <computed>
kubernetes_version: <computed>
linux_profile.#: "1"
linux_profile.0.admin_username: "ubuntu"
linux_profile.0.ssh_key.#: "1"
linux_profile.0.ssh_key.0.key_data: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPtSRa4N+XgHxg4F7DYWUyXs+C6+CRD2XbWLpYn1AreNDytKYPofG7hs+X+EC4rKAwiByqchmkrYhB/as29Amnv5nUnVdr5CwIXw0SlIkmLhtTEbpSDJkJlKVD2EWhzM4dZgAEHvH3+QeZX7+QR74E37DPmuprStnU9BqSikLSfL68MrCWsNaRUyy4+BxQx1LI6kctT2tJGtI1kDCUhsiPuwukpr4LLyVP+NQrzIfSvQlDcWvnXpEGpKo/KRArIYVfFEg+L1nixhvJ/zAqXii6h8rGziEs3KDDJ7BvTrL/e14E5Biioyd0vrcKpHQODPX3Ix2TH5Hc4rpiRwOg2YFL isaac@cc-b83e3c54-79998bc5bc-jgxt6\n"
location: "centralus"
name: "idj-k8stest"
network_profile.#: <computed>
node_resource_group: <computed>
resource_group_name: "idj-azure-k8stest"
role_based_access_control.#: <computed>
service_principal.#: "1"
service_principal.3467395296.client_id: "05ed2bfd-6340-4bae-85b4-f8e9a0ff5c76"
service_principal.3467395296.client_secret: <sensitive>
tags.%: "1"
tags.Environment: "Development"
+ azurerm_log_analytics_solution.test
id: <computed>
location: "eastus"
plan.#: "1"
plan.0.name: <computed>
plan.0.product: "OMSGallery/ContainerInsights"
plan.0.publisher: "Microsoft"
resource_group_name: "idj-azure-k8stest"
solution_name: "ContainerInsights"
workspace_name: "testLogAnalyticsWorkspaceName"
workspace_resource_id: "${azurerm_log_analytics_workspace.test.id}"
+ azurerm_log_analytics_workspace.test
id: <computed>
location: "eastus"
name: "testLogAnalyticsWorkspaceName"
portal_url: <computed>
primary_shared_key: <computed>
resource_group_name: "idj-azure-k8stest"
retention_in_days: <computed>
secondary_shared_key: <computed>
sku: "PerGB2018"
tags.%: <computed>
workspace_id: <computed>
+ azurerm_resource_group.k8s
id: <computed>
location: "centralus"
name: "idj-azure-k8stest"
tags.%: <computed>
Plan: 4 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
This plan was saved to: out.plan
To perform exactly these actions, run the following command to apply:
terraform apply "out.plan"
If it looks good to you, execute the plan with terraform apply:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ terraform apply out.plan
azurerm_resource_group.k8s: Creating...
location: "" => "centralus"
name: "" => "idj-azure-k8stest"
tags.%: "" => "<computed>"
azurerm_resource_group.k8s: Creation complete after 1s (ID: /subscriptions/fefd6730-7b26-49a2-8c24-...c9b1c/resourceGroups/idj-azure-k8stest)
azurerm_log_analytics_workspace.test: Creating...
location: "" => "eastus"
name: "" => "testLogAnalyticsWorkspaceName"
portal_url: "" => "<computed>"
primary_shared_key: "<sensitive>" => "<sensitive>"
resource_group_name: "" => "idj-azure-k8stest"
retention_in_days: "" => "<computed>"
secondary_shared_key: "<sensitive>" => "<sensitive>"
sku: "" => "PerGB2018"
tags.%: "" => "<computed>"
workspace_id: "" => "<computed>"
azurerm_log_analytics_workspace.test: Still creating... (10s elapsed)
azurerm_log_analytics_workspace.test: Still creating... (20s elapsed)
Error: Error applying plan:
1 error(s) occurred:
* azurerm_log_analytics_workspace.test: 1 error(s) occurred:
* azurerm_log_analytics_workspace.test: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="Conflict" Message="The workspace name 'testLogAnalyticsWorkspaceName' is not unique" Target="name"
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
Ah! We have an error. We need a unique workspace name for log analytics.
Let’s open variables.tf and fix that (add “idj-”)
variable log_analytics_workspace_name {
default = "idj-testLogAnalyticsWorkspaceName"
}
Then run the two steps again:
- terraform plan -out out.plan
- terraform apply out.plan
When done, we should have all the details we need to use the cluster:
azurerm_kubernetes_cluster.k8s: Still creating... (11m0s elapsed)
azurerm_kubernetes_cluster.k8s: Still creating... (11m10s elapsed)
azurerm_kubernetes_cluster.k8s: Creation complete after 11m18s (ID: /subscriptions/fefd6730-7b26-49a2-8c24-...nerService/managedClusters/idj-k8stest)
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
Outputs:
client_certificate = LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUUvVENDQXVXZ0F3SUJBZ0lSQUtpZHRLVkthNWVrSDZpKzR0bklJRTh3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdIaGNOTVRrd05ESTVNakV6TURFMFdoY05NakV3TkRJNE1qRXpNREUwV2pBdwpNUmN3RlFZRFZRUUtFdzV6ZVhOMFpXMDZiV0Z6ZEdWeWN6RVZNQk1HQTFVRUF4TU1iV0Z6ZEdWeVkyeHBaVzUwCk1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBd0d6cnpNUDhYWHhRYXA1aG5CWnkKQ0RCL2hQa3BIeSsvQkNLMWtnVjRETm53WXRJdExvcng3eGZQQUFlOHNuRXZGOHYreEhoY3BLYVhpZjBzUFNOcApKMURQQytSdm54cU5xRjIxNkQ5VXZESldOYVFjTWFCYUZBdGxVbEw2MnBvRUo3cEtEeHcyNk1hY0ZVek15Wm1ECkZzMnd6L1g3OExicUdhdm1lcmozWk5nclh5WjZNbmFYSkIrdjZXcFA0TGtBTzd5ZjVZRWpMRXpndk5FRVBlSHkKbFVjdW5TaTFJTWpSdmFmOWJmKzZ6WElqaDZZT1JHTDhrNmtSbE01SmN5N2lFdE93dm45QjA0QVFuRkQwbTY1aAorNkJxb3ZKV1BIak5DdC9XMHh2UHdHS2hkc2VCSzNaamVhZWhWcXlHcVNnNllFcmYzNjVMOUFWTVYyeDZKeEZvCkVidzZDcy9kTituOGI1VDB5aHhmZ0hIUnRWTEg0Szh5TDZrQ1Y4TzA1T20wbkltTzh6VGdwNjRUdjBGRWdVYUIKWnh0YTVsSXp1M0NrbE5RcWw1WlVNb0hQT29aVVAzREJrM1hPRWs2SFRFRUFMc2RaK2laSTNrdUZNZDA3RWJxQwpFT09GVEY1dWU1RUdaaGx4d2pLUllsaWVuci9iMEJGQzVPSkJILzM2NXNjSTcyaFlWQUF2SHRRVHRIOXk3a0lSClJMSFdvM09BRDA2d0JZaEJvUjJ5K3NDSmgrOGpoRUhwaEpRY1dLYnFJd3pZcm5uODRrZVMrRjlnV3V5OElzQXcKbGlEUnRpNUNPZnRyUjQ2RjBjWlVXT01oQkpvUkpYRlNJb1pWRVRCU21BTG9BekIvMnE2bWRqNDZvUUNvZjdwLwo5Y1YyL3EvK3V6MVZ4aWlXL0R5SjdiTUNBd0VBQWFNMU1ETXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkCkpRUU1NQW9HQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUIKQUxLQ1JlMjVBYUJlYVVJQU9MUC9kUzRHcVZqNG91c3BJaUNlb1A5SVQ3eC9LUGJ1NE5PTXI1R09adVRxN2x1egpZVy9tRFZXTzc1SGVpNGhsZjBUeGRLcEhCSVA0LzlrbW8xZTZlZ0NGK0lsVzhKS1lQd3ZMQnZLWDhsaTkrSTlHCitoZksybmRTbmtaa2xrRmgwOERtVDNBMmRPV3J4OXlSMWRsWG9Mb3kwZ2tpdC9kVDRQMHd2VDZNbHBCaDhxT0gKOFNwdXRLZE1wNmdrZTA0SDhVWitTOVByM0o2SzZjTExDNDlYcy9GR3A1dDR1Mm9VTnk5b1A2cUxNRXQ3QW12MQpaTjBXbTQzMnpIdXVOZGk4czJtVGZMcE94Tmw5bW01b2JFb0Y2REF6dnhtNWtoUXVpbzJGZEcwYmxuTUwvSXlZCk5MTVhpVEZrT2RmQk1HZGp6dHV0UDZrMHVwT3A4MEZsdHhkMThQbUZhUC9vTURDTE1Ob1pBRCtGVnZUWGZPZG0KeDVEMXd2WUlSTzF6OXIySUFhZ2dvaHEzQ3h2Nk1vQzhrSGQzMEVIVERuYllCMFkvWjV4WTMxZmVHMU10T1NzQgo1d1JaclFxd3NMQURsYWZTZDhYOVp0eGNKenc1YmR5SElEeHk0YmVpMVpUbUJIczdlNTVvZVg4MkFlUmVxeVI0CiszM1VOcHI0YVZ4c2RqVHhRUFB3WE1SRnFXMURGejgvbnNVYjdXSFhFcUtXbVd5enZYYU0yeEsyMjBtWmh1c0IKZFhmVlJhTHNwYzRFZ0VCZXh3NWdxT0MvODBObXhaMDdCU3I0SWo1aWJQNllaVzNVNFhHdXlKaEpzTFVDYmx0MQpwbHJVZHJGSGRiTnF1ZFFkSUFNR0FhMFhoem0xUnBrQUNBSXJjV29sRnBybwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
client_key = LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS1FJQkFBS0NBZ0VBd0d6cnpNUDhYWHhRYXA1aG5CWnlDREIvaFBrcEh5Ky9CQ0sxa2dWNERObndZdEl0Ckxvcng3eGZQQUFlOHNuRXZGOHYreEhoY3BLYVhpZjBzUFNOcEoxRFBDK1J2bnhxTnFGMjE2RDlVdkRKV05hUWMKTWFCYUZBdGxVbEw2MnBvRUo3cEtEeHcyNk1hY0ZVek15Wm1ERnMyd3ovWDc4TGJxR2F2bWVyajNaTmdyWHlaNgpNbmFYSkIrdjZXcFA0TGtBTzd5ZjVZRWpMRXpndk5FRVBlSHlsVWN1blNpMUlNalJ2YWY5YmYrNnpYSWpoNllPClJHTDhrNmtSbE01SmN5N2lFdE93dm45QjA0QVFuRkQwbTY1aCs2QnFvdkpXUEhqTkN0L1cweHZQd0dLaGRzZUIKSzNaamVhZWhWcXlHcVNnNllFcmYzNjVMOUFWTVYyeDZKeEZvRWJ3NkNzL2ROK244YjVUMHloeGZnSEhSdFZMSAo0Szh5TDZrQ1Y4TzA1T20wbkltTzh6VGdwNjRUdjBGRWdVYUJaeHRhNWxJenUzQ2tsTlFxbDVaVU1vSFBPb1pVClAzREJrM1hPRWs2SFRFRUFMc2RaK2laSTNrdUZNZDA3RWJxQ0VPT0ZURjV1ZTVFR1pobHh3aktSWWxpZW5yL2IKMEJGQzVPSkJILzM2NXNjSTcyaFlWQUF2SHRRVHRIOXk3a0lSUkxIV28zT0FEMDZ3QlloQm9SMnkrc0NKaCs4agpoRUhwaEpRY1dLYnFJd3pZcm5uODRrZVMrRjlnV3V5OElzQXdsaURSdGk1Q09mdHJSNDZGMGNaVVdPTWhCSm9SCkpYRlNJb1pWRVRCU21BTG9BekIvMnE2bWRqNDZvUUNvZjdwLzljVjIvcS8rdXoxVnhpaVcvRHlKN2JNQ0F3RUEKQVFLQ0FnQkxsbjNLSWZLd2huVFFiUWJLMmNaUmtUdUdjSm5HdG1yQ3BSL09ESGZzRkhCcFlVWitVZlpYaFhLNwpBRThQaVlhR1Fvd0JnWVppNTAzLzFvUGFRRjYrSXZBeUs3bnROZkVQZ2tCQ3l4M3VENWZ0MmxsNDVSUjlhenlSCjZpYkFXRi9wZXpROFcwajJoNDRvajlLaUhydHY2YnphOURYVGlRYk1yc29LeXozalQwYzZTVk1YSDU2ZmdoTUkKRGNyeTY3azd2Z3pSaXJqV1hoeXFEekVyWGVaODNVbGlVdXRUeFk4ZWdTNzhWc2g2aTZ1SlRhR3RHM1U4UzkycwpKcHlUc29iNy9rd2ZpTDNDNXp1ZksrM3FlY1hqb2xCQlVoQ3VFSVg3RDVMRHV0a1JLYTFqaTFzaVI0OHI2cjFsCjNyL0hFeFpiWFE0MFA3TjJWOTUxT1JlQzRVT1BoTktDYmNmY09kb0MxUElLRmhFRllTemREMkVid0R2T1pvUFoKbGZFSEN6ancrZ1p3UjM4NlhtdHo2QmRGS00vQWJabTJxU2JGVUdoT21vVFBUUzBoMm5XMGlJeG1vQk9xb0dnawpxVXVNUmt4WFJEUGY5Y0UrdFNiQ0svbDdadzNqNncxdWs3aEZBRy9yaUZYVTcxQkJ3YThPWUNrNWR3Nk9QdkFFCmZkSThBa08vMnlSeXVjWjUvd3ZzOUM2ZDVERWZFMUd6Wm16ZDJvemk2dVFZMWora3ZjQU5Id0FMQkZ6WENZdzcKODBGOCsrZEQrRUtUdUVGZTA5ZEJyaUw1aVB3T3ZIc2UrRmlucEQ0bVVyZlQ2Rlh3MU5Ta25HbVFNRHY5bHQ0VgplbUJYVGxoR2c2VUkzVEsxaURxY2MwN1JxbjFydGtlZlliYVdWNHBoVmxkUk03SnFnUUtDQVFFQTgzYjJDWG1yCmorR0ExbC9vTnhJYmVWMGRLMEFybGtIYzdLTU15VnRyQkd2dnJjNDBmZXlBQlBodFN6c1RZYlgwcmxPZWdYNk4KZGlPVUx2VDFzTlIxSTVCM3Z4OUhVb0hsTFo2alVZcUlJMnlpL2l5QnlGVlJlaDVQejh1ZGV3cTBPK3JKck5NVwpQeGt3ZCtUOUJ1NktpVkhGUTZqYUR2S1ZYRFYvL0VFWVI4VXZLcm5sNUs2Q3RWVlp5NjNTcDRsVStqMVN4b0FyCnJXNElXL1p0KzFqdGtjb2tzKzJPaVBnWDRid3Ric3BLYzNvQzlCdHUraUNRdk9vbWp3MlVockJMNWo2UXU5TXcKZVpiOXpYV0trOXR2cUF0THgrVGtWOG03OTFCRTQrN0FWT2kwd0k0aW1Nck1Ib2o1Y3Q2QzZuN1BoREcza1NUTgpqTDZIbmlpbkJOQVJUUUtDQVFFQXlsVTZBSzJYZU9qWTlSc3RPeHZNaHF4TWRURUxLTzdub3BNby9iNkpyekExCitqb3k0Y0lFcmt5ZW5VM2ZMUHJFdGFONnMweUE5Y0VodGdwcTA0NDZXV1dYclpUcThvMDN0VTNGcjZrRmZVV2IKMlowZTJxMS8zZXExVVJrbFU1U0hOTGRmTTYvVlp3bmpjMlZXM0FndXFXelVQaEtSSWpiTkZSa2RaeXI5M1ozSQpobkk0V1B3SnNFUUsxSmJ5WGhTMlBIR2hsRlhzSmRnMWsvWGg0ME1IOVVOV01iRlRaRGlYSFVML2VHMmxaS250CmptaEFtWExQeSsxNmVrWkQ5blpaUWtYZDBMVUxJUWtxUGlOMnhKZUc5TFM3cktScGt2Zkx1SmFUR0hiQ1gyZFEKRWFPaUcwL0l1T2tpdU10ZXVvak42alQrS0ZhV00xNHlTRkQ3aURKNi93S0NBUUVBdlNjT2psd0dLN255cEJ0TgpTZ1A2Ym5jbVkyVHV1RVdoaEFSRnVlY1pwUXpLNFFrVkJVU0tUZEpJNzZzTkt0djRKcngycUhzUEl0c1BsTS9pClZxa3AvU0NQdUdCdnhDMkJSY2s3YisybVNZZHN0dmNUT2ZiM1gxNVBlY3pqSkNZejAxMWRwTFUzVW55YWlIdlQKRGVDa3AzMXRPbHprVmJxb3lrUVNsSVJxZmRBNERnVHlGSkowb0xlWUZVT09KM1h1SEtFcVAyaHpCc0kzSW1PNgpIUTZCUE1TRmNYaytSVElqTTlGRXB3L3BlZ0F3YlVFbkhwUFFmOFdYZ1lyMmV4SmJWWjRFU3JrUXIxYmRCeFNzClBxVzlBUjBObWxjK0JPcUo2K0prTTRaWnpWNGltSHVWYTRvYnZobURYSGg5MC93RDg5dkFsN2ExTUFVVFRwQ2MKK29OQS9RS0NBUUVBcFZ2NytIdm0xSWt3YnpsVHdRQWFLbUo4WmphNzBTdFhJZlkxUlJQSXdPV2M1N0ZTemxEYgp0SHVvRWhXSGRPSkppODFjeFZhYityMXB5TFpBMGo5bURwUEozN0phUmZzRjhXWHJUNEhLbTdRZENEK3g3TFJwClVUdTJEZDZJTkFPcWNIb0JFMlA3KzZjSVBkSXZwOG9FSUxGeDBIMHFJaFU1akttbTd4OGl0VkZ6aGZXcGVZZ2EKZnZWTkNLdUl6ejBNSnVBVFZ3RFY2NlFGUGJSeGRXUGVDQ09OT1RXY1dzY3BNV0FGSEM0NFFzbmgvVVhVcmRRNApZWWpuVlRGQjhCYVBJbXVwT2g0TVIyYVBJSk9wU0VneW9xRm0xRlRXZDZlTzFvQXhIWWhKbU9EOWZqMEdpbDJ0ClJQbzhSMXptUVhyM0Z4ZFNORE4zcyttakhKa2F5UmZBandLQ0FRQmFscG9kOUdBOVdCbEt6ZE5ZYmVkQUd1aWQKWFZRSWxLazVtVmxOYmdHWjE3VUpPbkdScVVlTkgxbWExblF5bzhDZ2dkN3dNVHNUYXZwRWs4NFZFMEUxMFJHbApDQ05mN0FzZ1NMeVFnRjRyQ0FLZWZKbW80dDk2SVhITmEybEdndm44cjZ5N05zU0ZrMFplbU41aVBycjhkdFJ6CkNjTFlGSlcySGI4YTBNeDhqbDNWbmE4VHBnb0U3ZnMzZ2Rwd1EyL0ZCdTJlK2ZNS0VSczQ4bFJVNHV4WGg3L1EKQzdyYmpGbnBTRnovYi90T0ppZlpwV3k5MGNRTVRLQmZOemRWRUUwZzlGZ2tHbHJBZjdRdmU0Y1M2Y0xZUEVwdQpOZTc3bTBtT3lYUHZWa3hYU1lLc3VEcGF0QmxJZmh5bTFnM2o2dnptOFMySHJIVTAvelZURnQrRDBCNlUKLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
cluster_ca_certificate = LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUV5RENDQXJDZ0F3SUJBZ0lSQU4yRDMveFl6MnBEeGtISUViQ2JXVHN3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdIaGNOTVRrd05ESTVNakV6TURFMFdoY05NakV3TkRJNE1qRXpNREUwV2pBTgpNUXN3Q1FZRFZRUURFd0pqWVRDQ0FpSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnSVBBRENDQWdvQ2dnSUJBTmFWCkRSOWxKM1ZJaU0xYUdUSkVDQ0NoaDhFT3R1STgwU2Q1MjJaa2RJcC9sb0o1V2dqSHl5bWxDd29lZzVMWDMxZFMKYUVPZWxMN0EzVmxqUDZRMzF1ZlEra1RYU3c1WmtKY3dmcVpMeG9VYUg2NmttZ3pvSDA3Z0RyV3BYUzFZQnBmdQo1bHErd0M3ay9ETEg4WFZqZHMyWEd6YzBLdVRUbjNrWHk5ZGhuZUJlZkVoYk0vbnIrcWdVeERKWDlzUnRmTnRaCkI4Q29pT1NrMVJsQUpSRWF3d2ZKcFJrazhQQzZId2VIUnA3d2FnT2lRcGtsa1VETEJsZGI2azVISUJvUmZVUmsKMGdIQVpiRkljK3lTU3RXeVFHRDMvOU5xTDFWbWJnRitMZXgrd3VkNWY5dGlWUi9kYk5KZ3RsU0taNzNuSWRFcgpOT0h3MGFzUWxScUVTeTJkdlcxSlhQblN1UWtoQU1wYll5NmtlR3BXTndPaTZZL0htL1crM24vZ0JWT3ZzeTBZClh2dUNaUm1oeVhDMGdsSXVRY3k5dWJ0NTFQbE5Gd0doS0VJNkpjTHEwYUYrL1ZqYjNxK2Z0NlBrcTY5TVF5dWUKc3JxQktPVVFudWNRZHk0LzR5MzFOZXZmTGsrSU5NbUVYWVdXeWhDL2M2aGhZOGlnWVJ3TndIaXgwclE4emxkcgplS21FMmNHM3Zzay96VnNiZTYxV2ZnamwxNDhtNmh4RFZwQTBNbE1UdGdqVFZlUFJhNElwdXNIY0c0NnMzalFyClZPMTlnUVFzTmZkYzh4dXMwSnVlNGhBMFBXN3BMR0xDRnFWaEJFbklPZ0lsV3RPSWkwSzhuMlFSdG1jbEl3RmcKbzRtUStJMjRzVkR1YlRYOTY0b0lnN1orMUQ3QlVNTGU3aVFLWE1ubEFnTUJBQUdqSXpBaE1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQ0FRQlViU3IrCmNZbFU0dm81b3cyUFJXdlcxcHFlRGJ6RWhYcFd1blZ5WjVPeVZyTVJ6QUlKL3hrTVliTTV0QzRtNkZ2aEtsMDYKMEpBeHdNU1l6SllxL05tK2xBcXdOdkxJSjJocGd3eHYvZzJoN3JZTHJQUXQ2d0xleGZGSjhMVURVekVjZ1dsOQpvaHdUU1FyWHFGQjBVQUhKekR6WWRyRFhZd3BUaytGNFJaallvNDN0RWthaFVpOC9LUS9xNnZES2xmSU5yV2VxCmkxRk44RXFYZFFSdVkxWUxCZ1BzT2o0RDN4TG94RzRQM0lRMGdxZ3NIZ1JsaWZBN0ovYjhRM280WW15SkRmWlQKUWhrcE1JQjdWU0Q4VDUyVjRZNzUxcGdDQndvcTdqZWVzMVNxb2xoYzNvYlkwUDhnZ1dXZFR5U0dzdmFQaS9lVwpoTkJRazgyTnhINlcwemFmYWxZK014SnNLeDZMTklVSjVtVnNYejhHWXc3ai9lZkZlUHU5UkMwOHBNRis5Q3lOClY5ZXNSa2Z4Ym8ySE1xR0RXcXNqcWEyYUd4RGpXZmVZRDYrNG1ZVUZZem1MR0pZRjU2dCtjQ1ZYa3FkNEF6UkIKSHVXWEVuTXhKbytmT3R4S3d0SmdLRm9kWEVsMER4NG5MV3hLamdEVUVFSHQ4UXZoZGJsVTkrT0xsRnZJZXVEOQpBUDFsUlFFQUtCZzlYN1lLSlF6eDVhNGlDVXlaT3d5QndPVDZ0cTZaMWFBZ1NIU1d0dVhNNGxEbVdrVjdVL2IyCk1HVW5ORW1YQnlUUnVETTkyN2FUSXo0bis5RHlON3V6QlY5amFiZiswY0l5cjYxeTUzdDRBRDJHUzVmK2RDREgKam5vbHlFWFEya0plbVNCcUQzL0hyblJVa3RhVE8yaUJ4cHo0MHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
cluster_password = a9302e29cbec7ac082fe8515c3bd84f1
cluster_username = clusterUser_idj-azure-k8stest_idj-k8stest
host = https://k8stest-c43630f9.hcp.centralus.azmk8s.io:443
kube_config = apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUV5RENDQXJDZ0F3SUJBZ0lSQU4yRDMveFl6MnBEeGtISUViQ2JXVHN3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdIaGNOTVRrd05ESTVNakV6TURFMFdoY05NakV3TkRJNE1qRXpNREUwV2pBTgpNUXN3Q1FZRFZRUURFd0pqWVRDQ0FpSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnSVBBRENDQWdvQ2dnSUJBTmFWCkRSOWxKM1ZJaU0xYUdUSkVDQ0NoaDhFT3R1STgwU2Q1MjJaa2RJcC9sb0o1V2dqSHl5bWxDd29lZzVMWDMxZFMKYUVPZWxMN0EzVmxqUDZRMzF1ZlEra1RYU3c1WmtKY3dmcVpMeG9VYUg2NmttZ3pvSDA3Z0RyV3BYUzFZQnBmdQo1bHErd0M3ay9ETEg4WFZqZHMyWEd6YzBLdVRUbjNrWHk5ZGhuZUJlZkVoYk0vbnIrcWdVeERKWDlzUnRmTnRaCkI4Q29pT1NrMVJsQUpSRWF3d2ZKcFJrazhQQzZId2VIUnA3d2FnT2lRcGtsa1VETEJsZGI2azVISUJvUmZVUmsKMGdIQVpiRkljK3lTU3RXeVFHRDMvOU5xTDFWbWJnRitMZXgrd3VkNWY5dGlWUi9kYk5KZ3RsU0taNzNuSWRFcgpOT0h3MGFzUWxScUVTeTJkdlcxSlhQblN1UWtoQU1wYll5NmtlR3BXTndPaTZZL0htL1crM24vZ0JWT3ZzeTBZClh2dUNaUm1oeVhDMGdsSXVRY3k5dWJ0NTFQbE5Gd0doS0VJNkpjTHEwYUYrL1ZqYjNxK2Z0NlBrcTY5TVF5dWUKc3JxQktPVVFudWNRZHk0LzR5MzFOZXZmTGsrSU5NbUVYWVdXeWhDL2M2aGhZOGlnWVJ3TndIaXgwclE4emxkcgplS21FMmNHM3Zzay96VnNiZTYxV2ZnamwxNDhtNmh4RFZwQTBNbE1UdGdqVFZlUFJhNElwdXNIY0c0NnMzalFyClZPMTlnUVFzTmZkYzh4dXMwSnVlNGhBMFBXN3BMR0xDRnFWaEJFbklPZ0lsV3RPSWkwSzhuMlFSdG1jbEl3RmcKbzRtUStJMjRzVkR1YlRYOTY0b0lnN1orMUQ3QlVNTGU3aVFLWE1ubEFnTUJBQUdqSXpBaE1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQ0FRQlViU3IrCmNZbFU0dm81b3cyUFJXdlcxcHFlRGJ6RWhYcFd1blZ5WjVPeVZyTVJ6QUlKL3hrTVliTTV0QzRtNkZ2aEtsMDYKMEpBeHdNU1l6SllxL05tK2xBcXdOdkxJSjJocGd3eHYvZzJoN3JZTHJQUXQ2d0xleGZGSjhMVURVekVjZ1dsOQpvaHdUU1FyWHFGQjBVQUhKekR6WWRyRFhZd3BUaytGNFJaallvNDN0RWthaFVpOC9LUS9xNnZES2xmSU5yV2VxCmkxRk44RXFYZFFSdVkxWUxCZ1BzT2o0RDN4TG94RzRQM0lRMGdxZ3NIZ1JsaWZBN0ovYjhRM280WW15SkRmWlQKUWhrcE1JQjdWU0Q4VDUyVjRZNzUxcGdDQndvcTdqZWVzMVNxb2xoYzNvYlkwUDhnZ1dXZFR5U0dzdmFQaS9lVwpoTkJRazgyTnhINlcwemFmYWxZK014SnNLeDZMTklVSjVtVnNYejhHWXc3ai9lZkZlUHU5UkMwOHBNRis5Q3lOClY5ZXNSa2Z4Ym8ySE1xR0RXcXNqcWEyYUd4RGpXZmVZRDYrNG1ZVUZZem1MR0pZRjU2dCtjQ1ZYa3FkNEF6UkIKSHVXWEVuTXhKbytmT3R4S3d0SmdLRm9kWEVsMER4NG5MV3hLamdEVUVFSHQ4UXZoZGJsVTkrT0xsRnZJZXVEOQpBUDFsUlFFQUtCZzlYN1lLSlF6eDVhNGlDVXlaT3d5QndPVDZ0cTZaMWFBZ1NIU1d0dVhNNGxEbVdrVjdVL2IyCk1HVW5ORW1YQnlUUnVETTkyN2FUSXo0bis5RHlON3V6QlY5amFiZiswY0l5cjYxeTUzdDRBRDJHUzVmK2RDREgKam5vbHlFWFEya0plbVNCcUQzL0hyblJVa3RhVE8yaUJ4cHo0MHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://k8stest-c43630f9.hcp.centralus.azmk8s.io:443
name: idj-k8stest
contexts:
- context:
cluster: idj-k8stest
user: clusterUser_idj-azure-k8stest_idj-k8stest
name: idj-k8stest
current-context: idj-k8stest
kind: Config
preferences: {}
users:
- name: clusterUser_idj-azure-k8stest_idj-k8stest
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUUvVENDQXVXZ0F3SUJBZ0lSQUtpZHRLVkthNWVrSDZpKzR0bklJRTh3RFFZSktvWklodmNOQVFFTEJRQXcKRFRFTE1Ba0dBMVVFQXhNQ1kyRXdIaGNOTVRrd05ESTVNakV6TURFMFdoY05NakV3TkRJNE1qRXpNREUwV2pBdwpNUmN3RlFZRFZRUUtFdzV6ZVhOMFpXMDZiV0Z6ZEdWeWN6RVZNQk1HQTFVRUF4TU1iV0Z6ZEdWeVkyeHBaVzUwCk1JSUNJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBZzhBTUlJQ0NnS0NBZ0VBd0d6cnpNUDhYWHhRYXA1aG5CWnkKQ0RCL2hQa3BIeSsvQkNLMWtnVjRETm53WXRJdExvcng3eGZQQUFlOHNuRXZGOHYreEhoY3BLYVhpZjBzUFNOcApKMURQQytSdm54cU5xRjIxNkQ5VXZESldOYVFjTWFCYUZBdGxVbEw2MnBvRUo3cEtEeHcyNk1hY0ZVek15Wm1ECkZzMnd6L1g3OExicUdhdm1lcmozWk5nclh5WjZNbmFYSkIrdjZXcFA0TGtBTzd5ZjVZRWpMRXpndk5FRVBlSHkKbFVjdW5TaTFJTWpSdmFmOWJmKzZ6WElqaDZZT1JHTDhrNmtSbE01SmN5N2lFdE93dm45QjA0QVFuRkQwbTY1aAorNkJxb3ZKV1BIak5DdC9XMHh2UHdHS2hkc2VCSzNaamVhZWhWcXlHcVNnNllFcmYzNjVMOUFWTVYyeDZKeEZvCkVidzZDcy9kTituOGI1VDB5aHhmZ0hIUnRWTEg0Szh5TDZrQ1Y4TzA1T20wbkltTzh6VGdwNjRUdjBGRWdVYUIKWnh0YTVsSXp1M0NrbE5RcWw1WlVNb0hQT29aVVAzREJrM1hPRWs2SFRFRUFMc2RaK2laSTNrdUZNZDA3RWJxQwpFT09GVEY1dWU1RUdaaGx4d2pLUllsaWVuci9iMEJGQzVPSkJILzM2NXNjSTcyaFlWQUF2SHRRVHRIOXk3a0lSClJMSFdvM09BRDA2d0JZaEJvUjJ5K3NDSmgrOGpoRUhwaEpRY1dLYnFJd3pZcm5uODRrZVMrRjlnV3V5OElzQXcKbGlEUnRpNUNPZnRyUjQ2RjBjWlVXT01oQkpvUkpYRlNJb1pWRVRCU21BTG9BekIvMnE2bWRqNDZvUUNvZjdwLwo5Y1YyL3EvK3V6MVZ4aWlXL0R5SjdiTUNBd0VBQWFNMU1ETXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkCkpRUU1NQW9HQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUIKQUxLQ1JlMjVBYUJlYVVJQU9MUC9kUzRHcVZqNG91c3BJaUNlb1A5SVQ3eC9LUGJ1NE5PTXI1R09adVRxN2x1egpZVy9tRFZXTzc1SGVpNGhsZjBUeGRLcEhCSVA0LzlrbW8xZTZlZ0NGK0lsVzhKS1lQd3ZMQnZLWDhsaTkrSTlHCitoZksybmRTbmtaa2xrRmgwOERtVDNBMmRPV3J4OXlSMWRsWG9Mb3kwZ2tpdC9kVDRQMHd2VDZNbHBCaDhxT0gKOFNwdXRLZE1wNmdrZTA0SDhVWitTOVByM0o2SzZjTExDNDlYcy9GR3A1dDR1Mm9VTnk5b1A2cUxNRXQ3QW12MQpaTjBXbTQzMnpIdXVOZGk4czJtVGZMcE94Tmw5bW01b2JFb0Y2REF6dnhtNWtoUXVpbzJGZEcwYmxuTUwvSXlZCk5MTVhpVEZrT2RmQk1HZGp6dHV0UDZrMHVwT3A4MEZsdHhkMThQbUZhUC9vTURDTE1Ob1pBRCtGVnZUWGZPZG0KeDVEMXd2WUlSTzF6OXIySUFhZ2dvaHEzQ3h2Nk1vQzhrSGQzMEVIVERuYllCMFkvWjV4WTMxZmVHMU10T1NzQgo1d1JaclFxd3NMQURsYWZTZDhYOVp0eGNKenc1YmR5SElEeHk0YmVpMVpUbUJIczdlNTVvZVg4MkFlUmVxeVI0CiszM1VOcHI0YVZ4c2RqVHhRUFB3WE1SRnFXMURGejgvbnNVYjdXSFhFcUtXbVd5enZYYU0yeEsyMjBtWmh1c0IKZFhmVlJhTHNwYzRFZ0VCZXh3NWdxT0MvODBObXhaMDdCU3I0SWo1aWJQNllaVzNVNFhHdXlKaEpzTFVDYmx0MQpwbHJVZHJGSGRiTnF1ZFFkSUFNR0FhMFhoem0xUnBrQUNBSXJjV29sRnBybwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS1FJQkFBS0NBZ0VBd0d6cnpNUDhYWHhRYXA1aG5CWnlDREIvaFBrcEh5Ky9CQ0sxa2dWNERObndZdEl0Ckxvcng3eGZQQUFlOHNuRXZGOHYreEhoY3BLYVhpZjBzUFNOcEoxRFBDK1J2bnhxTnFGMjE2RDlVdkRKV05hUWMKTWFCYUZBdGxVbEw2MnBvRUo3cEtEeHcyNk1hY0ZVek15Wm1ERnMyd3ovWDc4TGJxR2F2bWVyajNaTmdyWHlaNgpNbmFYSkIrdjZXcFA0TGtBTzd5ZjVZRWpMRXpndk5FRVBlSHlsVWN1blNpMUlNalJ2YWY5YmYrNnpYSWpoNllPClJHTDhrNmtSbE01SmN5N2lFdE93dm45QjA0QVFuRkQwbTY1aCs2QnFvdkpXUEhqTkN0L1cweHZQd0dLaGRzZUIKSzNaamVhZWhWcXlHcVNnNllFcmYzNjVMOUFWTVYyeDZKeEZvRWJ3NkNzL2ROK244YjVUMHloeGZnSEhSdFZMSAo0Szh5TDZrQ1Y4TzA1T20wbkltTzh6VGdwNjRUdjBGRWdVYUJaeHRhNWxJenUzQ2tsTlFxbDVaVU1vSFBPb1pVClAzREJrM1hPRWs2SFRFRUFMc2RaK2laSTNrdUZNZDA3RWJxQ0VPT0ZURjV1ZTVFR1pobHh3aktSWWxpZW5yL2IKMEJGQzVPSkJILzM2NXNjSTcyaFlWQUF2SHRRVHRIOXk3a0lSUkxIV28zT0FEMDZ3QlloQm9SMnkrc0NKaCs4agpoRUhwaEpRY1dLYnFJd3pZcm5uODRrZVMrRjlnV3V5OElzQXdsaURSdGk1Q09mdHJSNDZGMGNaVVdPTWhCSm9SCkpYRlNJb1pWRVRCU21BTG9BekIvMnE2bWRqNDZvUUNvZjdwLzljVjIvcS8rdXoxVnhpaVcvRHlKN2JNQ0F3RUEKQVFLQ0FnQkxsbjNLSWZLd2huVFFiUWJLMmNaUmtUdUdjSm5HdG1yQ3BSL09ESGZzRkhCcFlVWitVZlpYaFhLNwpBRThQaVlhR1Fvd0JnWVppNTAzLzFvUGFRRjYrSXZBeUs3bnROZkVQZ2tCQ3l4M3VENWZ0MmxsNDVSUjlhenlSCjZpYkFXRi9wZXpROFcwajJoNDRvajlLaUhydHY2YnphOURYVGlRYk1yc29LeXozalQwYzZTVk1YSDU2ZmdoTUkKRGNyeTY3azd2Z3pSaXJqV1hoeXFEekVyWGVaODNVbGlVdXRUeFk4ZWdTNzhWc2g2aTZ1SlRhR3RHM1U4UzkycwpKcHlUc29iNy9rd2ZpTDNDNXp1ZksrM3FlY1hqb2xCQlVoQ3VFSVg3RDVMRHV0a1JLYTFqaTFzaVI0OHI2cjFsCjNyL0hFeFpiWFE0MFA3TjJWOTUxT1JlQzRVT1BoTktDYmNmY09kb0MxUElLRmhFRllTemREMkVid0R2T1pvUFoKbGZFSEN6ancrZ1p3UjM4NlhtdHo2QmRGS00vQWJabTJxU2JGVUdoT21vVFBUUzBoMm5XMGlJeG1vQk9xb0dnawpxVXVNUmt4WFJEUGY5Y0UrdFNiQ0svbDdadzNqNncxdWs3aEZBRy9yaUZYVTcxQkJ3YThPWUNrNWR3Nk9QdkFFCmZkSThBa08vMnlSeXVjWjUvd3ZzOUM2ZDVERWZFMUd6Wm16ZDJvemk2dVFZMWora3ZjQU5Id0FMQkZ6WENZdzcKODBGOCsrZEQrRUtUdUVGZTA5ZEJyaUw1aVB3T3ZIc2UrRmlucEQ0bVVyZlQ2Rlh3MU5Ta25HbVFNRHY5bHQ0VgplbUJYVGxoR2c2VUkzVEsxaURxY2MwN1JxbjFydGtlZlliYVdWNHBoVmxkUk03SnFnUUtDQVFFQTgzYjJDWG1yCmorR0ExbC9vTnhJYmVWMGRLMEFybGtIYzdLTU15VnRyQkd2dnJjNDBmZXlBQlBodFN6c1RZYlgwcmxPZWdYNk4KZGlPVUx2VDFzTlIxSTVCM3Z4OUhVb0hsTFo2alVZcUlJMnlpL2l5QnlGVlJlaDVQejh1ZGV3cTBPK3JKck5NVwpQeGt3ZCtUOUJ1NktpVkhGUTZqYUR2S1ZYRFYvL0VFWVI4VXZLcm5sNUs2Q3RWVlp5NjNTcDRsVStqMVN4b0FyCnJXNElXL1p0KzFqdGtjb2tzKzJPaVBnWDRid3Ric3BLYzNvQzlCdHUraUNRdk9vbWp3MlVockJMNWo2UXU5TXcKZVpiOXpYV0trOXR2cUF0THgrVGtWOG03OTFCRTQrN0FWT2kwd0k0aW1Nck1Ib2o1Y3Q2QzZuN1BoREcza1NUTgpqTDZIbmlpbkJOQVJUUUtDQVFFQXlsVTZBSzJYZU9qWTlSc3RPeHZNaHF4TWRURUxLTzdub3BNby9iNkpyekExCitqb3k0Y0lFcmt5ZW5VM2ZMUHJFdGFONnMweUE5Y0VodGdwcTA0NDZXV1dYclpUcThvMDN0VTNGcjZrRmZVV2IKMlowZTJxMS8zZXExVVJrbFU1U0hOTGRmTTYvVlp3bmpjMlZXM0FndXFXelVQaEtSSWpiTkZSa2RaeXI5M1ozSQpobkk0V1B3SnNFUUsxSmJ5WGhTMlBIR2hsRlhzSmRnMWsvWGg0ME1IOVVOV01iRlRaRGlYSFVML2VHMmxaS250CmptaEFtWExQeSsxNmVrWkQ5blpaUWtYZDBMVUxJUWtxUGlOMnhKZUc5TFM3cktScGt2Zkx1SmFUR0hiQ1gyZFEKRWFPaUcwL0l1T2tpdU10ZXVvak42alQrS0ZhV00xNHlTRkQ3aURKNi93S0NBUUVBdlNjT2psd0dLN255cEJ0TgpTZ1A2Ym5jbVkyVHV1RVdoaEFSRnVlY1pwUXpLNFFrVkJVU0tUZEpJNzZzTkt0djRKcngycUhzUEl0c1BsTS9pClZxa3AvU0NQdUdCdnhDMkJSY2s3YisybVNZZHN0dmNUT2ZiM1gxNVBlY3pqSkNZejAxMWRwTFUzVW55YWlIdlQKRGVDa3AzMXRPbHprVmJxb3lrUVNsSVJxZmRBNERnVHlGSkowb0xlWUZVT09KM1h1SEtFcVAyaHpCc0kzSW1PNgpIUTZCUE1TRmNYaytSVElqTTlGRXB3L3BlZ0F3YlVFbkhwUFFmOFdYZ1lyMmV4SmJWWjRFU3JrUXIxYmRCeFNzClBxVzlBUjBObWxjK0JPcUo2K0prTTRaWnpWNGltSHVWYTRvYnZobURYSGg5MC93RDg5dkFsN2ExTUFVVFRwQ2MKK29OQS9RS0NBUUVBcFZ2NytIdm0xSWt3YnpsVHdRQWFLbUo4WmphNzBTdFhJZlkxUlJQSXdPV2M1N0ZTemxEYgp0SHVvRWhXSGRPSkppODFjeFZhYityMXB5TFpBMGo5bURwUEozN0phUmZzRjhXWHJUNEhLbTdRZENEK3g3TFJwClVUdTJEZDZJTkFPcWNIb0JFMlA3KzZjSVBkSXZwOG9FSUxGeDBIMHFJaFU1akttbTd4OGl0VkZ6aGZXcGVZZ2EKZnZWTkNLdUl6ejBNSnVBVFZ3RFY2NlFGUGJSeGRXUGVDQ09OT1RXY1dzY3BNV0FGSEM0NFFzbmgvVVhVcmRRNApZWWpuVlRGQjhCYVBJbXVwT2g0TVIyYVBJSk9wU0VneW9xRm0xRlRXZDZlTzFvQXhIWWhKbU9EOWZqMEdpbDJ0ClJQbzhSMXptUVhyM0Z4ZFNORE4zcyttakhKa2F5UmZBandLQ0FRQmFscG9kOUdBOVdCbEt6ZE5ZYmVkQUd1aWQKWFZRSWxLazVtVmxOYmdHWjE3VUpPbkdScVVlTkgxbWExblF5bzhDZ2dkN3dNVHNUYXZwRWs4NFZFMEUxMFJHbApDQ05mN0FzZ1NMeVFnRjRyQ0FLZWZKbW80dDk2SVhITmEybEdndm44cjZ5N05zU0ZrMFplbU41aVBycjhkdFJ6CkNjTFlGSlcySGI4YTBNeDhqbDNWbmE4VHBnb0U3ZnMzZ2Rwd1EyL0ZCdTJlK2ZNS0VSczQ4bFJVNHV4WGg3L1EKQzdyYmpGbnBTRnovYi90T0ppZlpwV3k5MGNRTVRLQmZOemRWRUUwZzlGZ2tHbHJBZjdRdmU0Y1M2Y0xZUEVwdQpOZTc3bTBtT3lYUHZWa3hYU1lLc3VEcGF0QmxJZmh5bTFnM2o2dnptOFMySHJIVTAvelZURnQrRDBCNlUKLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
token: a9302e29cbec7ac082fe8515c3bd84f1
isaac@Azure:~/clouddrive/terraform-aks-k8s$
isaac@Azure:~/clouddrive/terraform-aks-k8s$
That's it! so let's test it...
Testing
We can do it on the azure cloud shell:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ echo "$(terraform output kube_config)" > ./azurek8s
isaac@Azure:~/clouddrive/terraform-aks-k8s$ export KUBECONFIG=./azurek8s
isaac@Azure:~/clouddrive/terraform-aks-k8s$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-79c89b8f4-2qkxl 1/1 Running 0 19m
kube-system coredns-79c89b8f4-rvl28 1/1 Running 0 10m
kube-system coredns-autoscaler-6fcdb7d64-m8pjg 1/1 Running 0 18m
kube-system heapster-6d879b9dc8-m8bhv 2/2 Running 0 11m
kube-system kube-proxy-g6l92 1/1 Running 0 14m
kube-system kube-proxy-htcwh 1/1 Running 0 14m
kube-system kube-proxy-wxh2z 1/1 Running 0 14m
kube-system kube-svc-redirect-gq6gf 2/2 Running 0 14m
kube-system kube-svc-redirect-khvls 2/2 Running 0 14m
kube-system kube-svc-redirect-zqktm 2/2 Running 0 14m
kube-system kubernetes-dashboard-dfbbfd8-f8zdj 1/1 Running 0 19m
kube-system metrics-server-7b97f9cd9-cnlj5 1/1 Running 1 18m
kube-system omsagent-5llv6 1/1 Running 1 14m
kube-system omsagent-78qzb 1/1 Running 1 14m
kube-system omsagent-rs-664c86dd9f-9h4jn 1/1 Running 0 19m
kube-system omsagent-tcngt 1/1 Running 1 14m
kube-system tunnelfront-78f7fdcf7c-4bcnc 1/1 Running 0 19m
But we can also run the same locally using the AKS we already installed before - just copy manually the kubeconfig into ~/.kube:
C:\Users\johnsi10\azure-devops-node-api>vim C:\Users\johnsi10\.kube\config
C:\Users\johnsi10\azure-devops-node-api>
C:\Users\johnsi10\azure-devops-node-api>kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-79c89b8f4-2qkxl 1/1 Running 0 18m
kube-system coredns-79c89b8f4-rvl28 1/1 Running 0 9m38s
kube-system coredns-autoscaler-6fcdb7d64-m8pjg 1/1 Running 0 18m
kube-system heapster-6d879b9dc8-m8bhv 2/2 Running 0 10m
kube-system kube-proxy-g6l92 1/1 Running 0 14m
kube-system kube-proxy-htcwh 1/1 Running 0 14m
kube-system kube-proxy-wxh2z 1/1 Running 0 14m
kube-system kube-svc-redirect-gq6gf 2/2 Running 0 14m
kube-system kube-svc-redirect-khvls 2/2 Running 0 14m
kube-system kube-svc-redirect-zqktm 2/2 Running 0 14m
kube-system kubernetes-dashboard-dfbbfd8-f8zdj 1/1 Running 0 18m
kube-system metrics-server-7b97f9cd9-cnlj5 1/1 Running 1 18m
kube-system omsagent-5llv6 1/1 Running 1 14m
kube-system omsagent-78qzb 1/1 Running 1 14m
kube-system omsagent-rs-664c86dd9f-9h4jn 1/1 Running 0 18m
kube-system omsagent-tcngt 1/1 Running 1 14m
kube-system tunnelfront-78f7fdcf7c-4bcnc 1/1 Running 0 18m
Cleaning up:
We can destroy that which Terraform created (all but the storage account) with a plan and destroy:
isaac@Azure:~/clouddrive/terraform-aks-k8s$ terraform plan -destroy -out destroy-out.plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
azurerm_resource_group.k8s: Refreshing state... (ID: /subscriptions/fefd6730-7b26-49a2-8c24-...c9b1c/resourceGroups/idj-azure-k8stest)
azurerm_log_analytics_workspace.test: Refreshing state... (ID: /subscriptions/fefd6730-7b26-49a2-8c24-...aces/idj-testloganalyticsworkspacename)
azurerm_kubernetes_cluster.k8s: Refreshing state... (ID: /subscriptions/fefd6730-7b26-49a2-8c24-...nerService/managedClusters/idj-k8stest)
azurerm_log_analytics_solution.test: Refreshing state... (ID: /subscriptions/fefd6730-7b26-49a2-8c24-...hts(idj-testLogAnalyticsWorkspaceName))
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
- azurerm_kubernetes_cluster.k8s
- azurerm_log_analytics_solution.test
- azurerm_log_analytics_workspace.test
- azurerm_resource_group.k8s
Plan: 0 to add, 0 to change, 4 to destroy.
------------------------------------------------------------------------
This plan was saved to: destroy-out.plan
To perform exactly these actions, run the following command to apply:
terraform apply "destroy-out.plan"
We can also destroy it from the resource group in the azure portal:
ARM Template
Another way, and the 4th way we’ll cover is to use an ARM template.
The easiest way to accomplish this is to use the wizard in the portal, then save the template to use:
First, create a new Kubernetes Service via Create a Resource :
Go through the wizard, but at the end, instead of clicking Create, choose to Download the template:
From there, we can click the Download link to download a zip of the template:
Now head into that directory after you expand the zip and we have a couple things to do before we can run it:
Login and double check our subscription id
az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all the subscriptions to which you have access...
[
{
"cloudName": "AzureCloud",
"id": "abcd1234-abab-abab-dcdc-dcdc123456",
"isDefault": true,
"name": "AHEAD-Azure-Lab",
"state": "Enabled",
"tenantId": "ef12e89-33ee-44dd-55cc-abab12434",
"user": {
"name": "Isaac.Johnson@thinkahead.com",
"type": "user"
}
}
]
AHD-MBP13-048:azure-devops-node-api isaac.johnson$ az account show --out json
{
"environmentName": "AzureCloud",
"id": "abcd1234-abab-abab-dcdc-dcdc123456",
"isDefault": true,
"name": "AHEAD-Azure-Lab",
"state": "Enabled",
"tenantId": "ef12e89-33ee-44dd-55cc-abab12434",
"user": {
"name": "Isaac.Johnson@thinkahead.com",
"type": "user"
}
}
Create a resource group
Edit parameter.json and add our SP id and secret
Launch the deployment:
./deploy.sh
Your subscription ID can be looked up with the CLI using: az account show --out json
Enter your subscription ID:
abcd1234-abab-abab-dcdc-dcdc123456
This script will look for an existing resource group, otherwise a new one will be created
You can create new resource groups with the CLI using: az group create
Enter a resource group name
idjaksarmrg
Enter a name for this deployment:
my-arm-aks-deployment
If creating a *new* resource group, you need to set a location
You can lookup locations with the CLI using: az account list-locations
Enter resource group location:
centralus
…
FYI: in my first try, i had a former (and errant) resource group breaking deploy:
+ az group deployment create --name my-arm-aks-deployment --resource-group idjaksarmrg --template-file template.json --parameters @parameters.json
Azure Error: InvalidTemplate
Message: Deployment template validation failed: 'The resource 'Microsoft.ContainerService/managedClusters/idj-arm-aks-service' is not defined in the template. Please see https://aka.ms/arm-template for usage details.'.
I just opened up template.json and replaced all “idj-arm-aks-service” (the bad resource group name) with “idjaksarmrg” and saved it.. Seemed to fix my issues
Summary
Deploying kubernetes can be accomplished in a variety of ways including CLI, Portal, ARM Templates, Kubespray and Terraform and they all have a various merits. For instance, ARM, CLI and Portal will create a native AKV cluster that can use autoscaling. Terraform and Kubespray can create a relatively platform-independant cluster in Azure, AWS or any other provider (including on-prem).