Azure Arc for Kubernetes
Azure Arc for Kuberentes has finally gone out to public preview. Like Arc for Servers, Arc for Kubernetes let’s us add our external kubernetes clusters, be them on-prem or in other clouds into Azure to be managed with Policies, RBAC and GitOps deployment.
What I hope to discover is:
- What kind of clusters will it support? Old versions/new versions of k8s? ARM?
- Can i use Azure Pipelines Kubernetes Tasks with these?
Creating an CIVO K8s cluster
First, let’s get the CIVO client. We need to install Ruby if we don’t have it already:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ sudo apt update && sudo apt install ruby-full
[sudo] password for builder:
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 https://packages.microsoft.com/repos/azure-cli bionic InRelease [3965 B]
Get:4 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease [4003 B]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:6 https://packages.microsoft.com/repos/azure-cli bionic/main amd64 Packages [9314 B]
Get:7 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 Packages [112 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [717 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [947 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [227 kB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [41.9 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [322 kB]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [10.5 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [54.9 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [665 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [13.7 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1075 kB]
Get:19 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [221 kB]
Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [7596 B]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [334 kB]
Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2824 B]
Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [15.7 kB]
Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6384 B]
Get:25 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [7516 B]
Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/main Translation-en [4764 B]
Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [7484 B]
Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [4436 B]
Fetched 5068 kB in 3s (1856 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
69 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
fonts-lato javascript-common libgmp-dev libgmpxx4ldbl libjs-jquery libruby2.5 rake ri ruby ruby-dev ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit
ruby2.5 ruby2.5-dev ruby2.5-doc rubygems-integration
Suggested packages:
apache2 | lighttpd | httpd gmp-doc libgmp10-doc libmpfr-dev bundler
The following NEW packages will be installed:
fonts-lato javascript-common libgmp-dev libgmpxx4ldbl libjs-jquery libruby2.5 rake ri ruby ruby-dev ruby-did-you-mean ruby-full ruby-minitest ruby-net-telnet ruby-power-assert
ruby-test-unit ruby2.5 ruby2.5-dev ruby2.5-doc rubygems-integration
0 upgraded, 20 newly installed, 0 to remove and 69 not upgraded.
Need to get 8208 kB/8366 kB of archives.
After this operation, 48.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 fonts-lato all 2.0-2 [2698 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgmpxx4ldbl amd64 2:6.1.2+dfsg-2 [8964 B]
Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgmp-dev amd64 2:6.1.2+dfsg-2 [316 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 rubygems-integration all 1.11 [4994 B]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ruby2.5 amd64 2.5.1-1ubuntu1.6 [48.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby amd64 1:2.5.1 [5712 B]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 rake all 12.3.1-1ubuntu0.1 [44.9 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-did-you-mean all 1.2.0-2 [9700 B]
Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-minitest all 5.10.3-1 [38.6 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-net-telnet all 0.1.1-2 [12.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-power-assert all 0.3.0-1 [7952 B]
Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-test-unit all 3.2.5-1 [61.1 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libruby2.5 amd64 2.5.1-1ubuntu1.6 [3069 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ruby2.5-doc all 2.5.1-1ubuntu1.6 [1806 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 ri all 1:2.5.1 [4496 B]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ruby2.5-dev amd64 2.5.1-1ubuntu1.6 [63.7 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-dev amd64 1:2.5.1 [4604 B]
Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 ruby-full all 1:2.5.1 [2716 B]
Fetched 8208 kB in 3s (2550 kB/s)
Selecting previously unselected package fonts-lato.
(Reading database ... 78441 files and directories currently installed.)
Preparing to unpack .../00-fonts-lato_2.0-2_all.deb .................................................................................................................................]
Unpacking fonts-lato (2.0-2) ........................................................................................................................................................]
Selecting previously unselected package javascript-common............................................................................................................................]
Preparing to unpack .../01-javascript-common_11_all.deb .............................................................................................................................]
Unpacking javascript-common (11) ...
Selecting previously unselected package libgmpxx4ldbl:amd64..........................................................................................................................]
Preparing to unpack .../02-libgmpxx4ldbl_2%3a6.1.2+dfsg-2_amd64.deb .................................................................................................................]
Unpacking libgmpxx4ldbl:amd64 (2:6.1.2+dfsg-2) ......................................................................................................................................]
Selecting previously unselected package libgmp-dev:amd64.............................................................................................................................]
Preparing to unpack .../03-libgmp-dev_2%3a6.1.2+dfsg-2_amd64.deb ...
Unpacking libgmp-dev:amd64 (2:6.1.2+dfsg-2) .........................................................................................................................................]
Selecting previously unselected package libjs-jquery.
Preparing to unpack .../04-libjs-jquery_3.2.1-1_all.deb .............................................................................................................................]
Unpacking libjs-jquery (3.2.1-1) ....................................................................................................................................................]
Selecting previously unselected package rubygems-integration.
Preparing to unpack .../05-rubygems-integration_1.11_all.deb ........................................................................................................................]
Unpacking rubygems-integration (1.11) ...
Selecting previously unselected package ruby2.5......................................................................................................................................]
Preparing to unpack .../06-ruby2.5_2.5.1-1ubuntu1.6_amd64.deb ...
Unpacking ruby2.5 (2.5.1-1ubuntu1.6) ................................................................................................................................................]
Selecting previously unselected package ruby.
Preparing to unpack .../07-ruby_1%3a2.5.1_amd64.deb .................................................................................................................................]
Unpacking ruby (1:2.5.1) ...
Selecting previously unselected package rake.
Preparing to unpack .../08-rake_12.3.1-1ubuntu0.1_all.deb ...........................................................................................................................]
Unpacking rake (12.3.1-1ubuntu0.1) ...
Selecting previously unselected package ruby-did-you-mean............................................................................................................................]
Preparing to unpack .../09-ruby-did-you-mean_1.2.0-2_all.deb ........................................................................................................................]
Unpacking ruby-did-you-mean (1.2.0-2) ...
Selecting previously unselected package ruby-minitest................................................................................................................................]
Preparing to unpack .../10-ruby-minitest_5.10.3-1_all.deb ...........................................................................................................................]
Unpacking ruby-minitest (5.10.3-1) ......................................................................................................................................................]
Selecting previously unselected package ruby-net-telnet.
Preparing to unpack .../11-ruby-net-telnet_0.1.1-2_all.deb ...
Unpacking ruby-net-telnet (0.1.1-2) ...
Selecting previously unselected package ruby-power-assert.
Preparing to unpack .../12-ruby-power-assert_0.3.0-1_all.deb ...
Unpacking ruby-power-assert (0.3.0-1) ...
Selecting previously unselected package ruby-test-unit.
Preparing to unpack .../13-ruby-test-unit_3.2.5-1_all.deb ...
Unpacking ruby-test-unit (3.2.5-1) ...
Selecting previously unselected package libruby2.5:amd64.
Preparing to unpack .../14-libruby2.5_2.5.1-1ubuntu1.6_amd64.deb ...
Unpacking libruby2.5:amd64 (2.5.1-1ubuntu1.6) .........................................................................................................................................]
Selecting previously unselected package ruby2.5-doc.
Preparing to unpack .../15-ruby2.5-doc_2.5.1-1ubuntu1.6_all.deb ...
Unpacking ruby2.5-doc (2.5.1-1ubuntu1.6) ...
Selecting previously unselected package ri.
Preparing to unpack .../16-ri_1%3a2.5.1_all.deb ...
Unpacking ri (1:2.5.1) ...
Selecting previously unselected package ruby2.5-dev:amd64.
Preparing to unpack .../17-ruby2.5-dev_2.5.1-1ubuntu1.6_amd64.deb ...
Unpacking ruby2.5-dev:amd64 (2.5.1-1ubuntu1.6) ...
Selecting previously unselected package ruby-dev:amd64.
Preparing to unpack .../18-ruby-dev_1%3a2.5.1_amd64.deb ...
Unpacking ruby-dev:amd64 (1:2.5.1) ...
Selecting previously unselected package ruby-full.
Preparing to unpack .../19-ruby-full_1%3a2.5.1_all.deb ...
Unpacking ruby-full (1:2.5.1) ...
Setting up libjs-jquery (3.2.1-1) ...
Setting up fonts-lato (2.0-2) ...
Setting up ruby-did-you-mean (1.2.0-2) ...
Setting up ruby-net-telnet (0.1.1-2) ...
Setting up rubygems-integration (1.11) ...
Setting up ruby2.5-doc (2.5.1-1ubuntu1.6) ...
Setting up javascript-common (11) ...
Setting up libgmpxx4ldbl:amd64 (2:6.1.2+dfsg-2) ...
Setting up ruby-minitest (5.10.3-1) ...
Setting up ruby-power-assert (0.3.0-1) ...
Setting up libgmp-dev:amd64 (2:6.1.2+dfsg-2) ...
Setting up ruby2.5 (2.5.1-1ubuntu1.6) ...
Setting up ri (1:2.5.1) ...
Setting up ruby (1:2.5.1) ...
Setting up ruby-test-unit (3.2.5-1) ...
Setting up rake (12.3.1-1ubuntu0.1) ...
Setting up libruby2.5:amd64 (2.5.1-1ubuntu1.6) ...
Setting up ruby2.5-dev:amd64 (2.5.1-1ubuntu1.6) ...
Setting up ruby-dev:amd64 (1:2.5.1) ...
Setting up ruby-full (1:2.5.1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...Next let’s install the CIVO client:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ sudo gem install civo_cli
Fetching: unicode-display_width-1.7.0.gem (100%).......................................................................................................................................]
Successfully installed unicode-display_width-1.7.0
Fetching: terminal-table-1.8.0.gem (100%)
Successfully installed terminal-table-1.8.0
Fetching: thor-1.0.1.gem (100%)
Successfully installed thor-1.0.1
Fetching: colorize-0.8.1.gem (100%)
Successfully installed colorize-0.8.1
Fetching: bundler-2.1.4.gem (100%)
Successfully installed bundler-2.1.4
Fetching: mime-types-data-3.2020.0512.gem (100%)
Successfully installed mime-types-data-3.2020.0512
Fetching: mime-types-3.3.1.gem (100%)
Successfully installed mime-types-3.3.1
Fetching: multi_json-1.14.1.gem (100%)
Successfully installed multi_json-1.14.1
Fetching: safe_yaml-1.0.5.gem (100%)
Successfully installed safe_yaml-1.0.5
Fetching: crack-0.4.3.gem (100%)
Successfully installed crack-0.4.3
Fetching: multipart-post-2.1.1.gem (100%)
Successfully installed multipart-post-2.1.1
Fetching: faraday-1.0.1.gem (100%)
Successfully installed faraday-1.0.1
Fetching: concurrent-ruby-1.1.6.gem (100%)
Successfully installed concurrent-ruby-1.1.6
Fetching: i18n-1.8.2.gem (100%)
HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.
But that may break your application.
If you are upgrading your Rails application from an older version of Rails:
Please check your Rails app for 'config.i18n.fallbacks = true'.
If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be
'config.i18n.fallbacks = [I18n.default_locale]'.
If not, fallbacks will be broken in your app by I18n 1.1.x.
If you are starting a NEW Rails application, you can ignore this notice.
For more info see:
https://github.com/svenfuchs/i18n/releases/tag/v1.1.0
Successfully installed i18n-1.8.2
Fetching: thread_safe-0.3.6.gem (100%)
Successfully installed thread_safe-0.3.6
Fetching: tzinfo-1.2.7.gem (100%)
Successfully installed tzinfo-1.2.7
Fetching: zeitwerk-2.3.0.gem (100%)
Successfully installed zeitwerk-2.3.0
Fetching: activesupport-6.0.3.1.gem (100%)
Successfully installed activesupport-6.0.3.1
Fetching: flexirest-1.9.15.gem (100%)
Successfully installed flexirest-1.9.15
Fetching: parslet-1.8.2.gem (100%)
Successfully installed parslet-1.8.2
Fetching: toml-0.2.0.gem (100%)
Successfully installed toml-0.2.0
Fetching: highline-2.0.3.gem (100%)
Successfully installed highline-2.0.3
Fetching: commander-4.5.2.gem (100%)
Successfully installed commander-4.5.2
Fetching: civo-1.3.3.gem (100%)
Successfully installed civo-1.3.3
Fetching: civo_cli-0.5.9.gem (100%)
Successfully installed civo_cli-0.5.9
Parsing documentation for unicode-display_width-1.7.0
Installing ri documentation for unicode-display_width-1.7.0
Parsing documentation for terminal-table-1.8.0
Installing ri documentation for terminal-table-1.8.0
Parsing documentation for thor-1.0.1
Installing ri documentation for thor-1.0.1
Parsing documentation for colorize-0.8.1
Installing ri documentation for colorize-0.8.1
Parsing documentation for bundler-2.1.4
Installing ri documentation for bundler-2.1.4
Parsing documentation for mime-types-data-3.2020.0512
Installing ri documentation for mime-types-data-3.2020.0512
Parsing documentation for mime-types-3.3.1
Installing ri documentation for mime-types-3.3.1
Parsing documentation for multi_json-1.14.1
Installing ri documentation for multi_json-1.14.1
Parsing documentation for safe_yaml-1.0.5
Installing ri documentation for safe_yaml-1.0.5
Parsing documentation for crack-0.4.3
Installing ri documentation for crack-0.4.3
Parsing documentation for multipart-post-2.1.1
Installing ri documentation for multipart-post-2.1.1
Parsing documentation for faraday-1.0.1
Installing ri documentation for faraday-1.0.1
Parsing documentation for concurrent-ruby-1.1.6
Installing ri documentation for concurrent-ruby-1.1.6
Parsing documentation for i18n-1.8.2
Installing ri documentation for i18n-1.8.2
Parsing documentation for thread_safe-0.3.6
Installing ri documentation for thread_safe-0.3.6
Parsing documentation for tzinfo-1.2.7
Installing ri documentation for tzinfo-1.2.7
Parsing documentation for zeitwerk-2.3.0
Installing ri documentation for zeitwerk-2.3.0
Parsing documentation for activesupport-6.0.3.1
Installing ri documentation for activesupport-6.0.3.1
Parsing documentation for flexirest-1.9.15
Installing ri documentation for flexirest-1.9.15
Parsing documentation for parslet-1.8.2
Installing ri documentation for parslet-1.8.2
Parsing documentation for toml-0.2.0
Installing ri documentation for toml-0.2.0
Parsing documentation for highline-2.0.3
Installing ri documentation for highline-2.0.3
Parsing documentation for commander-4.5.2
Installing ri documentation for commander-4.5.2
Parsing documentation for civo-1.3.3
Installing ri documentation for civo-1.3.3
Parsing documentation for civo_cli-0.5.9
Installing ri documentation for civo_cli-0.5.9
Done installing documentation for unicode-display_width, terminal-table, thor, colorize, bundler, mime-types-data, mime-types, multi_json, safe_yaml, crack, multipart-post, faraday, concurrent-ruby, i18n, thread_safe, tzinfo, zeitwerk, activesupport, flexirest, parslet, toml, highline, commander, civo, civo_cli after 21 seconds
25 gems installed
Verification that it’s installed:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo help
Commands:
/var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/shell/basic.rb:399: warning: Insecure world writable dir /mnt/c in PATH, mode 040777
civo apikey # manage API keys stored in the client
civo applications # list and add marketplace applications to Kubernetes clusters. Alias: apps, addons, marketplace, k8s-apps, k3s-apps
civo blueprint # manage blueprints
civo domain # manage DNS domains
civo domainrecord # manage domain name DNS records for a domain
civo firewall # manage firewalls
civo help [COMMAND] # Describe available commands or one specific command
civo instance # manage instances
civo kubernetes # manage Kubernetes. Aliases: k8s, k3s
civo loadbalancer # manage load balancers
civo network # manage networks
civo quota # view the quota for the active account
civo region # manage regions
civo size # manage sizes
civo snapshot # manage snapshots
civo sshkey # manage uploaded SSH keys
civo template # manage templates
civo update # update to the latest Civo CLI
civo version # show the version of Civo CLI used
civo volume # manage volumes
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo version
You are running the current v0.5.9 of Civo CLI
Next we need to login to CIVO and get/set our API key. It’s now shown under Other/Security:
I have since reset it, but as you see, right now it’s listed as: I2fTuGRQYognBHZ0MdXhtJAlPD7ViU5q68yCepzS9OFLasbvWK
We now need to save that in the client:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo apikey add K8s I2fTuGRQYognBHZ0MdXhtJAlPD7ViU5q68yCepzS9OFLasbvWK
Saved the API Key I2fTuGRQYognBHZ0MdXhtJAlPD7ViU5q68yCepzS9OFLasbvWK as K8s
The current API Key is now K8s
We’ll need to refresh our memory on CIVO sizes before moving on:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo sizes
+------------+----------------------------------------------------+-----+----------+-----------+
| Name | Description | CPU | RAM (MB) | Disk (GB) |
+------------+----------------------------------------------------+-----+----------+-----------+
| g2.xsmall | Extra Small - 1GB RAM, 1 CPU Core, 25GB SSD Disk | 1 | 1024 | 25 |
| g2.small | Small - 2GB RAM, 1 CPU Core, 25GB SSD Disk | 1 | 2048 | 25 |
| g2.medium | Medium - 4GB RAM, 2 CPU Cores, 50GB SSD Disk | 2 | 4096 | 50 |
| g2.large | Large - 8GB RAM, 4 CPU Cores, 100GB SSD Disk | 4 | 8192 | 100 |
| g2.xlarge | Extra Large - 16GB RAM, 6 CPU Core, 150GB SSD Disk | 6 | 16386 | 150 |
| g2.2xlarge | 2X Large - 32GB RAM, 8 CPU Core, 200GB SSD Disk | 8 | 32768 | 200 |
+------------+----------------------------------------------------+-----+----------+-----------+
And the costs haven’t changed since the last time we blogged:
Lastly we’ll pick a version. Since our last post, they’ve added 1.17.2 in preview:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo kubernetes versions
+-------------+-------------+---------+
| Version | Type | Default |
+-------------+-------------+---------+
| 1.17.2+k3s1 | development | |
| 1.0.0 | stable | <===== |
| 0.10.2 | deprecated | |
| 0.10.0 | deprecated | |
| 0.9.1 | deprecated | |
| 0.8.1 | legacy | |
+-------------+-------------+---------+
And like before, we can review applications we can install at the same time:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo applications list
+----------------------+-------------+--------------+-----------------+--------------+
| Name | Version | Category | Plans | Dependencies |
+----------------------+-------------+--------------+-----------------+--------------+
| cert-manager | v0.11.0 | architecture | Not applicable | Helm |
| Helm | 2.14.3 | management | Not applicable | |
| Jenkins | 2.190.1 | ci_cd | 5GB, 10GB, 20GB | Longhorn |
| KubeDB | v0.12.0 | database | Not applicable | Longhorn |
| Kubeless | 1.0.5 | architecture | Not applicable | |
| kubernetes-dashboard | v2.0.0-rc7 | management | Not applicable | |
| Linkerd | 2.5.0 | architecture | Not applicable | |
| Longhorn | 0.7.0 | storage | Not applicable | |
| Maesh | Latest | architecture | Not applicable | Helm |
| MariaDB | 10.4.7 | database | 5GB, 10GB, 20GB | Longhorn |
| metrics-server | (default) | architecture | Not applicable | |
| MinIO | 2019-08-29 | storage | 5GB, 10GB, 20GB | Longhorn |
| MongoDB | 4.2.0 | database | 5GB, 10GB, 20GB | Longhorn |
| OpenFaaS | 0.18.0 | architecture | Not applicable | Helm |
| Portainer | beta | management | Not applicable | |
| PostgreSQL | 11.5 | database | 5GB, 10GB, 20GB | Longhorn |
| prometheus-operator | 0.35.0 | monitoring | Not applicable | |
| Rancher | v2.3.0 | management | Not applicable | |
| Redis | 3.2 | database | Not applicable | |
| Selenium | 3.141.59-r1 | ci_cd | Not applicable | |
| Traefik | (default) | architecture | Not applicable | |
+----------------------+-------------+--------------+-----------------+--------------+
I notice since February they dropped Klum but added Portainer and upgraded prometheus to 0.35.0 (from 0.34.0).
The last step is to create our cluster:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo kubernetes create MyNextCluster --size g2.medium --applications=Kubernetes-dashboard --nodes=2 --wait=true --save
Building new Kubernetes cluster MyNextCluster: Done
Created Kubernetes cluster MyNextCluster in 02 min 01 sec
KUBECONFIG=~/.kube/config:/tmp/import_kubeconfig20200520-3031-r5ix5y kubectl config view --flatten
/var/lib/gems/2.5.0/gems/civo_cli-0.5.9/lib/kubernetes.rb:350: warning: Insecure world writable dir /mnt/c in PATH, mode 040777
Saved config to ~/.kube/config
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-node-8727 Ready <none> 22s v1.16.3-k3s.2
kube-master-e748 Ready master 33s v1.16.3-k3s.2
Azure Setup
First, login to Azure..
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code FF64MDUEP to authenticate.
…
Create a Resource Group we’ll use to contain our clusters:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az group create --name myArcClusters --location centralus
{
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters",
"location": "centralus",
"managedBy": null,
"name": "myArcClusters",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}
Next, let’s create a Service Principal for Arc
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az ad sp create-for-RBAC --skip-assignment --name "https://azure-arc-for-k8s-onboarding"
{
"appId": "9f7e213f-3b03-411e-b874-ab272e46de41",
"displayName": "azure-arc-for-k8s-onboarding",
"name": "https://azure-arc-for-k8s-onboarding",
"password": "4545458-aaee-3344-4321-12345678ad",
"tenant": "d73a39db-6eda-495d-8000-7579f56d68b7"
}
Give it access to our subscription:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az role assignment create --role 34e09817-6cbe-4d01-b1a2-e0eac5743d41 --a
ssignee 9f7e213f-3b03-411e-b874-ab272e46de41 --scope /subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8
{
"canDelegate": null,
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/providers/Microsoft.Authorization/roleAssignments/c2cc591e-adca-4f91-af83-bdcea56a56f2",
"name": "c2cc591e-adca-4f91-af83-bdcea56a56f2",
"principalId": "e3349b46-3444-4f4a-b9f8-c36a8cf0bf38",
"principalType": "ServicePrincipal",
"roleDefinitionId": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/providers/Microsoft.Authorization/roleDefinitions/34e09817-6cbe-4d01-b1a2-e0eac5743d41",
"scope": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8",
"type": "Microsoft.Authorization/roleAssignments"
}
Next we need to login:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az login --service-principal -u 9f7e213f-3b03-411e-b874-ab272e46de41 -p 4545458-aaee-3344-4321-12345678ad --tenant d73a39db-6eda-495d-8000-7579f56d68b7
[
{
"cloudName": "AzureCloud",
"homeTenantId": "d73a39db-6eda-495d-8000-7579f56d68b7",
"id": "70b42e6a-asdf-asdf-asdf-9f3995b1aca8",
"isDefault": true,
"managedByTenants": [],
"name": "Visual Studio Enterprise Subscription",
"state": "Enabled",
"tenantId": "d73a39db-6eda-495d-8000-7579f56d68b7",
"user": {
"name": "9f7e213f-3b03-411e-b874-ab272e46de41",
"type": "servicePrincipal"
}
}
]
If your Azure CLI is out of date, you may get an error:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az connectedk8s connect -n myCIVOCluster -g myArcClusters
az: 'connectedk8s' is not in the 'az' command group. See 'az --help'. If the command is from an extension, please make sure the corresponding extension is installed. To learn more about extensions, please visit https://docs.microsoft.com/en-us/cli/azure/azure-cli-extensions-overview
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az version
This command is in preview. It may be changed/removed in a future release.
{
"azure-cli": "2.3.1",
"azure-cli-command-modules-nspkg": "2.0.3",
"azure-cli-core": "2.3.1",
"azure-cli-nspkg": "3.0.4",
"azure-cli-telemetry": "1.0.4",
"extensions": {
"azure-devops": "0.18.0"
}
}
Updating is easy:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ sudo apt-get update && sudo apt-get install azure-cli
[sudo] password for builder:
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 https://packages.microsoft.com/repos/azure-cli bionic InRelease
Hit:4 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [334 kB]
Fetched 586 kB in 1s (396 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
azure-cli
1 upgraded, 0 newly installed, 0 to remove and 68 not upgraded.
Need to get 47.7 MB of archives.
After this operation, 23.5 MB of additional disk space will be used.
Get:1 https://packages.microsoft.com/repos/azure-cli bionic/main amd64 azure-cli all 2.6.0-1~bionic [47.7 MB]
Fetched 47.7 MB in 1s (32.0 MB/s)
(Reading database ... 94395 files and directories currently installed.)
Preparing to unpack .../azure-cli_2.6.0-1~bionic_all.deb ...
Unpacking azure-cli (2.6.0-1~bionic) over (2.3.1-1~bionic) ...
Setting up azure-cli (2.6.0-1~bionic) ...
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az version
{
"azure-cli": "2.6.0",
"azure-cli-command-modules-nspkg": "2.0.3",
"azure-cli-core": "2.6.0",
"azure-cli-nspkg": "3.0.4",
"azure-cli-telemetry": "1.0.4",
"extensions": {
"azure-devops": "0.18.0"
}
}
We need to register two providers.. Note, for me, i had permission issues so i had to elevate the role of the SP in my Subscription to get it done:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az provider register --namespace Microsoft.Kubernetes
Registering is still on-going. You can monitor using 'az provider show -n Microsoft.Kubernetes'
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az provider register --namespace Microsoft.KubernetesConfiguration
Registering is still on-going. You can monitor using 'az provider show -n Microsoft.KubernetesConfiguration'
Verification
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az provider show -n Microsoft.Kubernetes -o table
Namespace RegistrationPolicy RegistrationState
-------------------- -------------------- -------------------
Microsoft.Kubernetes RegistrationRequired Registered
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az provider show -n Microsoft.KubernetesConfiguration -o table
Namespace RegistrationPolicy RegistrationState
--------------------------------- -------------------- -------------------
Microsoft.KubernetesConfiguration RegistrationRequired Registered
Install and update to the latest version of the extensions:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az extension add --name connectedk8s
The installed extension 'connectedk8s' is in preview.
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az extension add --name k8sconfiguration
The installed extension 'k8sconfiguration' is in preview.
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az extension update --name connectedk8s
No updates available for 'connectedk8s'. Use --debug for more information.
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az extension update --name k8sconfiguration
No updates available for 'k8sconfiguration'. Use --debug for more information.
Onboarding to Arc
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az connectedk8s connect -n myCIVOCluster -g myArcClusters
Command group 'connectedk8s' is in preview. It may be changed/removed in a future release.
Ensure that you have the latest helm version installed before proceeding.
This operation might take a while...
Connected cluster resource creation is supported only in the following locations: eastus, westeurope. Use the --location flag to specify one of these locations.
That would have been good to know.. Lets create a new Group in eastus and do it again…
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az group create --name myArcClusters2 --location eastus
{
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters2",
"location": "eastus",
"managedBy": null,
"name": "myArcClusters2",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}
Let’s add it!
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az connectedk8s connect -n myCIVOCluster -g myArcClusters2
Command group 'connectedk8s' is in preview. It may be changed/removed in a future release.
Ensure that you have the latest helm version installed before proceeding.
This operation might take a while...
{- Finished ..
"aadProfile": {
"clientAppId": "",
"serverAppId": "",
"tenantId": ""
},
"agentPublicKeyCertificate": "MIICCgKCAgEAsux97vWSl97JnRSsED6UyXq7TtIOBSR/eSy8fGAfGHL1wUCYQQeYI+8bgqSN0n7fvIELalrQ8efvCLdNE3C8ocBC/0RpfjRrIpdg7LPXrCCAA1hvR2PymNMWSyAJcPjyCPu5jbKkRSj4XSSLk1sVgVeJpWXjl4ZKZluwVEGD1EWA3Xwd43fwI8gYNKu55qpyO/knUJngoMV3jwnE0UJuEMzbVyoeKWNQJJe4g0vflA5j4cvU7QYBW6XRGGKiFYgWq1053kzvVuucZIl0QPt9hZU8NbbMnU/b8NPLIue4S/w7cZobpYf46CV2ZzySZ0OIe+kcZ7T8v0yw+i+77C76OcDwDaojov+DBEMLJ7AJtoQ/dwf9JebVO92ZuWFtHbdT3M3222xF2R1KWGVXxmOnqEtB3D+zsNpcSp8abKOH8du9l07Jts1ss2j24Lqf/QFVDrBHUTxidvLENFSCh4qQ2npfJq/v/ndLUv4qPXyRmbNLiFn5/+4shhSQBPBj+GopzMdkZn6Zoh4uPGX77svBS5jmBjyDYj0OIJobt/cfPwQzn9iV3/t2ED7IHauT5WGrGPeqFyF/43YAkanPOQUWDaWy2YLHyrR12Hh2KHLfVq/9k9VvwlcJ0Md96QOhqicBqGDZRR9YDoQjVkOP1Gb11AStvvMmeumWfeCqkVsBy9ECAwEAAQ==",
"agentVersion": null,
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters2/providers/Microsoft.Kubernetes/connectedClusters/myCIVOCluster",
"identity": {
"principalId": "23947e5a-0b1b-45fc-a265-7cad37625959",
"tenantId": "d73a39db-6eda-495d-8000-7579f56d68b7",
"type": "SystemAssigned"
},
"kubernetesVersion": null,
"location": "eastus",
"name": "myCIVOCluster",
"provisioningState": "Succeeded",
"resourceGroup": "myArcClusters2",
"tags": null,
"totalNodeCount": null,
"type": "Microsoft.Kubernetes/connectedClusters"
}
The details are missing until we add in at least one tag in Tags.
We can see the pods are running fine:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get pods -n azure-arc
NAME READY STATUS RESTARTS AGE
flux-logs-agent-6989466ff6-f7znc 2/2 Running 0 20m
metrics-agent-bb65d876-gh7bd 2/2 Running 0 20m
clusteridentityoperator-5649f66cf8-spngt 3/3 Running 0 20m
config-agent-dcf745b57-lsnj7 3/3 Running 0 20m
controller-manager-98947d4f6-lg2hw 3/3 Running 0 20m
resource-sync-agent-75f6885587-d76mf 3/3 Running 0 20m
cluster-metadata-operator-5959f77f6d-7hqtf 2/2 Running 0 20m
Pods look good
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get deployments -n azure-arc
NAME READY UP-TO-DATE AVAILABLE AGE
flux-logs-agent 1/1 1 1 30m
metrics-agent 1/1 1 1 30m
clusteridentityoperator 1/1 1 1 30m
config-agent 1/1 1 1 30m
controller-manager 1/1 1 1 30m
resource-sync-agent 1/1 1 1 30m
cluster-metadata-operator 1/1 1 1 30m
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
azure-arc default 1 2020-05-20 08:45:13.2639004 -0500 CDT deployed azure-arc-k8sagents-0.2.2 1.0
Let’s stop here, and tear down this cluster.
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo kubernetes list
+--------------------------------------+---------------+---------+-----------+---------+--------+
| ID | Name | # Nodes | Size | Version | Status |
+--------------------------------------+---------------+---------+-----------+---------+--------+
| 88f155ad-08e2-4935-8f2a-788deebaa451 | MyNextCluster | 2 | g2.medium | 1.0.0 | ACTIVE |
+--------------------------------------+---------------+---------+-----------+---------+--------+
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ civo kubernetes remove 88f155ad-08e2-4935-8f2a-788deebaa451
Removing Kubernetes cluster MyNextCluster
*update* the problem was not having Tags. For some reason, Arc for K8s needs at least 1 tag to function properly. Once i set a tag (i used cloudProvider = CIVO) then it loaded just fine
AWS EKS
Let’s keep it simple and use a Quickstart guide to launch EKS into a new VPC: https://aws.amazon.com/quickstart/architecture/amazon-eks/
I did set the Allow external access to 0.0.0.0/0:
And set EKS public access endpoint to Enabled
We can now see the EKS cluster via the CLI:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ aws eks list-clusters --region us-east-2
{
"clusters": [
"EKS-YVA1LL5Y"
]
}
We can now download the kubeconfig:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ aws --region us-east-2 eks update-kubeconfig --name EKS-YVA1LL5Y
Added new context arn:aws:eks:us-east-2:01234512345:cluster/EKS-YVA1LL5Y to /home/builder/.kube/config
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 73m
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-0-74.us-east-2.compute.internal Ready <none> 58m v1.15.10-eks-bac369
ip-10-0-61-98.us-east-2.compute.internal Ready <none> 58m v1.15.10-eks-bac369
ip-10-0-82-16.us-east-2.compute.internal Ready <none> 58m v1.15.10-eks-bac369
Onboarding to Azure Arc
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az connectedk8s connect -n myEKSCluster -g myArcClusters2
Command group 'connectedk8s' is in preview. It may be changed/removed in a future release.
Ensure that you have the latest helm version installed before proceeding.
This operation might take a while...
{- Finished ..
"aadProfile": {
"clientAppId": "",
"serverAppId": "",
"tenantId": ""
},
"agentPublicKeyCertificate": "MIICCgKCAgEAlIaZLZ0eHd1zwSlXflD8L5tQcH8MAF/t3tHOewN4+6EsVW6wtwHL8rup+1i/1jbnArKHFrMCPpok+JCkxXvTgWsD5VSnZxDg/dkkACHw1I2RTq+9Q7E4SwT9QxreuxTejyLI/0w+kCnNwWlAvS6I4J/M709Bvz6KhxzFCWsX5fgRCoIO5zE0wjlxGG0s4c4f1hmHz3Mb4utPaX4gk6+Bo5MZs5EVNbhuHKdasDvJWe1itrtKVKQh7LfWICb1M34Kmo+TPhADKcoV4Ltaxb3VHnHLZpStGfcFI5SqTppn5dULOcGGob2wG8Mlw2INy17QUsyZ728qIKrBaNaJXgeQmZ93xUh7H0IBjV7VXuCOScsDBP7Cx5KZnXT7uWPRLjWCzYlOodCNeCMnBAabEGPECbUq3juvkmrbYiktjn6ZnWm32bDEYqiobqyFz3XMZVkK1zTnrVY0Bguc+n3RHXQeKKHZLyg84Cm1p83uadKFB7RZdCmBw5a8rS4PJEF+bpgPgS5FEcuNa5qp35BaWKTPifiDx9vo8uIADtEwSqpsrxqDnTf2gm8ORoww3vNfTO9WdiAzpBMDToJQamauVzGLB/HFO7pfrvjzRTdeKjamv8lzI/IyStr6Wf2QHAZVF95XfmmZ0iCHmkVHRwbQE4IABUeyZPzB6VKJD3yC9K8q4EsCAwEAAQ==",
"agentVersion": null,
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters2/providers/Microsoft.Kubernetes/connectedClusters/myEKSCluster",
"identity": {
"principalId": "e5c729a1-ef9c-4839-89cd-a850536a400a",
"tenantId": "d73a39db-6eda-495d-8000-7579f56d68b7",
"type": "SystemAssigned"
},
"kubernetesVersion": null,
"location": "eastus",
"name": "myEKSCluster",
"provisioningState": "Succeeded",
"resourceGroup": "myArcClusters2",
"tags": null,
"totalNodeCount": null,
"type": "Microsoft.Kubernetes/connectedClusters"
}
Linode LKE
Since my last blog entry, LKE CLI now can create clusters.
First, let’s get a list of types:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ linode-cli linodes types
┌──────────────────┬──────────────────────────────────┬───────────┬─────────┬────────┬───────┬─────────────┬──────────┬────────┬─────────┬──────┐
│ id │ label │ class │ disk │ memory │ vcpus │ network_out │ transfer │ hourly │ monthly │ gpus │
├──────────────────┼──────────────────────────────────┼───────────┼─────────┼────────┼───────┼─────────────┼──────────┼────────┼─────────┼──────┤
│ g6-nanode-1 │ Nanode 1GB │ nanode │ 25600 │ 1024 │ 1 │ 1000 │ 1000 │ 0.0075 │ 5.0 │ 0 │
│ g6-standard-1 │ Linode 2GB │ standard │ 51200 │ 2048 │ 1 │ 2000 │ 2000 │ 0.015 │ 10.0 │ 0 │
│ g6-standard-2 │ Linode 4GB │ standard │ 81920 │ 4096 │ 2 │ 4000 │ 4000 │ 0.03 │ 20.0 │ 0 │
│ g6-standard-4 │ Linode 8GB │ standard │ 163840 │ 8192 │ 4 │ 5000 │ 5000 │ 0.06 │ 40.0 │ 0 │
│ g6-standard-6 │ Linode 16GB │ standard │ 327680 │ 16384 │ 6 │ 6000 │ 8000 │ 0.12 │ 80.0 │ 0 │
│ g6-standard-8 │ Linode 32GB │ standard │ 655360 │ 32768 │ 8 │ 7000 │ 16000 │ 0.24 │ 160.0 │ 0 │
│ g6-standard-16 │ Linode 64GB │ standard │ 1310720 │ 65536 │ 16 │ 9000 │ 20000 │ 0.48 │ 320.0 │ 0 │
│ g6-standard-20 │ Linode 96GB │ standard │ 1966080 │ 98304 │ 20 │ 10000 │ 20000 │ 0.72 │ 480.0 │ 0 │
│ g6-standard-24 │ Linode 128GB │ standard │ 2621440 │ 131072 │ 24 │ 11000 │ 20000 │ 0.96 │ 640.0 │ 0 │
│ g6-standard-32 │ Linode 192GB │ standard │ 3932160 │ 196608 │ 32 │ 12000 │ 20000 │ 1.44 │ 960.0 │ 0 │
│ g6-highmem-1 │ Linode 24GB │ highmem │ 20480 │ 24576 │ 1 │ 5000 │ 5000 │ 0.09 │ 60.0 │ 0 │
│ g6-highmem-2 │ Linode 48GB │ highmem │ 40960 │ 49152 │ 2 │ 6000 │ 6000 │ 0.18 │ 120.0 │ 0 │
│ g6-highmem-4 │ Linode 90GB │ highmem │ 92160 │ 92160 │ 4 │ 7000 │ 7000 │ 0.36 │ 240.0 │ 0 │
│ g6-highmem-8 │ Linode 150GB │ highmem │ 204800 │ 153600 │ 8 │ 8000 │ 8000 │ 0.72 │ 480.0 │ 0 │
│ g6-highmem-16 │ Linode 300GB │ highmem │ 348160 │ 307200 │ 16 │ 9000 │ 9000 │ 1.44 │ 960.0 │ 0 │
│ g6-dedicated-2 │ Dedicated 4GB │ dedicated │ 81920 │ 4096 │ 2 │ 4000 │ 4000 │ 0.045 │ 30.0 │ 0 │
│ g6-dedicated-4 │ Dedicated 8GB │ dedicated │ 163840 │ 8192 │ 4 │ 5000 │ 5000 │ 0.09 │ 60.0 │ 0 │
│ g6-dedicated-8 │ Dedicated 16GB │ dedicated │ 327680 │ 16384 │ 8 │ 6000 │ 6000 │ 0.18 │ 120.0 │ 0 │
│ g6-dedicated-16 │ Dedicated 32GB │ dedicated │ 655360 │ 32768 │ 16 │ 7000 │ 7000 │ 0.36 │ 240.0 │ 0 │
│ g6-dedicated-32 │ Dedicated 64GB │ dedicated │ 1310720 │ 65536 │ 32 │ 8000 │ 8000 │ 0.72 │ 480.0 │ 0 │
│ g6-dedicated-48 │ Dedicated 96GB │ dedicated │ 1966080 │ 98304 │ 48 │ 9000 │ 9000 │ 1.08 │ 720.0 │ 0 │
│ g1-gpu-rtx6000-1 │ Dedicated 32GB + RTX6000 GPU x1 │ gpu │ 655360 │ 32768 │ 8 │ 10000 │ 16000 │ 1.5 │ 1000.0 │ 1 │
│ g1-gpu-rtx6000-2 │ Dedicated 64GB + RTX6000 GPU x2 │ gpu │ 1310720 │ 65536 │ 16 │ 10000 │ 20000 │ 3.0 │ 2000.0 │ 2 │
│ g1-gpu-rtx6000-3 │ Dedicated 96GB + RTX6000 GPU x3 │ gpu │ 1966080 │ 98304 │ 20 │ 10000 │ 20000 │ 4.5 │ 3000.0 │ 3 │
│ g1-gpu-rtx6000-4 │ Dedicated 128GB + RTX6000 GPU x4 │ gpu │ 2621440 │ 131072 │ 24 │ 10000 │ 20000 │ 6.0 │ 4000.0 │ 4 │
└──────────────────┴──────────────────────────────────┴───────────┴─────────┴────────┴───────┴─────────────┴──────────┴────────┴─────────┴──────┘
Next, let’s create a cluster:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ linode-cli lke cluster-create --label mycluster --k8s_version 1.15 --node_pools.type g6-standard-2 --node_pools.count 2
┌──────┬───────────┬────────────┐
│ id │ label │ region │
├──────┼───────────┼────────────┤
│ 5193 │ mycluster │ us-central │
└──────┴───────────┴────────────┘
The Kubeconfig view is still screwy so we can use our last hack to get it:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ linode-cli lke kubeconfig-view 5193 > t.o && cat t.o | tail -n2 | head -n1 | sed 's/.\{8\}$//' | sed 's/^.\{8\}//' | base64 --decode > ~/.kube/config
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
lke5193-6544-5ec54610dcac NotReady <none> 2s v1.15.6
lke5193-6544-5ec54610e234 NotReady <none> 9s v1.15.6
Onboarding to Azure Arc
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ az connectedk8s connect -n myLKECluster -g myArcClusters2
Command group 'connectedk8s' is in preview. It may be changed/removed in a future release.
Ensure that you have the latest helm version installed before proceeding.
This operation might take a while...
{- Finished ..
"aadProfile": {
"clientAppId": "",
"serverAppId": "",
"tenantId": ""
},
"agentPublicKeyCertificate": "MIICCgKCAgEAqjk+8jskhpDa+O6cDqL37tad130qQCRBcCPimQ5wExwKPE0+BLrprfDGdXfAEbH9sQXTl6yWiXLiJkrV8PP/D+Qhkze4WINTY0pSu5yRT9x0MO/bu1E7ygcXjxuVvd7fJV0JF5fisWMaB/maY3FtmpoX1zOd0Sbz/yKg3pTrZE8R7QLDuNmKEkbHj5/ZmeZAmGYBUguKpqtnNM7++OE5BKjtHcI5H5A8fEV5xUJEA1+F7TTtrFTrFZdMPUR5QqQ+WQzsN3hrDakW51+SU5IkkppmzORHCehAn9TG/36mD7C8k05h/JKPzyU0UWlLb1yi7wOZqh9FksmEdqix6XcAMQCIp3CzdJc5M8KXa9rUNPmHf+KvPaqQtuZYPUiErMWHMiSqK/arZEj7N4NAToPjIjT1oPHB7g01yQFCmHcgs5gYt7H3b0ZSxWQJ78aZaYVc3HskegmYwL2qsg/P6uOIoS6+SqlPcb4U/ER+J+vKPLm71sOydDG2uMU02JiBJqrOqoxFRFtbrksOcL0zsFe3axjTrEJTz18NIhOJ98Af0en/q5r/h6zG7SR+xp/VkIk81UzYlCKPTMF01lbm1nKtrIdmYchp/ggC21bWqClhmOMPDw0eoiINvYWGoOfsugm0ceEbpxkKEv/avir+Z+krVnOrb/enVP5qli6leZRgSTsCAwEAAQ==",
"agentVersion": null,
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters2/providers/Microsoft.Kubernetes/connectedClusters/myLKECluster",
"identity": {
"principalId": "0f31b18c-743c-4697-9aa2-b1d3585a401c",
"tenantId": "d73a39db-6eda-495d-8000-7579f56d68b7",
"type": "SystemAssigned"
},
"kubernetesVersion": null,
"location": "eastus",
"name": "myLKECluster",
"provisioningState": "Succeeded",
"resourceGroup": "myArcClusters2",
"tags": null,
"totalNodeCount": null,
"type": "Microsoft.Kubernetes/connectedClusters"
}
Once I set a tag in the Azure Portal:
We can now see Kubernetes clusters hosted in two other different Cloud providers now in my Azure Resource Group:
GitOps
First, we need a repo to store our YAML/Helm charts. We can create one in Azure Repos: https://princessking.visualstudio.com/_git/IAC-Demo
I’ll then initialize it with a README.md and Create new GIT Credentials:
We’ll clone the repo then add a directory for manifests.I could create a simple app, but i’ll just borrow from Azure’s Hello World Voting app and put that YAML in my manifests folder.
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ mkdir manifests
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ cd manifests/
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ curl -O https://raw.githubusercontent.com/Azure-Samples/azure-voting-app-redis/master/azure-vote-all-in-one-redis.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 100 1405 100 1405 0 0 9493 0 --:--:-- --:--:-- --:--:-- 9493
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ cat azure-vote-all-in-one-redis.yaml | tail -n5
type: LoadBalancer
ports:
- port: 80
selector:
app: azure-vote-front
We can now launch a configuration that would deploy to the default namespace the YAMLs in manifests.. Now note - i have NOT yet pushed the manifest folder so there is nothing to deploy:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ az k8sconfiguration create --name iam-demo-config --cluster-name myEKSCluster --resource-group myArcClusters2 --operator-instance-name app-config --operator-namespace default --repository-url https://princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo --cluster-type connectedClusters --scope cluster --operator-params="--git-readonly --git-path=manifests"
Command group 'k8sconfiguration' is in preview. It may be changed/removed in a future release.
{
"complianceStatus": {
"complianceState": "Pending",
"lastConfigApplied": "0001-01-01T00:00:00",
"message": "{\"OperatorMessage\":null,\"ClusterState\":null}",
"messageLevel": "3"
},
"enableHelmOperator": "False",
"helmOperatorProperties": null,
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters2/providers/Microsoft.Kubernetes/connectedClusters/myEKSCluster/providers/Microsoft.KubernetesConfiguration/sourceControlConfigurations/iam-demo-config",
"name": "iam-demo-config",
"operatorInstanceName": "app-config",
"operatorNamespace": "default",
"operatorParams": "--git-readonly --git-path=manifests",
"operatorScope": "cluster",
"operatorType": "Flux",
"provisioningState": "Succeeded",
"repositoryPublicKey": "",
"repositoryUrl": "https://princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo",
"resourceGroup": "myArcClusters2",
"type": "Microsoft.KubernetesConfiguration/sourceControlConfigurations"
}
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$
And we see a deployment launched in EKS
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get deployments --all-namespaces
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
azure-arc cluster-metadata-operator 1/1 1 1 50m
azure-arc clusteridentityoperator 1/1 1 1 50m
azure-arc config-agent 1/1 1 1 50m
azure-arc controller-manager 1/1 1 1 50m
azure-arc flux-logs-agent 1/1 1 1 50m
azure-arc metrics-agent 1/1 1 1 50m
azure-arc resource-sync-agent 1/1 1 1 50m
default app-config 0/1 1 0 4s
default memcached 0/1 1 0 4s
kube-system coredns 2/2 2 2 125m
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl describe deployment app-config
Name: app-config
Namespace: default
CreationTimestamp: Wed, 20 May 2020 12:12:10 -0500
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
Selector: instanceName=app-config,name=flux
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: instanceName=app-config
name=flux
Annotations: prometheus.io/port: 3031
Service Account: app-config
Containers:
flux:
Image: docker.io/fluxcd/flux:1.18.0
Port: 3030/TCP
Host Port: 0/TCP
Args:
--k8s-secret-name=app-config-git-deploy
--memcached-service=
--ssh-keygen-dir=/var/fluxd/keygen
--git-url=https://princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo
--git-branch=master
--git-path=manifests
--git-label=flux
--git-user=Flux
--git-readonly
--connect=ws://flux-logs-agent.azure-arc/default/iam-demo-config
--listen-metrics=:3031
Requests:
cpu: 50m
memory: 64Mi
Liveness: http-get http://:3030/api/flux/v6/identity.pub delay=5s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:3030/api/flux/v6/identity.pub delay=5s timeout=5s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/fluxd/ssh from git-key (ro)
/var/fluxd/keygen from git-keygen (rw)
Volumes:
git-key:
Type: Secret (a volume populated by a Secret)
SecretName: app-config-git-deploy
Optional: false
git-keygen:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: app-config-768cc4fc89 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 18s deployment-controller Scaled up replica set app-config-768cc4fc89 to 1
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get pods
NAME READY STATUS RESTARTS AGE
app-config-768cc4fc89-f5n66 1/1 Running 0 45s
memcached-86869f57fd-r2kgj 1/1 Running 0 45s
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl describe app-config-768cc4fc89-f5n66
error: the server doesn't have a resource type "app-config-768cc4fc89-f5n66"
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl describe pod app-config-768cc4fc89-f5n66
Name: app-config-768cc4fc89-f5n66
Namespace: default
Priority: 0
Node: ip-10-0-82-16.us-east-2.compute.internal/10.0.82.16
Start Time: Wed, 20 May 2020 12:12:10 -0500
Labels: instanceName=app-config
name=flux
pod-template-hash=768cc4fc89
Annotations: kubernetes.io/psp: eks.privileged
prometheus.io/port: 3031
Status: Running
IP: 10.0.84.119
Controlled By: ReplicaSet/app-config-768cc4fc89
Containers:
flux:
Container ID: docker://d638ec677da94ca4d4c32fcd5ab9e35b390d33fa433b350976152f744f105244
Image: docker.io/fluxcd/flux:1.18.0
Image ID: docker-pullable://fluxcd/flux@sha256:8fcf24dccd7774b87a33d87e42fa0d9233b5c11481c8414fe93a8bdc870b4f5b
Port: 3030/TCP
Host Port: 0/TCP
Args:
--k8s-secret-name=app-config-git-deploy
--memcached-service=
--ssh-keygen-dir=/var/fluxd/keygen
--git-url=https://princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo
--git-branch=master
--git-path=manifests
--git-label=flux
--git-user=Flux
--git-readonly
--connect=ws://flux-logs-agent.azure-arc/default/iam-demo-config
--listen-metrics=:3031
State: Running
Started: Wed, 20 May 2020 12:12:18 -0500
Ready: True
Restart Count: 0
Requests:
cpu: 50m
memory: 64Mi
Liveness: http-get http://:3030/api/flux/v6/identity.pub delay=5s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:3030/api/flux/v6/identity.pub delay=5s timeout=5s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/fluxd/ssh from git-key (ro)
/var/fluxd/keygen from git-keygen (rw)
/var/run/secrets/kubernetes.io/serviceaccount from app-config-token-rpzr5 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
git-key:
Type: Secret (a volume populated by a Secret)
SecretName: app-config-git-deploy
Optional: false
git-keygen:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
app-config-token-rpzr5:
Type: Secret (a volume populated by a Secret)
SecretName: app-config-token-rpzr5
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 63s default-scheduler Successfully assigned default/app-config-768cc4fc89-f5n66 to ip-10-0-82-16.us-east-2.compute.internal
Normal Pulling 62s kubelet, ip-10-0-82-16.us-east-2.compute.internal Pulling image "docker.io/fluxcd/flux:1.18.0"
Normal Pulled 55s kubelet, ip-10-0-82-16.us-east-2.compute.internal Successfully pulled image "docker.io/fluxcd/flux:1.18.0"
Normal Created 55s kubelet, ip-10-0-82-16.us-east-2.compute.internal Created container flux
Normal Started 55s kubelet, ip-10-0-82-16.us-east-2.compute.internal Started container flux
Let’s push it:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ git add manifests/
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ git status
On branch master
Your branch is up to date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: manifests/azure-vote-all-in-one-redis.yaml
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ git commit -m "push changes for a redis app"
[master f9f49cc] push changes for a redis app
1 file changed, 78 insertions(+)
create mode 100644 manifests/azure-vote-all-in-one-redis.yaml
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ git push
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 815 bytes | 815.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0)
remote: Analyzing objects... (4/4) (41 ms)
remote: Storing packfile... done (143 ms)
remote: Storing index... done (51 ms)
remote: We noticed you're using an older version of Git. For the best experience, upgrade to a newer version.
To https://princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo
d11c1e8..f9f49cc master -> master
However i did not see it change. That’s when i realized i needed to pack the credentials into the URL:
Once saved, I saw it start to update:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get pods
NAME READY STATUS RESTARTS AGE
app-config-8c4b5fd6f-jj995 1/1 Running 0 13s
azure-vote-back-5775d78ff5-s2mh9 0/1 ContainerCreating 0 4s
azure-vote-front-559d85d4f7-jhb84 0/1 ContainerCreating 0 4s
memcached-86869f57fd-r2kgj 1/1 Running 0 20m
And we can see it launched:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
azure-vote-back ClusterIP 172.20.46.228 <none> 6379/TCP 6m55s
azure-vote-front LoadBalancer 172.20.208.101 a2863a6eaeda34906a8583a1b4175842-1241817646.us-east-2.elb.amazonaws.com 80:30262/TCP 6m55s
kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 153m
memcached ClusterIP 172.20.171.75 <none> 11211/TCP 27m
Let’s then launch the same app on LKE:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ az k8sconfiguration create --name iam-demo-config --cluster-name myLKECluster --resource-group myArcClusters2 --operator-instance-name app-config --operator-namespace default --repository-url https://isaac.johnson:***********************************@princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo --cluster-type connectedClusters --scope cluster --operator-params="--git-readonly --git-path=manifests"
Command group 'k8sconfiguration' is in preview. It may be changed/removed in a future release.
{
"complianceStatus": {
"complianceState": "Pending",
"lastConfigApplied": "0001-01-01T00:00:00",
"message": "{\"OperatorMessage\":null,\"ClusterState\":null}",
"messageLevel": "3"
},
"enableHelmOperator": "False",
"helmOperatorProperties": null,
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters2/providers/Microsoft.Kubernetes/connectedClusters/myLKECluster/providers/Microsoft.KubernetesConfiguration/sourceControlConfigurations/iam-demo-config",
"name": "iam-demo-config",
"operatorInstanceName": "app-config",
"operatorNamespace": "default",
"operatorParams": "--git-readonly --git-path=manifests",
"operatorScope": "cluster",
"operatorType": "Flux",
"provisioningState": "Succeeded",
"repositoryPublicKey": "",
"repositoryUrl": "https://isaac.johnson:abcdef12345abcdef12345abcdef12345abcdef12345@princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo",
"resourceGroup": "myArcClusters2",
"type": "Microsoft.KubernetesConfiguration/sourceControlConfigurations"
}
Verification:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ !1675
linode-cli lke kubeconfig-view 5193 > t.o && cat t.o | tail -n2 | head -n1 | sed 's/.\{8\}$//' | sed 's/^.\{8\}//' | base64 --decode > ~/.kube/config
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
azure-vote-back ClusterIP 10.128.43.37 <none> 6379/TCP 2m
azure-vote-front LoadBalancer 10.128.85.35 45.79.243.16 80:32408/TCP 2m
kubernetes ClusterIP 10.128.0.1 <none> 443/TCP 160m
memcached ClusterIP 10.128.82.70 <none> 11211/TCP 2m24s
Updating
Let’s now change the values:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ git diff
diff --git a/manifests/azure-vote-all-in-one-redis.yaml b/manifests/azure-vote-all-in-one-redis.yaml
index 80a8edb..d64cfce 100644
--- a/manifests/azure-vote-all-in-one-redis.yaml
+++ b/manifests/azure-vote-all-in-one-redis.yaml
@@ -20,6 +20,9 @@ spec:
ports:
- containerPort: 6379
name: redis
+ env:
+ - name: VOTE1VALUE
+ value: "Elephants"
---
apiVersion: v1
kind: Service
@@ -65,6 +68,8 @@ spec:
env:
- name: REDIS
value: "azure-vote-back"
+ - name: VOTE1VALUE
+ value: "Elephants"
---
apiVersion: v1
kind: Service
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ git add azure-vote-all-in-one-redis.yaml
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ git commit -m "Change the Cats option to Elephants"
[master 46cdb15] Change the Cats option to Elephants
1 file changed, 5 insertions(+)
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ git push
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 430 bytes | 430.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Analyzing objects... (4/4) (169 ms)
remote: Storing packfile... done (175 ms)
remote: Storing index... done (204 ms)
remote: We noticed you're using an older version of Git. For the best experience, upgrade to a newer version.
To https://princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo
f9f49cc..46cdb15 master -> master
Deleting
I watched for a while, but deleting the configuration did not remove the former configurations
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
app-config 1/1 1 1 52m
azure-vote-back 1/1 1 1 32m
azure-vote-front 1/1 1 1 32m
memcached 1/1 1 1 52m
What happens when we relaunch?
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ az k8sconfiguration create --name iam-demo-config --cluster-name myEKSCluster --resource-group myArcClusters2 --operator-instance-name app-config --operator-namespace default --repository-url https://isaac.johnson:abcdef12345abcdef12345abcdef12345abcdef12345@princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo --cluster-type connectedClusters --scope cluster --operator-params="--git-path=manifests"
Command group 'k8sconfiguration' is in preview. It may be changed/removed in a future release.
{
"complianceStatus": {
"complianceState": "Pending",
"lastConfigApplied": "0001-01-01T00:00:00",
"message": "{\"OperatorMessage\":null,\"ClusterState\":null}",
"messageLevel": "3"
},
"enableHelmOperator": "False",
"helmOperatorProperties": null,
"id": "/subscriptions/70b42e6a-asdf-asdf-asdf-9f3995b1aca8/resourceGroups/myArcClusters2/providers/Microsoft.Kubernetes/connectedClusters/myEKSCluster/providers/Microsoft.KubernetesConfiguration/sourceControlConfigurations/iam-demo-config",
"name": "iam-demo-config",
"operatorInstanceName": "app-config",
"operatorNamespace": "default",
"operatorParams": "--git-readonly --git-path=manifests",
"operatorScope": "cluster",
"operatorType": "Flux",
"provisioningState": "Succeeded",
"repositoryPublicKey": "",
"repositoryUrl": "https://isaac.johnson:abcdef12345abcdef12345abcdef12345abcdef12345@princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo",
"resourceGroup": "myArcClusters2",
"type": "Microsoft.KubernetesConfiguration/sourceControlConfigurations"
}
Within moments, i did see the former deployment get removed
Every 2.0s: kubectl get deployments DESKTOP-JBA79RT: Wed May 20 13:08:36 2020
NAME READY UP-TO-DATE AVAILABLE AGE
azure-vote-back 1/1 1 1 36m
azure-vote-front 1/1 1 1 36m
I noticed an error in the portal
{"OperatorMessage":"unable to add the configuration with configId {iam-demo-config} due to error: {error while adding the CRD configuration: error {object is being deleted: gitconfigs.clusterconfig.azure.com \"iam-demo-config\" already exists}}","ClusterState":"Failed","LastGitCommitInformation":"","ErrorsInTheLastSynced":""}very 2.0s: kubectl get deployments DESKTOP-JBA79RT: Wed May 20 13:09:36 2020
NAME READY UP-TO-DATE AVAILABLE AGE
app-config 1/1 1 1 31s
azure-vote-back 1/1 1 1 37m
azure-vote-front 1/1 1 1 37m
memcached 1/1 1 1 31s
It seemed that while it did clean up old deployments, it did not relaunch and got hung up on the --readonly flag.
No matter what, it keeps that flag in there. I recreated in new namespaces, and ensured i just set --operator-params="--git-path=manifests" but then it shows in Azure “--git-readonly --git-path=manifests
I tried namepaces, deleting deployments, updating the token for git and applying it.. Nothing seemed to relaunch. This is in preview so I would expect these issues to get sorted out.
Slow… eventually it came back:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
azure-arc cluster-metadata-operator-57fc656db4-zdf5p 2/2 Running 0 157m
azure-arc clusteridentityoperator-7598664f64-5cvk2 3/3 Running 0 157m
azure-arc config-agent-798648fbd5-g74gb 3/3 Running 0 157m
azure-arc controller-manager-64985bcb6b-7kwl9 3/3 Running 0 157m
azure-arc flux-logs-agent-6f8fdb7d58-4hbmc 2/2 Running 0 157m
azure-arc metrics-agent-5f55ccddfd-pmcc8 2/2 Running 0 157m
azure-arc resource-sync-agent-95c997779-rnpf8 3/3 Running 0 157m
default app-config-55c7974448-rwdkb 1/1 Running 0 15m
default azure-vote-back-567c66c8d8-5xqp8 1/1 Running 0 30s
default azure-vote-front-fd7d8d4d7-wvhnx 1/1 Running 0 30s
default memcached-86869f57fd-t7g5k 1/1 Running 0 49m
Indeed i scan verify i passed in the SHOWHOSTS var:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ kubectl describe pod azure-vote-front-fd7d8d4d7-wvhnx
Name: azure-vote-front-fd7d8d4d7-wvhnx
Namespace: default
Priority: 0
Node: ip-10-0-61-98.us-east-2.compute.internal/10.0.61.98
Start Time: Wed, 20 May 2020 13:58:23 -0500
Labels: app=azure-vote-front
pod-template-hash=fd7d8d4d7
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.0.52.147
Controlled By: ReplicaSet/azure-vote-front-fd7d8d4d7
Containers:
azure-vote-front:
Container ID: docker://41ddabfcab73e3e4d9c1e8e0d78aac5427cdf91df3d8fc56167b3e55e3fe71a1
Image: microsoft/azure-vote-front:v1
Image ID: docker-pullable://microsoft/azure-vote-front@sha256:9ace3ce43db1505091c11d15edce7b520cfb598d38402be254a3024146920859
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 20 May 2020 13:58:24 -0500
Ready: True
Restart Count: 0
Limits:
cpu: 500m
Requests:
cpu: 250m
Environment:
REDIS: azure-vote-back
SHOWHOST: true
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7c4vq (ro)
Let’s make an AKS cluster in the same RG:
I then created a new YAML pipeline and verified that indeed, i cannot see Arc clusters listed via ARM connections:
One last check. I added another helloworld yaml (different app).
version: v2
spec:
replicas: 1
selector:
matchLabels:
app: helloworld
version: v2
template:
metadata:
labels:
app: helloworld
version: v2
spec:
containers:
- name: helloworld
image: docker.io/istio/examples-helloworld-v2
resources:
requests:
cpu: "100m"
imagePullPolicy: IfNotPresent #Always
ports:
- containerPort: 5000
I saw that indeed it was deployed, albeit to the first configuration we had applied
EKS:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ kubectl get pods | grep hello
helloworld-v1-6757db4ff5-26h26 1/1 Running 0 23m
helloworld-v2-85bc988875-lh7cg 1/1 Running 0 23m
LKE:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ kubectl get pods --all-namespaces | grep hello
default helloworld-v1-6757db4ff5-vw7zq 1/1 Running 0 27m
default helloworld-v2-85bc988875-k698z 1/1 Running 0 27m
I then removed it:
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ git commit -m "remove helloworld"
[master 2c12de4] remove helloworld
1 file changed, 1 insertion(+), 1 deletion(-)
rename manifests/helloworld.yaml => helloworld.md (98%)
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo$ git push
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 704 bytes | 704.00 KiB/s, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Analyzing objects... (4/4) (189 ms)
remote: Storing packfile... done (252 ms)
remote: Storing index... done (92 ms)
remote: We noticed you're using an older version of Git. For the best experience, upgrade to a newer version.
To https://princessking.visualstudio.com/IAC-Demo/_git/IAC-Demo
5b88894..2c12de4 master -> master
However, it was not removed from the EKS or LKE clusters.
I did one more test. I added sleep deployment to ensure content was still getting pushed. Within about 4 minutes on both clusters, i could see the Sleep deployment, but importantly, the helloworld deployments still existed.
So it’s clearly additive (ie. kubectl create) and not deleting anything.
Cleanup
LKE
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ linode-cli lke clusters-list
┌──────┬───────────┬────────────┐
│ id │ label │ region │
├──────┼───────────┼────────────┤
│ 5193 │ mycluster │ us-central │
└──────┴───────────┴────────────┘
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ linode-cli lke cluster-delete 5193
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ linode-cli lke clusters-list
┌────┬───────┬────────┐
│ id │ label │ region │
└────┴───────┴────────┘
EKS
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ aws cloudformation list-stacks --region us-east-2 | grep Amazon-EKS
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC-BastionStack-75ZZA7D9Y8VZ/5df70170-9aad-11ea-aa5c-02582149edd0",
"StackName": "Amazon-EKS-EKSStack-SDZCUE9193SC-BastionStack-75ZZA7D9Y8VZ",
"ParentId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC/7ab4d9f0-9aa7-11ea-81b8-064e3beebd9c",
"RootId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC-NodeGroupStack-14OM4SDFRQ1HZ/5dd48550-9aad-11ea-856e-063c30684966",
"StackName": "Amazon-EKS-EKSStack-SDZCUE9193SC-NodeGroupStack-14OM4SDFRQ1HZ",
"ParentId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC/7ab4d9f0-9aa7-11ea-81b8-064e3beebd9c",
"RootId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC-EKSControlPlane-RLECTP00VA5Z/4c73eb70-9aa8-11ea-bc6c-026789fff09e",
"StackName": "Amazon-EKS-EKSStack-SDZCUE9193SC-EKSControlPlane-RLECTP00VA5Z",
"ParentId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC/7ab4d9f0-9aa7-11ea-81b8-064e3beebd9c",
"RootId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC-FunctionStack-J0MM85F3VKX3/ec211b30-9aa7-11ea-9fe1-0696b95a38c4",
"StackName": "Amazon-EKS-EKSStack-SDZCUE9193SC-FunctionStack-J0MM85F3VKX3",
"ParentId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC/7ab4d9f0-9aa7-11ea-81b8-064e3beebd9c",
"RootId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC-IamStack-144D49GJXUSOB/8bb803d0-9aa7-11ea-a843-0a9fc5c4303c",
"StackName": "Amazon-EKS-EKSStack-SDZCUE9193SC-IamStack-144D49GJXUSOB",
"ParentId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC/7ab4d9f0-9aa7-11ea-81b8-064e3beebd9c",
"RootId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-EKSStack-SDZCUE9193SC/7ab4d9f0-9aa7-11ea-81b8-064e3beebd9c",
"StackName": "Amazon-EKS-EKSStack-SDZCUE9193SC",
"ParentId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"RootId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS-VPCStack-1LUZPCD2BF1J/e78197e0-9aa6-11ea-9392-062398501aa6",
"StackName": "Amazon-EKS-VPCStack-1LUZPCD2BF1J",
"ParentId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"RootId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackId": "arn:aws:cloudformation:us-east-2:01234512345:stack/Amazon-EKS/e35fb390-9aa6-11ea-9ffc-021e2396e2cc",
"StackName": "Amazon-EKS",
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ aws cloudformation delete-stack --stack-name "Amazon-EKS" --region us-east-2
builder@DESKTOP-JBA79RT:~/Workspaces/AzureArc/IAC-Demo/manifests$ aws cloudformation list-stacks --region us-east-2 | grep StackStatus
"StackStatus": "DELETE_IN_PROGRESS",
"StackStatus": "DELETE_IN_PROGRESS",
"StackStatus": "UPDATE_COMPLETE",
"StackStatus": "CREATE_COMPLETE",
"StackStatus": "CREATE_COMPLETE",
"StackStatus": "DELETE_IN_PROGRESS",
"StackStatus": "CREATE_COMPLETE",
"StackStatus": "DELETE_IN_PROGRESS",
"StackStatus": "DELETE_COMPLETE",
"StackStatus": "DELETE_COMPLETE",
"StackStatus": "DELETE_COMPLETE",
Cleaning up the Azure RG:
Summary
Azure Arc for Kubernetes is a really solid offering from Microsoft to manage all your Kubernetes Clusters’ configurations in one place. It is in public preview so there are some issues I experienced - namely handling bad YAML and no method to remove deployments (Add and Update, but not Deprecation).
We did not test Helm support but one can enable Helm chart deployments as well.