Published: Nov 10, 2021 by Isaac Johnson
One of the more scary operations one must perform is upgrading a Kubernetes cluster. While new versions often bring better stability and new functionality, they also can deprecate APIs upon which you may depend.
Knowing how to handle upgrades is key to maintaining an operational production cluster. Today we’ll work through upgrades in several ways:
- K3s directly with the System Upgrade Controller
- K3s in Terraform (one way working, one not)
- Azure Kubernetes Service (AKS)
- Minikube (with Hyper-V)
- Rancher to install and upgrade k3s in Azure
Upgrading K3s
This first type of upgrade involves my on-prem production cluster that has quite a bit of content on it. This k3s cluster is hosted on several physical machines networked together.
First, we need to setup the system upgrade controller on the cluster:
$ kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/v0.8.0/manifests/system-upgrade-controller.yaml
Set the Master node and set all the nodes to upgrade. The Master Node label should already exist, but not the k3s-upgrade one.
# kubectl label node isaac-macbookair node-role.kubernetes.io/master=true
$ kubectl label node anna-macbookair isaac-macbookpro isaac-macbookair hp-hp-elitebook-850-g2 builder-hp-elitebook-850-g1 k3s-upgrade=true
Next, apply the upgrade plan:
$ cat UpgradePlan.yml
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: k3s-server
namespace: system-upgrade
labels:
k3s-upgrade: server
spec:
concurrency: 1
version: v1.21.5+k3s1
nodeSelector:
matchExpressions:
- {key: k3s-upgrade, operator: Exists}
- {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
- {key: k3s.io/hostname, operator: Exists}
- {key: k3os.io/mode, operator: DoesNotExist}
- {key: node-role.kubernetes.io/master, operator: In, values: ["true"]}
serviceAccountName: system-upgrade
cordon: true
drain:
force: true
upgrade:
image: rancher/k3s-upgrade
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: k3s-agent
namespace: system-upgrade
labels:
k3s-upgrade: agent
spec:
concurrency: 3 # in general, this should be the number of workers - 1
version: v1.21.5+k3s1
nodeSelector:
matchExpressions:
- {key: k3s-upgrade, operator: Exists}
- {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
- {key: k3s.io/hostname, operator: Exists}
- {key: k3os.io/mode, operator: DoesNotExist}
- {key: node-role.kubernetes.io/master, operator: NotIn, values: ["true"]}
serviceAccountName: system-upgrade
prepare:
# Since v0.5.0-m1 SUC will use the resolved version of the plan for the tag on the prepare container.
# image: rancher/k3s-upgrade:v1.17.4-k3s1
image: rancher/k3s-upgrade
args: ["prepare", "k3s-server"]
drain:
force: true
skipWaitForDeleteTimeout: 60 # set this to prevent upgrades from hanging on small clusters since k8s v1.18
upgrade:
image: rancher/k3s-upgrade
$ kubectl apply -f UpgradePlan.yml
This should upgrade us from v1.19.5+k3s2 to v1.21.5+k3s1.
Note: This is my oldest cluster, the nodes had been running nearly a year in some cases. It has many many things on it, from Dapr to Datadog monitors to full stack apps like Harbor. I worried that it wasn’t going to upgrade because I saw no status updates and after hours I assumed it just was a no go.
However, after about 4 hours, i checked back and it migrated nodes and by 5 hours, it was live again.
When the migration is complete, we see the jobs disappear:
$ kubectl get jobs -n system-upgrade
No resources found in system-upgrade namespace.
You may need to rotate pods to get them to pick up changes. In my case, since everything rotates, the stuck redis master blocked the redis slaves which in turn blocked Dapr side cars (which use Redis for state management and message queues).
perl-subscriber-57778545dc-5qp99 1/2 CrashLoopBackOff 16 58m
node-subscriber-5c75bfc99d-h57s8 1/2 CrashLoopBackOff 18 52m
daprtweeter-deployment-68b49d6856-w9fnw 1/2 CrashLoopBackOff 18 58m
python-subscriber-7d7694f594-nr6dn 1/2 CrashLoopBackOff 16 52m
dapr-workflows-host-85d9d74777-h7tpk 1/2 CrashLoopBackOff 16 52m
echoapp-c56bfd446-94zzg 1/2 CrashLoopBackOff 17 52m
react-form-79c7989844-gpgzg 1/2 CrashLoopBackOff 15 52m
redis-master-0 1/1 Running 0 3m38s
redis-slave-0 1/1 Running 0 2m50s
redis-slave-1 1/1 Running 0 2m29s
nfs-client-provisioner-d54c468c4-r2969 1/1 Running 0 106s
nodeeventwatcher-deployment-6dddc4858c-b685p 0/2 ContainerCreating 0 93s
nodesqswatcher-deployment-67b65f645f-c5xhl 2/2 Running 0 38s
builder@DESKTOP-72D2D9T:~$ kubectl delete pod perl-subscriber-57778545dc-5qp99 && kubectl delete pod daprtweeter-deployment-68b49d6856-w9fnw
pod "perl-subscriber-57778545dc-5qp99" deleted
pod "daprtweeter-deployment-68b49d6856-w9fnw" deleted
Now we see they are all up or coming up:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
svclb-my-release-ingress-nginx-controller-dgmgt 2/2 Running 2 150d
svclb-react-form-hkw8j 1/1 Running 1 193d
svclb-react-form-btvf7 1/1 Running 2 20d
datadogrelease-g5kqq 3/3 Running 0 6d13h
svclb-my-release-ingress-nginx-controller-2kbgb 0/2 Pending 0 76m
svclb-react-form-tbcvj 0/1 Pending 0 76m
svclb-my-release-ingress-nginx-controller-hw92x 0/2 Pending 0 76m
svclb-my-release-ingress-nginx-controller-z4qwg 0/2 Pending 0 76m
svclb-react-form-g486p 0/1 Pending 0 76m
vote-front-azure-vote-1608995981-588948447b-2cv6f 1/1 Running 0 75m
dapr-sidecar-injector-56b8954855-48dmd 1/1 Running 0 75m
dapr-placement-server-0 1/1 Running 0 75m
svclb-my-release-ingress-nginx-controller-hbtjx 2/2 Running 0 75m
svclb-react-form-cc9lg 1/1 Running 0 75m
otel-collector-67f645b9b7-w4flh 1/1 Running 0 75m
kubewatch-5d466cffc8-tf579 1/1 Running 0 60m
dapr-dashboard-6ff6f44778-tlssf 1/1 Running 0 60m
dapr-sentry-958fdd984-ts7tk 1/1 Running 0 60m
harbor-registry-harbor-notary-server-779c6bddd5-6rb94 1/1 Running 0 60m
vote-back-azure-vote-1608995981-5df9f78fd8-7vhk6 1/1 Running 0 60m
harbor-registry-harbor-exporter-655dd658bb-79nrj 1/1 Running 0 60m
datadogrelease-lzkqn 3/3 Running 0 4h3m
datadogrelease-p8zmz 3/3 Running 0 6d13h
cm-acme-http-solver-t94vj 1/1 Running 0 59m
harbor-registry-harbor-redis-0 1/1 Running 0 75m
harbor-registry-harbor-registry-86dbbfd48f-6gbmx 2/2 Running 0 75m
datadogrelease-s45zf 3/3 Running 3 5h32m
harbor-registry-harbor-core-7b4594d78d-rdfgw 1/1 Running 1 60m
harbor-registry-harbor-jobservice-95968c6d9-lgs6q 1/1 Running 0 75m
busybox-6c446876c6-7pzsx 1/1 Running 0 55m
docker-registry-6d9dc74c67-9h2k2 1/1 Running 0 54m
harbor-registry-harbor-portal-76bdcc7969-cnc8r 1/1 Running 0 54m
dapr-operator-7867c79bf9-5h2dv 1/1 Running 0 54m
mongo-x86-release-mongodb-7cb86d48f8-ks2f6 1/1 Running 0 54m
my-release-ingress-nginx-controller-7978f85c6f-467hr 1/1 Running 0 55m
datadogrelease-kube-state-metrics-6b5956746b-nm6kh 1/1 Running 0 54m
perl-debugger-5967f99ff6-lbtd2 1/1 Running 0 75m
datadogrelease-cluster-agent-6cf556df59-r56kd 1/1 Running 0 54m
datadogrelease-glxn2 3/3 Running 0 6d13h
harbor-registry-harbor-chartmuseum-559bd98f8f-42h4n 1/1 Running 0 55m
harbor-registry-harbor-trivy-0 1/1 Running 0 52m
harbor-registry-harbor-database-0 1/1 Running 0 52m
harbor-registry-harbor-notary-signer-c97648889-mnw7v 1/1 Running 6 54m
redis-master-0 1/1 Running 0 6m26s
redis-slave-0 1/1 Running 0 5m38s
redis-slave-1 1/1 Running 0 5m17s
nfs-client-provisioner-d54c468c4-r2969 1/1 Running 0 4m34s
nodesqswatcher-deployment-67b65f645f-c5xhl 2/2 Running 0 3m26s
nodeeventwatcher-deployment-6dddc4858c-b685p 2/2 Running 0 4m21s
node-subscriber-5c75bfc99d-h57s8 2/2 Running 19 54m
dapr-workflows-host-85d9d74777-h7tpk 2/2 Running 17 55m
python-subscriber-7d7694f594-nr6dn 2/2 Running 17 54m
perl-subscriber-57778545dc-tztsq 2/2 Running 0 108s
echoapp-c56bfd446-94zzg 2/2 Running 18 54m
react-form-79c7989844-gpgzg 2/2 Running 16 55m
daprtweeter-deployment-68b49d6856-k7tml 2/2 Running 0 67s
We can see results in the Datadog Kubernetes Dashboard:
Another example: K3s one node
We can use the Upgrade Controller method on very small k3s clusters as well.
Here we have a very simple computer running a one node k3sL
builder@builder-HP-EliteBook-745-G5:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
builder-hp-elitebook-745-g5 Ready control-plane,master 45h v1.21.5+k3s1
builder@builder-HP-EliteBook-745-G5:~$ kubectl get pods
No resources found in default namespace.
builder@builder-HP-EliteBook-745-G5:~$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
traefik-5dd496474-8bbnv 1/1 Running 0 45h
metrics-server-86cbb8457f-qgq9w 1/1 Running 0 45h
local-path-provisioner-5ff76fc89d-j748k 1/1 Running 0 45h
svclb-traefik-58bb8 2/2 Running 0 45h
coredns-7448499f4d-szkpl 1/1 Running 0 45h
helm-install-traefik-t8ntj 0/1 Completed 0 45h
We’ll get the latest system upgrader version
$ curl -s "https://api.github.com/repos/rancher/system-upgrade-controller/releases/latest" | awk -F '"' '/tag_name/{print $4}'
v0.8.0
Then apply:
$ kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/v0.8.0/manifests/system-upgrade-controller.yaml
namespace/system-upgrade unchanged
serviceaccount/system-upgrade unchanged
clusterrolebinding.rbac.authorization.k8s.io/system-upgrade unchanged
configmap/default-controller-env unchanged
deployment.apps/system-upgrade-controller unchanged
Since we upgraded once already, there were no changes
We can bring up the k3s releases page to see the latest available: https://github.com/k3s-io/k3s/releases
And at time of writing, v1.22.3-rc1+k3s1 was just published yesterday.
Since we already had the UpgradePlan.yaml file from last time, we can just sed in the new version
builder@builder-HP-EliteBook-745-G5:~$ sed -i 's/v1.21.5+k3s1/v1.22.3-rc1+k3s1/g' UpgradePlan.yaml
builder@builder-HP-EliteBook-745-G5:~$ cat UpgradePlan.yaml
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: k3s-server
namespace: system-upgrade
labels:
k3s-upgrade: server
spec:
concurrency: 1
version: v1.22.3-rc1+k3s1
nodeSelector:
matchExpressions:
- {key: k3s-upgrade, operator: Exists}
- {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
- {key: k3s.io/hostname, operator: Exists}
- {key: k3os.io/mode, operator: DoesNotExist}
- {key: node-role.kubernetes.io/master, operator: In, values: ["true"]}
serviceAccountName: system-upgrade
cordon: true
# drain:
# force: true
upgrade:
image: rancher/k3s-upgrade
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: k3s-agent
namespace: system-upgrade
labels:
k3s-upgrade: agent
spec:
concurrency: 2 # in general, this should be the number of workers - 1
version: v1.22.3-rc1+k3s1
nodeSelector:
matchExpressions:
- {key: k3s-upgrade, operator: Exists}
- {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
- {key: k3s.io/hostname, operator: Exists}
- {key: k3os.io/mode, operator: DoesNotExist}
- {key: node-role.kubernetes.io/master, operator: NotIn, values: ["true"]}
serviceAccountName: system-upgrade
prepare:
# Since v0.5.0-m1 SUC will use the resolved version of the plan for the tag on the prepare container.
# image: rancher/k3s-upgrade:v1.17.4-k3s1
image: rancher/k3s-upgrade
args: ["prepare", "k3s-server"]
drain:
force: true
skipWaitForDeleteTimeout: 60 # set this to prevent upgrades from hanging on small clusters since k8s v1.18
upgrade:
image: rancher/k3s-upgrade
Because we are interested in how it handles deprecations, we can look at what are the Major Themes and Major Changes in 1.22: https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/
Notably:
A number of deprecated beta APIs have been removed in 1.22 in favor of the GA version of those same APIs. All existing objects can be interacted with via stable APIs. This removal includes beta versions of the Ingress, IngressClass, Lease, APIService, ValidatingWebhookConfiguration, MutatingWebhookConfiguration, CustomResourceDefinition, TokenReview, SubjectAccessReview, and CertificateSigningRequest APIs.
How Upgrades handle Deprecations
To test how the upgrades will handle that, let’s test the Ingress change - removal of v1beta1 Ingresses.
First, we can install Nginx. This is not required since by default our cluster has Traefik:
builder@builder-HP-EliteBook-745-G5:~$ helm repo add nginx-stable https://helm.nginx.com/stable
"nginx-stable" has been added to your repositories
builder@builder-HP-EliteBook-745-G5:~$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nginx-stable" chart repository
Update Complete. ⎈Happy Helming!⎈
builder@builder-HP-EliteBook-745-G5:~$ export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
builder@builder-HP-EliteBook-745-G5:~$ helm install my-release nginx-stable/nginx-ingress
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
NAME: my-release
LAST DEPLOYED: Fri Oct 29 16:45:53 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The NGINX Ingress Controller has been installed.
As mentioned, at the moment we do not need it as we have Traefik handling our ingress already
$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 2d1h
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 2d1h
kube-system metrics-server ClusterIP 10.43.106.155 <none> 443/TCP 2d1h
kube-system traefik-prometheus ClusterIP 10.43.97.166 <none> 9100/TCP 2d1h
kube-system traefik LoadBalancer 10.43.222.161 192.168.1.159 80:31400/TCP,443:31099/TCP 2d1h
default my-release-nginx-ingress LoadBalancer 10.43.141.8 <pending> 80:31933/TCP,443:32383/TCP 3h18m
default nginx-service ClusterIP 10.43.174.8 <none> 80/TCP 3m9s
Next, let’s deploy a basic Nginx container. This is just a simple webserver.
$ cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
$ kubectl create -f https://k8s.io/examples/application/deployment.yaml
deployment.apps/nginx-deployment created
So as to expose via Ingress, we need a service to match the pods:
$ cat service.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
$ kubectl apply -f service.yaml
service/nginx-service created
Lastly, let’s apply the Ingress:
$ cat testIngress.yaml
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-v1betatest
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /path1
backend:
serviceName: nginx-service
servicePort: 80
$ kubectl apply -f testIngress.yaml
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-v1betatest <none> * 192.168.1.159 80 6s
We can see Traefik
Then a basic nginx response using /path1
Applying the upgrade we can first check that the node version has yet to change:
NAME STATUS ROLES AGE VERSION
builder-hp-elitebook-745-g5 Ready control-plane,master 2d1h v1.21.5+k3s1
and pods are still running
builder@builder-HP-EliteBook-745-G5:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
svclb-my-release-nginx-ingress-h6thm 0/2 Pending 0 3h22m
my-release-nginx-ingress-865865d656-gtkzq 1/1 Running 0 3h22m
nginx-deployment-66b6c48dd5-pcfbg 1/1 Running 0 3h16m
nginx-deployment-66b6c48dd5-bfqb2 1/1 Running 0 3h16m
builder@builder-HP-EliteBook-745-G5:~$ kubectl get jobs --all-namespaces
NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system helm-install-traefik 1/1 19s 2d1h
system-upgrade apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b 0/1 32s 32s
We can check status of the upgrade job using describe:
builder@builder-HP-EliteBook-745-G5:~$ kubectl describe jobs apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b -n system-upgrade
Name: apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b
Namespace: system-upgrade
Selector: controller-uid=c79b144d-9a55-44cc-bda5-1528363514c8
Labels: objectset.rio.cattle.io/hash=9f2ac612f7e407418de10c3f7b09695f3acb1dfa
plan.upgrade.cattle.io/k3s-server=151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
upgrade.cattle.io/controller=system-upgrade-controller
upgrade.cattle.io/node=builder-hp-elitebook-745-g5
upgrade.cattle.io/plan=k3s-server
upgrade.cattle.io/version=v1.22.3-rc1-k3s1
Annotations: objectset.rio.cattle.io/applied:
H4sIAAAAAAAA/+xXUW/iRhD+K9U828QGQsBSH2jgeujuCDpyrU6nKFrvjmHLetfdHZOgyP+9WkM4kxCSa58qnSIFbO98M9/MfMz4AXIkJhgxSB5AsxwhAVYUahOuOi50aNdoQ6PDtJ...
objectset.rio.cattle.io/id: system-upgrade-controller
objectset.rio.cattle.io/owner-gvk: upgrade.cattle.io/v1, Kind=Plan
objectset.rio.cattle.io/owner-name: k3s-server
objectset.rio.cattle.io/owner-namespace: system-upgrade
upgrade.cattle.io/ttl-seconds-after-finished: 900
Parallelism: 1
Completions: 1
Start Time: Fri, 29 Oct 2021 20:07:53 -0500
Active Deadline Seconds: 900s
Pods Statuses: 0 Running / 0 Succeeded / 0 Failed
Pod Template:
Labels: controller-uid=c79b144d-9a55-44cc-bda5-1528363514c8
job-name=apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b
plan.upgrade.cattle.io/k3s-server=151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
upgrade.cattle.io/controller=system-upgrade-controller
upgrade.cattle.io/node=builder-hp-elitebook-745-g5
upgrade.cattle.io/plan=k3s-server
upgrade.cattle.io/version=v1.22.3-rc1-k3s1
Service Account: system-upgrade
Init Containers:
cordon:
Image: rancher/kubectl:v1.18.20
Port: <none>
Host Port: <none>
Args:
cordon
builder-hp-elitebook-745-g5
Environment:
SYSTEM_UPGRADE_NODE_NAME: (v1:spec.nodeName)
SYSTEM_UPGRADE_POD_NAME: (v1:metadata.name)
SYSTEM_UPGRADE_POD_UID: (v1:metadata.uid)
SYSTEM_UPGRADE_PLAN_NAME: k3s-server
SYSTEM_UPGRADE_PLAN_LATEST_HASH: 151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
SYSTEM_UPGRADE_PLAN_LATEST_VERSION: v1.22.3-rc1-k3s1
Mounts:
/host from host-root (rw)
/run/system-upgrade/pod from pod-info (ro)
Containers:
upgrade:
Image: rancher/k3s-upgrade:v1.22.3-rc1-k3s1
Port: <none>
Host Port: <none>
Environment:
SYSTEM_UPGRADE_NODE_NAME: (v1:spec.nodeName)
SYSTEM_UPGRADE_POD_NAME: (v1:metadata.name)
SYSTEM_UPGRADE_POD_UID: (v1:metadata.uid)
SYSTEM_UPGRADE_PLAN_NAME: k3s-server
SYSTEM_UPGRADE_PLAN_LATEST_HASH: 151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
SYSTEM_UPGRADE_PLAN_LATEST_VERSION: v1.22.3-rc1-k3s1
Mounts:
/host from host-root (rw)
/run/system-upgrade/pod from pod-info (ro)
Volumes:
host-root:
Type: HostPath (bare host directory volume)
Path: /
HostPathType: Directory
pod-info:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
Events: <none>
builder@builder-HP-EliteBook-745-G5:~$ kubectl get jobs apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b -n system-upgrade
NAME COMPLETIONS DURATION AGE
apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b 0/1 93s 93s
builder@builder-HP-EliteBook-745-G5:~$ kubectl get jobs apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b -n system-upgrade -o yaml
apiVersion: batch/v1
kind: Job
metadata:
annotations:
objectset.rio.cattle.io/applied: H4sIAAAAAAAA/+xXUW/iRhD+K9U828QGQsBSH2jgeujuCDpyrU6nKFrvjmHLetfdHZOgyP+9WkM4kxCSa58qnSIFbO98M9/MfMz4AXIkJhgxSB5AsxwhAVYUahOuOi50aNdoQ6PDtJRKoA2XRYhKEqbGrMKL7nm4OA/vJC3D+DyOwl63308hqIFcwbhHcxtHmIdlsbBMIATALTKSRl/LHB2xvIBEl0oFoFiKyvlATPoXcnJILStNizMihS1pzpbMLSGBQdZmvBe3swvsRhfduC8wjngnu0ijQW9wnnUYT2ORMQigUEy3dr4bQN/JQQI+9N5FJ+oJHos+63X72SDqnXejdr+d9c7baRu7UTbosq64YFEWibiTdjDlEMBzZG40WaNUjXzIPWw8O2aqjfAJO5Hqo2aeIiTQoHTs1Bqtk8YfXMetdrvVCS2PfZFjqAJgWhuqq3Iy/1K8QuolQ3On0YaL9QqSY7HFwS8fpBa/zjyV11B2XXrA93WLF9vxeThEKnTIjRYuZBmhDTOppVuiZz+IIqiqAFyB3KeKcZJrHCETSmqcb80gGURRACnjK5NlH2UuCZLBIADCvFCM0Fs2lfcGTfzs5Dd0cqMwa6PKHB0k3/Y/bEvjKLTGEAT19xmjZZ3a+hPOIADaFP7kSFrkZOzGQ+7tCyNCqTMDAQhzp++YFcPZxCNIwnzraoe1q1sAmUQlPmPmT9Xft0731W/tTtZ+dsZNNb4BoXm8qm6q6iYAqSVdGk1MarQHSeDGCuNVJnO28Dcs03yJ9mxVpshJJeu4Ffdb7QgCYHbhbb/bnKroTQCo101X86/z6/Gn2y+z3z8PR+Pb6ZX/N/w0hgDWTJX4zpp8z+oIQ1/Klu+lqcermqV4Aj27Gv0Q8j53+i3IXyajHwYupTiN+3E4PQj5sPFfMfw4vB7Pr2/fD+fvG/b/Wv5v8/bH+PN8cjVtOHwuwJsALDpTWu6l91AFOx1+MqWmF8WY+6e7DJ75B3BcdRaZuNJqAwnZEp/Y2VKfHf5MnRVG1CHVrT4rlZoZJfkGEhiqO7Zx9UN+VCbfB8QznXTco4PkWQJ+auCnBv5nGgjAIS+tpI0fGHhP9ULCCpZKJUnWHIAJ4QfB5XB2O/86v/3t6uoabqoACivXUuHC70Y+nmrHnpilvaMpbme50G5/71KVjtC+k9bRn5KW742jKVIdjV1LjkPOPbPpNgHP1rbl9vydsavHTNQjfTJqXk5ml4+XLPNbHG3qFx0jcNi4tvh3KS2KUWmlXsz5EkWppF5MFtrsb4/vkZdUbx5bhDmqekm4Rrsb/jkjvhzfFxad2+7S3x5ghZ6vn65WI6GrX2SMo7q0AZgCLSPjl6yJfuyreuqeHLaV//MFMGKoSf4XNt8etuvKI596OX2ZyQt72ykmDVH7sAMgUxhlFpsPp3KzJUhGedgnQfj8tw7tSu22VFmqniR2fC8d+U7HLENOviXNLi97N6R2y/swq/vyceMfRJHfKolRWau5+icAAP//fDy55DQPAAA
objectset.rio.cattle.io/id: system-upgrade-controller
objectset.rio.cattle.io/owner-gvk: upgrade.cattle.io/v1, Kind=Plan
objectset.rio.cattle.io/owner-name: k3s-server
objectset.rio.cattle.io/owner-namespace: system-upgrade
upgrade.cattle.io/ttl-seconds-after-finished: "900"
creationTimestamp: "2021-10-30T01:07:53Z"
labels:
objectset.rio.cattle.io/hash: 9f2ac612f7e407418de10c3f7b09695f3acb1dfa
plan.upgrade.cattle.io/k3s-server: 151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
upgrade.cattle.io/controller: system-upgrade-controller
upgrade.cattle.io/node: builder-hp-elitebook-745-g5
upgrade.cattle.io/plan: k3s-server
upgrade.cattle.io/version: v1.22.3-rc1-k3s1
name: apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b
namespace: system-upgrade
resourceVersion: "127447"
uid: c79b144d-9a55-44cc-bda5-1528363514c8
spec:
activeDeadlineSeconds: 900
backoffLimit: 99
completions: 1
parallelism: 1
selector:
matchLabels:
controller-uid: c79b144d-9a55-44cc-bda5-1528363514c8
template:
metadata:
creationTimestamp: null
labels:
controller-uid: c79b144d-9a55-44cc-bda5-1528363514c8
job-name: apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b
plan.upgrade.cattle.io/k3s-server: 151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
upgrade.cattle.io/controller: system-upgrade-controller
upgrade.cattle.io/node: builder-hp-elitebook-745-g5
upgrade.cattle.io/plan: k3s-server
upgrade.cattle.io/version: v1.22.3-rc1-k3s1
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- builder-hp-elitebook-745-g5
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: upgrade.cattle.io/plan
operator: In
values:
- k3s-server
topologyKey: kubernetes.io/hostname
containers:
- env:
- name: SYSTEM_UPGRADE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: SYSTEM_UPGRADE_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SYSTEM_UPGRADE_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: SYSTEM_UPGRADE_PLAN_NAME
value: k3s-server
- name: SYSTEM_UPGRADE_PLAN_LATEST_HASH
value: 151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
- name: SYSTEM_UPGRADE_PLAN_LATEST_VERSION
value: v1.22.3-rc1-k3s1
image: rancher/k3s-upgrade:v1.22.3-rc1-k3s1
imagePullPolicy: Always
name: upgrade
resources: {}
securityContext:
capabilities:
add:
- CAP_SYS_BOOT
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /host
name: host-root
- mountPath: /run/system-upgrade/pod
name: pod-info
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostIPC: true
hostNetwork: true
hostPID: true
initContainers:
- args:
- cordon
- builder-hp-elitebook-745-g5
env:
- name: SYSTEM_UPGRADE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: SYSTEM_UPGRADE_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SYSTEM_UPGRADE_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: SYSTEM_UPGRADE_PLAN_NAME
value: k3s-server
- name: SYSTEM_UPGRADE_PLAN_LATEST_HASH
value: 151067306dc1d8a648f906540282f652b2e40f94a4d7a0f0d13b3ebc
- name: SYSTEM_UPGRADE_PLAN_LATEST_VERSION
value: v1.22.3-rc1-k3s1
image: rancher/kubectl:v1.18.20
imagePullPolicy: Always
name: cordon
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /host
name: host-root
- mountPath: /run/system-upgrade/pod
name: pod-info
readOnly: true
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: system-upgrade
serviceAccountName: system-upgrade
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
volumes:
- hostPath:
path: /
type: Directory
name: host-root
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: pod-info
ttlSecondsAfterFinished: 900
status:
startTime: "2021-10-30T01:07:53Z"
And as the process happens, we can continue to check status of the jobs and node:
builder@builder-HP-EliteBook-745-G5:~$ kubectl get jobs apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b -n system-upgrade
NAME COMPLETIONS DURATION AGE
apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b 0/1 105s 105s
builder@builder-HP-EliteBook-745-G5:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
builder-hp-elitebook-745-g5 Ready control-plane,master 2d1h v1.21.5+k3s1
builder@builder-HP-EliteBook-745-G5:~$ kubectl get jobs --all-namespaces
NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system helm-install-traefik 1/1 19s 2d1h
system-upgrade apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b 0/1 4m55s 4m55s
NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system helm-install-traefik 1/1 19s 2d1h
system-upgrade apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b 0/1 9m45s 9m45s
builder@builder-HP-EliteBook-745-G5:~$ kubectl get jobs --all-namespaces
NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system helm-install-traefik 1/1 19s 2d1h
system-upgrade apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b 0/1 22m 22m
builder@builder-HP-EliteBook-745-G5:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
builder-hp-elitebook-745-g5 Ready control-plane,master 2d1h v1.21.5+k3s1
I knew something finally happened when my kubectl responses were rejected:
builder@builder-HP-EliteBook-745-G5:~$ kubectl logs system-upgrade-controller-85b58fbc5f-b6764 -n system-upgrade
WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions
error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied
I quick fixed the perms and tried again:
builder@builder-HP-EliteBook-745-G5:~$ sudo chmod a+rx /etc/rancher/k3s
[sudo] password for builder:
builder@builder-HP-EliteBook-745-G5:~$ sudo chmod a+r /etc/rancher/k3s/k3s.yaml
And saw the job finally had completed:
builder@builder-HP-EliteBook-745-G5:~$ kubectl get jobs --all-namespaces
NAMESPACE NAME COMPLETIONS DURATION AGE
system-upgrade apply-k3s-server-on-builder-hp-elitebook-745-g5-with-1510-6488b 1/1 36s 3m23s
kube-system helm-install-traefik 1/1 22s 3m
and our Node is now the new version
builder@builder-HP-EliteBook-745-G5:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
builder-hp-elitebook-745-g5 Ready control-plane,master 2d2h v1.22.3-rc1+k3s1
I did see some errors on the upgrade pod:
$ kubectl logs system-upgrade-controller-85b58fbc5f-b6764 -n system-upgrade | tail -n10
E1030 01:38:27.032464 1 reflector.go:178] k8s.io/client-go@v1.18.0-k3s.1/tools/cache/reflector.go:125: Failed to list *v1.Plan: an error on the server ("apiserver not ready") has prevented the request from succeeding (get plans.meta.k8s.io)
I1030 01:38:27.033843 1 trace.go:116] Trace[1499268105]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.18.0-k3s.1/tools/cache/reflector.go:125 (started: 2021-10-30 01:38:12.029042049 +0000 UTC m=+179464.469007321) (total time: 15.004765082s):
Trace[1499268105]: [15.004765082s] [15.004765082s] END
E1030 01:38:27.033866 1 reflector.go:178] k8s.io/client-go@v1.18.0-k3s.1/tools/cache/reflector.go:125: Failed to list *v1.Job: an error on the server ("apiserver not ready") has prevented the request from succeeding (get jobs.meta.k8s.io)
I1030 01:38:27.033878 1 trace.go:116] Trace[406771844]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.18.0-k3s.1/tools/cache/reflector.go:125 (started: 2021-10-30 01:38:12.031850557 +0000 UTC m=+179464.471815669) (total time: 15.001994391s):
Trace[406771844]: [15.001994391s] [15.001994391s] END
E1030 01:38:27.033895 1 reflector.go:178] k8s.io/client-go@v1.18.0-k3s.1/tools/cache/reflector.go:125: Failed to list *v1.Node: an error on the server ("apiserver not ready") has prevented the request from succeeding (get nodes.meta.k8s.io)
I1030 01:38:27.034970 1 trace.go:116] Trace[196119233]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.18.0-k3s.1/tools/cache/reflector.go:125 (started: 2021-10-30 01:38:12.027816345 +0000 UTC m=+179464.467781708) (total time: 15.007104388s):
Trace[196119233]: [15.007104388s] [15.007104388s] END
E1030 01:38:27.034999 1 reflector.go:178] k8s.io/client-go@v1.18.0-k3s.1/tools/cache/reflector.go:125: Failed to list *v1.Secret: an error on the server ("apiserver not ready") has prevented the request from succeeding (get secrets.meta.k8s.io)
But the errors did not seem to affect the upgrade itself.
Checking the ingress:
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-v1betatest <none> * 192.168.1.159 80 41m
And we see it was converted to networking from v1beta1
$ kubectl get ingress -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-v1betatest","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"nginx-service","servicePort":80},"path":"/path1"}]}}]}}
creationTimestamp: "2021-10-30T01:01:05Z"
generation: 1
name: ingress-v1betatest
namespace: default
resourceVersion: "127095"
uid: f0b7c8a3-b23d-45ca-9500-b748d1ec596d
spec:
rules:
- http:
paths:
- backend:
service:
name: nginx-service
port:
number: 80
path: /path1
pathType: ImplementationSpecific
status:
loadBalancer:
ingress:
- ip: 192.168.1.159
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Interestingly when I checked the webserver, Nginx had taken over as the Ingress controller:
$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 2d2h
kube-system traefik-prometheus ClusterIP 10.43.97.166 <none> 9100/TCP 2d2h
default nginx-service ClusterIP 10.43.174.8 <none> 80/TCP 43m
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 2d2h
kube-system metrics-server ClusterIP 10.43.106.155 <none> 443/TCP 2d2h
kube-system traefik LoadBalancer 10.43.222.161 <pending> 80:31400/TCP,443:31099/TCP 2d2h
default my-release-nginx-ingress LoadBalancer 10.43.141.8 192.168.1.159 80:31933/TCP,443:32383/TCP 3h58m
But because we never explicitly set a handler in our Ingress, Our ingress request does not actually care which controller services it.
k3s in Terraform (not working)
I tried multiple runs with Terraform and the k3s module to provision on-prem hardware:
$ terraform plan -out tf.plan
module.k3s.random_password.k3s_cluster_secret: Refreshing state... [id=none]
module.k3s.tls_private_key.master_user[0]: Refreshing state... [id=06c54c3b737e0a8aff52f77584937576fe06a469]
module.k3s.tls_private_key.kubernetes_ca[2]: Refreshing state... [id=652d60b5f4d92d1f75f731b792f7dc853b5376ea]
module.k3s.tls_private_key.kubernetes_ca[0]: Refreshing state... [id=9439143ccc493e939f6f9d0cc4e88a737aad8e8b]
module.k3s.tls_private_key.kubernetes_ca[1]: Refreshing state... [id=1f4059d9261c921cdef8e3deefbb22af035e2d20]
module.k3s.tls_cert_request.master_user[0]: Refreshing state... [id=d4db109d0cd07ea192541cc36ec61cdbd4cae6ca]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Refreshing state... [id=168650678377305405053929084634561712060]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Refreshing state... [id=166863271464135357871740700137491090258]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Refreshing state... [id=236627667009521719126057669130413543445]
module.k3s.tls_locally_signed_cert.master_user[0]: Refreshing state... [id=147110137514903271286127930568225412108]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Refreshing state... [id=822643167695070475]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Refreshing state... [id=7180920018107439379]
module.k3s.null_resource.k8s_ca_certificates_install[2]: Refreshing state... [id=6746882777759482513]
module.k3s.null_resource.k8s_ca_certificates_install[0]: Refreshing state... [id=6465958346775660162]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Refreshing state... [id=751239155582439824]
module.k3s.null_resource.k8s_ca_certificates_install[4]: Refreshing state... [id=2521296445112125164]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
- destroy
Terraform will perform the following actions:
# module.k3s.null_resource.k8s_ca_certificates_install[0] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "6465958346775660162" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[1] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "751239155582439824" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[2] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "6746882777759482513" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[3] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "822643167695070475" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[4] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "2521296445112125164" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[5] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "7180920018107439379" -> null
}
# module.k3s.null_resource.kubernetes_ready will be created
+ resource "null_resource" "kubernetes_ready" {
+ id = (known after apply)
}
# module.k3s.null_resource.servers_drain["server-one"] will be created
+ resource "null_resource" "servers_drain" {
+ id = (known after apply)
+ triggers = {
+ "connection_json" = "eyJhZ2VudCI6bnVsbCwiYWdlbnRfaWRlbnRpdHkiOm51bGwsImJhc3Rpb25fY2VydGlmaWNhdGUiOm51bGwsImJhc3Rpb25faG9zdCI6bnVsbCwiYmFzdGlvbl9ob3N0X2tleSI6bnVsbCwiYmFzdGlvbl9wYXNzd29yZCI6bnVsbCwiYmFzdGlvbl9wb3J0IjpudWxsLCJiYXN0aW9uX3ByaXZhdGVfa2V5IjpudWxsLCJiYXN0aW9uX3VzZXIiOm51bGwsImNhY2VydCI6bnVsbCwiY2VydGlmaWNhdGUiOm51bGwsImhvc3QiOiIxOTIuMTY4LjEuMTU5IiwiaG9zdF9rZXkiOiIvaG9tZS9idWlsZGVyLy5zc2gvaWRfcnNhLnB1YiIsImh0dHBzIjpudWxsLCJpbnNlY3VyZSI6bnVsbCwicGFzc3dvcmQiOm51bGwsInBvcnQiOm51bGwsInByaXZhdGVfa2V5IjpudWxsLCJzY3JpcHRfcGF0aCI6bnVsbCwidGltZW91dCI6bnVsbCwidHlwZSI6InNzaCIsInVzZV9udGxtIjpudWxsLCJ1c2VyIjoiYnVpbGRlciJ9"
+ "drain_timeout" = "30s"
+ "server_name" = "server-one"
}
}
# module.k3s.null_resource.servers_install["server-one"] will be created
+ resource "null_resource" "servers_install" {
+ id = (known after apply)
+ triggers = {
+ "on_immutable_changes" = "2dc596d87cd024ad552e7c4ba7dcb41aeb3fd6ea"
+ "on_new_version" = "v1.0.0"
}
}
# module.k3s.null_resource.servers_label["server-one|node.kubernetes.io/type"] will be created
+ resource "null_resource" "servers_label" {
+ id = (known after apply)
+ triggers = {
+ "connection_json" = "eyJhZ2VudCI6bnVsbCwiYWdlbnRfaWRlbnRpdHkiOm51bGwsImJhc3Rpb25fY2VydGlmaWNhdGUiOm51bGwsImJhc3Rpb25faG9zdCI6bnVsbCwiYmFzdGlvbl9ob3N0X2tleSI6bnVsbCwiYmFzdGlvbl9wYXNzd29yZCI6bnVsbCwiYmFzdGlvbl9wb3J0IjpudWxsLCJiYXN0aW9uX3ByaXZhdGVfa2V5IjpudWxsLCJiYXN0aW9uX3VzZXIiOm51bGwsImNhY2VydCI6bnVsbCwiY2VydGlmaWNhdGUiOm51bGwsImhvc3QiOiIxOTIuMTY4LjEuMTU5IiwiaG9zdF9rZXkiOiIvaG9tZS9idWlsZGVyLy5zc2gvaWRfcnNhLnB1YiIsImh0dHBzIjpudWxsLCJpbnNlY3VyZSI6bnVsbCwicGFzc3dvcmQiOm51bGwsInBvcnQiOm51bGwsInByaXZhdGVfa2V5IjpudWxsLCJzY3JpcHRfcGF0aCI6bnVsbCwidGltZW91dCI6bnVsbCwidHlwZSI6InNzaCIsInVzZV9udGxtIjpudWxsLCJ1c2VyIjoiYnVpbGRlciJ9"
+ "label_name" = "node.kubernetes.io/type"
+ "on_value_changes" = "master"
+ "server_name" = "server-one"
}
}
# module.k3s.tls_cert_request.master_user[0] will be destroyed
- resource "tls_cert_request" "master_user" {
- cert_request_pem = <<-EOT
-----BEGIN CERTIFICATE REQUEST-----
MIIBJjCBrgIBADAvMRcwFQYDVQQKEw5zeXN0ZW06bWFzdGVyczEUMBIGA1UEAxML
bWFzdGVyLXVzZXIwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASnI5ylVELX3yF6jlas
SSchEM8khh6vaxP/GIkCga0MS1MzT+LV4FKPxKjX9tN5OR5F/wrJ7Ggl2p+c8+ZD
Uj8BgTWNUsdfcRgPTiGkLqAt6rZTulvL2ACeFqO/1xAEZBmgADAKBggqhkjOPQQD
AwNnADBkAjBAAD+vleMn2iNwbsvPdLYzoIvqLxYDvj7lUHUnsjVY6WMQdHxYpKaA
UBEn4KAiqHACMG/OX0JVaLCkCi4b3kTDX+hRR/aupV4PJQKOtZTZn8GtI2YGOLQY
rpW7lmgS+y4KJQ==
-----END CERTIFICATE REQUEST-----
EOT -> null
- id = "d4db109d0cd07ea192541cc36ec61cdbd4cae6ca" -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- subject {
- common_name = "master-user" -> null
- organization = "system:masters" -> null
- street_address = [] -> null
}
}
# module.k3s.tls_locally_signed_cert.master_user[0] will be destroyed
- resource "tls_locally_signed_cert" "master_user" {
- allowed_uses = [
- "key_encipherment",
- "digital_signature",
- "client_auth",
] -> null
- ca_cert_pem = "52fea89ecedbd5cef88d97f5120e226fb2026695" -> null
- ca_key_algorithm = "ECDSA" -> null
- ca_private_key_pem = (sensitive value)
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIB4TCCAWegAwIBAgIQbqxhw/SB0wl2fKytLxVUDDAKBggqhkjOPQQDAzAfMR0w
GwYDVQQDExRrdWJlcm5ldGVzLWNsaWVudC1jYTAgFw0yMTEwMzExNDMzMzdaGA8y
MTIxMTEwMTE0MzMzN1owLzEXMBUGA1UEChMOc3lzdGVtOm1hc3RlcnMxFDASBgNV
BAMTC21hc3Rlci11c2VyMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEpyOcpVRC198h
eo5WrEknIRDPJIYer2sT/xiJAoGtDEtTM0/i1eBSj8So1/bTeTkeRf8KyexoJdqf
nPPmQ1I/AYE1jVLHX3EYD04hpC6gLeq2U7pby9gAnhajv9cQBGQZo1YwVDAOBgNV
HQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAf
BgNVHSMEGDAWgBRh1LWdfpqAFlMbYsJ/fA1hRga2TjAKBggqhkjOPQQDAwNoADBl
AjBlK2+CErGdtb/gUqq2REXfEnwrAbvcxBjuRSdNNtmzBDaafeHVpu1WiCYIoijL
Z4cCMQDicw18wxRgoyiTKDOsOLFFKbqlKsdOvGSmZA3N8kRMyMpNIzREPyrtbB3O
ubBeXFk=
-----END CERTIFICATE-----
EOT -> null
- cert_request_pem = "7e813333339796598fb6b4dada1867c115a30bbe" -> null
- early_renewal_hours = 0 -> null
- id = "147110137514903271286127930568225412108" -> null
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-01T09:33:37.859226051-05:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-10-31T09:33:37.859226051-05:00" -> null
}
# module.k3s.tls_private_key.kubernetes_ca[0] will be destroyed
- resource "tls_private_key" "kubernetes_ca" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "9439143ccc493e939f6f9d0cc4e88a737aad8e8b" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "e5:63:b5:47:6a:17:d4:89:6b:f7:ea:fc:a4:40:af:83" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBKb/WSaldfNKwco2BxXLXFWSVaPcXf5RDrg43MRmKhT/+eWoqH85f64SEnnBZ4iFqCV/ArVSnOho6eT0wSuCIUlPSrVhwemi1JE4Y6Ay9VBDBxslTycnaBPtBTuPFX8gKw==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEpv9ZJqV180rByjYHFctcVZJVo9xd/lEO
uDjcxGYqFP/55aiofzl/rhISecFniIWoJX8CtVKc6Gjp5PTBK4IhSU9KtWHB6aLU
kThjoDL1UEMHGyVPJydoE+0FO48VfyAr
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_private_key.kubernetes_ca[1] will be destroyed
- resource "tls_private_key" "kubernetes_ca" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "1f4059d9261c921cdef8e3deefbb22af035e2d20" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "e6:d9:74:01:9d:1d:6b:1d:fe:f7:ff:58:3e:79:77:40" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBENV3UV35Ul5T9OsGlT2MWajQxV6TtAus3HuLgHXR6jsCMfXip1dUSRAI3+uOBpPkJusOkBv9xIr02HqYMGGUGl4sNaSS2oNhDwxXjAQO4D3LSRC42KXorXjZz6jA60QPQ==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEQ1XdRXflSXlP06waVPYxZqNDFXpO0C6z
ce4uAddHqOwIx9eKnV1RJEAjf644Gk+Qm6w6QG/3EivTYepgwYZQaXiw1pJLag2E
PDFeMBA7gPctJELjYpeiteNnPqMDrRA9
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_private_key.kubernetes_ca[2] will be destroyed
- resource "tls_private_key" "kubernetes_ca" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "652d60b5f4d92d1f75f731b792f7dc853b5376ea" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "93:1e:1f:4a:9a:ba:50:17:b6:41:b3:78:4d:d3:08:9f" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBBKbFrYPGeiZ8rwV95lqL7ZPakccjGATcW3db4QE+ABt4XYPJ0QQtnOd6rJfV9Cyhmk0Sf7IzuNKTS58ybcsvTTIXDWbhZj+S2Scrl4dy6c83cCnIBKEGq6Z+vPwif4NZg==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEEpsWtg8Z6JnyvBX3mWovtk9qRxyMYBNx
bd1vhAT4AG3hdg8nRBC2c53qsl9X0LKGaTRJ/sjO40pNLnzJtyy9NMhcNZuFmP5L
ZJyuXh3LpzzdwKcgEoQarpn68/CJ/g1m
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_private_key.master_user[0] will be destroyed
- resource "tls_private_key" "master_user" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "06c54c3b737e0a8aff52f77584937576fe06a469" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "8b:9e:c5:8e:f2:82:2c:61:5c:7e:db:1a:f3:70:b2:9d" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBKcjnKVUQtffIXqOVqxJJyEQzySGHq9rE/8YiQKBrQxLUzNP4tXgUo/EqNf203k5HkX/CsnsaCXan5zz5kNSPwGBNY1Sx19xGA9OIaQuoC3qtlO6W8vYAJ4Wo7/XEARkGQ==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEpyOcpVRC198heo5WrEknIRDPJIYer2sT
/xiJAoGtDEtTM0/i1eBSj8So1/bTeTkeRf8KyexoJdqfnPPmQ1I/AYE1jVLHX3EY
D04hpC6gLeq2U7pby9gAnhajv9cQBGQZ
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"] will be destroyed
- resource "tls_self_signed_cert" "kubernetes_ca_certs" {
- allowed_uses = [
- "critical",
- "digitalSignature",
- "keyEncipherment",
- "keyCertSign",
] -> null
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIBrjCCATSgAwIBAgIRALIE1LmPOgiw+vnsQPhmEBUwCgYIKoZIzj0EAwMwHzEd
MBsGA1UEAxMUa3ViZXJuZXRlcy1jbGllbnQtY2EwIBcNMjExMDMxMTQzMzM3WhgP
MjEyMTExMDExNDMzMzdaMB8xHTAbBgNVBAMTFGt1YmVybmV0ZXMtY2xpZW50LWNh
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEpv9ZJqV180rByjYHFctcVZJVo9xd/lEO
uDjcxGYqFP/55aiofzl/rhISecFniIWoJX8CtVKc6Gjp5PTBK4IhSU9KtWHB6aLU
kThjoDL1UEMHGyVPJydoE+0FO48VfyArozIwMDAPBgNVHRMBAf8EBTADAQH/MB0G
A1UdDgQWBBRh1LWdfpqAFlMbYsJ/fA1hRga2TjAKBggqhkjOPQQDAwNoADBlAjBR
NlzxHcDlRGapH5gqbHN9Jh5S8gdyCJNWT48HMdzI7UCcNBY9DNIkvboQonCz0KUC
MQCFYmdNKQt3cCrvIxgKvrEX1tvV3eH7gtDxvUEZpz3Dc4e+QpU63rqCGSgVk5ZD
EWw=
-----END CERTIFICATE-----
EOT -> null
- early_renewal_hours = 0 -> null
- id = "236627667009521719126057669130413543445" -> null
- is_ca_certificate = true -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-01T09:33:37.823487581-05:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-10-31T09:33:37.823487581-05:00" -> null
- subject {
- common_name = "kubernetes-client-ca" -> null
- street_address = [] -> null
}
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"] will be destroyed
- resource "tls_self_signed_cert" "kubernetes_ca_certs" {
- allowed_uses = [
- "critical",
- "digitalSignature",
- "keyEncipherment",
- "keyCertSign",
] -> null
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIBrjCCATOgAwIBAgIQfYiydgFhoN/ErdbL5ZlLUjAKBggqhkjOPQQDAzAfMR0w
GwYDVQQDExRrdWJlcm5ldGVzLXNlcnZlci1jYTAgFw0yMTEwMzExNDMzMzdaGA8y
MTIxMTEwMTE0MzMzN1owHzEdMBsGA1UEAxMUa3ViZXJuZXRlcy1zZXJ2ZXItY2Ew
djAQBgcqhkjOPQIBBgUrgQQAIgNiAARDVd1Fd+VJeU/TrBpU9jFmo0MVek7QLrNx
7i4B10eo7AjH14qdXVEkQCN/rjgaT5CbrDpAb/cSK9Nh6mDBhlBpeLDWkktqDYQ8
MV4wEDuA9y0kQuNil6K142c+owOtED2jMjAwMA8GA1UdEwEB/wQFMAMBAf8wHQYD
VR0OBBYEFB7ICCTNAZfXnY6CLvYEUiG0djyBMAoGCCqGSM49BAMDA2kAMGYCMQDk
w7HDSRvlI7VdzvSbqmISAgsJiX97oxor6cBGqwPP9uqQNQuc7Hz9zHVQveFs2FIC
MQCPQjF2BQNvCqzQtOpD+k6BemLod+PQ1kHzkbFxMkRQLxBn4ltyTrnQN7IL2L15
bCs=
-----END CERTIFICATE-----
EOT -> null
- early_renewal_hours = 0 -> null
- id = "166863271464135357871740700137491090258" -> null
- is_ca_certificate = true -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-01T09:33:37.822034333-05:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-10-31T09:33:37.822034333-05:00" -> null
- subject {
- common_name = "kubernetes-server-ca" -> null
- street_address = [] -> null
}
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"] will be destroyed
- resource "tls_self_signed_cert" "kubernetes_ca_certs" {
- allowed_uses = [
- "critical",
- "digitalSignature",
- "keyEncipherment",
- "keyCertSign",
] -> null
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIBxDCCAUugAwIBAgIQfuDwbPlHJPbeKSfgTScPvDAKBggqhkjOPQQDAzArMSkw
JwYDVQQDEyBrdWJlcm5ldGVzLXJlcXVlc3QtaGVhZGVyLWtleS1jYTAgFw0yMTEw
MzExNDMzMzdaGA8yMTIxMTEwMTE0MzMzN1owKzEpMCcGA1UEAxMga3ViZXJuZXRl
cy1yZXF1ZXN0LWhlYWRlci1rZXktY2EwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQS
mxa2DxnomfK8FfeZai+2T2pHHIxgE3Ft3W+EBPgAbeF2DydEELZzneqyX1fQsoZp
NEn+yM7jSk0ufMm3LL00yFw1m4WY/ktknK5eHcunPN3ApyAShBqumfrz8In+DWaj
MjAwMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFMCbZjGuN3oyiY/zpQ9l8yB1
QmiBMAoGCCqGSM49BAMDA2cAMGQCMBZBw39j+hSIjh3Pxntr9iPeL5+CU9BYsK1L
5KUd4478ZDr4wIO7lCkuZFS6Gg5YNwIwZrNBq6NzJmYAShmPZeRCFcmOMy+XInsZ
ZcA7XyKDKH9yUIHrJ++zr8nEPPyUd8bI
-----END CERTIFICATE-----
EOT -> null
- early_renewal_hours = 0 -> null
- id = "168650678377305405053929084634561712060" -> null
- is_ca_certificate = true -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-01T09:33:37.824259526-05:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-10-31T09:33:37.824259526-05:00" -> null
- subject {
- common_name = "kubernetes-request-header-key-ca" -> null
- street_address = [] -> null
}
}
Plan: 4 to add, 0 to change, 15 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tf.plan
To perform exactly these actions, run the following command to apply:
terraform apply "tf.plan"
builder@DESKTOP-QADGF36:~/Workspaces/tfk3s$ terraform apply "tf.plan"
module.k3s.null_resource.k8s_ca_certificates_install[4]: Destroying... [id=2521296445112125164]
module.k3s.null_resource.k8s_ca_certificates_install[2]: Destroying... [id=6746882777759482513]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Destroying... [id=751239155582439824]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Destroying... [id=822643167695070475]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Destroying... [id=7180920018107439379]
module.k3s.null_resource.k8s_ca_certificates_install[2]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[3]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[0]: Destroying... [id=6465958346775660162]
module.k3s.tls_locally_signed_cert.master_user[0]: Destroying... [id=147110137514903271286127930568225412108]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[5]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[4]: Destruction complete after 0s
module.k3s.tls_locally_signed_cert.master_user[0]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[0]: Destruction complete after 0s
module.k3s.tls_cert_request.master_user[0]: Destroying... [id=d4db109d0cd07ea192541cc36ec61cdbd4cae6ca]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Destroying... [id=166863271464135357871740700137491090258]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Destroying... [id=236627667009521719126057669130413543445]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Destroying... [id=168650678377305405053929084634561712060]
module.k3s.tls_cert_request.master_user[0]: Destruction complete after 0s
module.k3s.null_resource.servers_install["server-one"]: Creating...
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Destruction complete after 0s
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Destruction complete after 0s
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Destruction complete after 0s
module.k3s.tls_private_key.master_user[0]: Destroying... [id=06c54c3b737e0a8aff52f77584937576fe06a469]
module.k3s.tls_private_key.master_user[0]: Destruction complete after 0s
module.k3s.tls_private_key.kubernetes_ca[0]: Destroying... [id=9439143ccc493e939f6f9d0cc4e88a737aad8e8b]
module.k3s.tls_private_key.kubernetes_ca[0]: Destruction complete after 0s
module.k3s.tls_private_key.kubernetes_ca[2]: Destroying... [id=652d60b5f4d92d1f75f731b792f7dc853b5376ea]
module.k3s.tls_private_key.kubernetes_ca[1]: Destroying... [id=1f4059d9261c921cdef8e3deefbb22af035e2d20]
module.k3s.null_resource.servers_install["server-one"]: Provisioning with 'file'...
module.k3s.tls_private_key.kubernetes_ca[2]: Destruction complete after 0s
module.k3s.tls_private_key.kubernetes_ca[1]: Destruction complete after 0s
╷
│ Error: file provisioner error
│
│ with module.k3s.null_resource.servers_install["server-one"],
│ on .terraform/modules/k3s/server_nodes.tf line 200, in resource "null_resource" "servers_install":
│ 200: provisioner "file" {
│
│ knownhosts: /tmp/tf-known_hosts825875476:1: knownhosts: missing key type pattern
╵
and again
builder@DESKTOP-QADGF36:~/Workspaces/tfk3s$ terraform plan -out tf.plan
module.k3s.random_password.k3s_cluster_secret: Refreshing state... [id=none]
module.k3s.null_resource.servers_install["server-one"]: Refreshing state... [id=1316345895109891785]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.k3s.null_resource.kubernetes_ready will be created
+ resource "null_resource" "kubernetes_ready" {
+ id = (known after apply)
}
# module.k3s.null_resource.servers_drain["server-one"] will be created
+ resource "null_resource" "servers_drain" {
+ id = (known after apply)
+ triggers = {
+ "connection_json" = "eyJhZ2VudCI6bnVsbCwiYWdlbnRfaWRlbnRpdHkiOm51bGwsImJhc3Rpb25fY2VydGlmaWNhdGUiOm51bGwsImJhc3Rpb25faG9zdCI6bnVsbCwiYmFzdGlvbl9ob3N0X2tleSI6bnVsbCwiYmFzdGlvbl9wYXNzd29yZCI6bnVsbCwiYmFzdGlvbl9wb3J0IjpudWxsLCJiYXN0aW9uX3ByaXZhdGVfa2V5IjpudWxsLCJiYXN0aW9uX3VzZXIiOm51bGwsImNhY2VydCI6bnVsbCwiY2VydGlmaWNhdGUiOm51bGwsImhvc3QiOiIxOTIuMTY4LjEuMTU5IiwiaG9zdF9rZXkiOm51bGwsImh0dHBzIjpudWxsLCJpbnNlY3VyZSI6bnVsbCwicGFzc3dvcmQiOm51bGwsInBvcnQiOm51bGwsInByaXZhdGVfa2V5IjpudWxsLCJzY3JpcHRfcGF0aCI6bnVsbCwidGltZW91dCI6bnVsbCwidHlwZSI6InNzaCIsInVzZV9udGxtIjpudWxsLCJ1c2VyIjoiYnVpbGRlciJ9"
+ "drain_timeout" = "30s"
+ "server_name" = "server-one"
}
}
# module.k3s.null_resource.servers_install["server-one"] is tainted, so must be replaced
-/+ resource "null_resource" "servers_install" {
~ id = "1316345895109891785" -> (known after apply)
# (1 unchanged attribute hidden)
}
# module.k3s.null_resource.servers_label["server-one|node.kubernetes.io/type"] will be created
+ resource "null_resource" "servers_label" {
+ id = (known after apply)
+ triggers = {
+ "connection_json" = "eyJhZ2VudCI6bnVsbCwiYWdlbnRfaWRlbnRpdHkiOm51bGwsImJhc3Rpb25fY2VydGlmaWNhdGUiOm51bGwsImJhc3Rpb25faG9zdCI6bnVsbCwiYmFzdGlvbl9ob3N0X2tleSI6bnVsbCwiYmFzdGlvbl9wYXNzd29yZCI6bnVsbCwiYmFzdGlvbl9wb3J0IjpudWxsLCJiYXN0aW9uX3ByaXZhdGVfa2V5IjpudWxsLCJiYXN0aW9uX3VzZXIiOm51bGwsImNhY2VydCI6bnVsbCwiY2VydGlmaWNhdGUiOm51bGwsImhvc3QiOiIxOTIuMTY4LjEuMTU5IiwiaG9zdF9rZXkiOm51bGwsImh0dHBzIjpudWxsLCJpbnNlY3VyZSI6bnVsbCwicGFzc3dvcmQiOm51bGwsInBvcnQiOm51bGwsInByaXZhdGVfa2V5IjpudWxsLCJzY3JpcHRfcGF0aCI6bnVsbCwidGltZW91dCI6bnVsbCwidHlwZSI6InNzaCIsInVzZV9udGxtIjpudWxsLCJ1c2VyIjoiYnVpbGRlciJ9"
+ "label_name" = "node.kubernetes.io/type"
+ "on_value_changes" = "master"
+ "server_name" = "server-one"
}
}
Plan: 4 to add, 0 to change, 1 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tf.plan
To perform exactly these actions, run the following command to apply:
terraform apply "tf.plan"
builder@DESKTOP-QADGF36:~/Workspaces/tfk3s$ terraform apply "tf.plan"
module.k3s.null_resource.servers_install["server-one"]: Destroying... [id=1316345895109891785]
module.k3s.null_resource.servers_install["server-one"]: Destruction complete after 0s
module.k3s.null_resource.servers_install["server-one"]: Creating...
module.k3s.null_resource.servers_install["server-one"]: Provisioning with 'file'...
module.k3s.null_resource.servers_install["server-one"]: Still creating... [10s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [20s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [30s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [40s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [50s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [1m0s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [1m10s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [1m20s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [1m30s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [1m40s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [1m50s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [2m0s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [2m10s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [2m20s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [2m30s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [2m40s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [2m50s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [3m0s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [3m10s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [3m20s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [3m30s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [3m40s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [3m50s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [4m0s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [4m10s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [4m20s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [4m30s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [4m40s elapsed]
module.k3s.null_resource.servers_install["server-one"]: Still creating... [4m50s elapsed]
╷
│ Error: file provisioner error
│
│ with module.k3s.null_resource.servers_install["server-one"],
│ on .terraform/modules/k3s/server_nodes.tf line 200, in resource "null_resource" "servers_install":
│ 200: provisioner "file" {
│
│ timeout - last error: SSH authentication failed (builder@192.168.1.159:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods
│ remain
╵
but nothing would create directly on my local ubuntu hosts with terraform. I assumed it needed kubespray/adm/ansible root ssh passwordless perms, but that did not seem to suffice.
CIVO Terraform and K3s
Using Terraform for k3s is doable. There are a few community terraform modules one can use. The one I’ll cover here that I was most successful with is called k3s and on github https://github.com/xunleii/terraform-module-k3s.
builder@DESKTOP-QADGF36:~/Workspaces/terraform-module-k3s/examples/civo-k3s$ terraform apply -var-file="tf.vars"
module.k3s.random_password.k3s_cluster_secret: Refreshing state... [id=none]
module.k3s.tls_private_key.kubernetes_ca[2]: Refreshing state... [id=80ca4a9bebd3fa1437efef27571e4927e873a146]
module.k3s.tls_private_key.master_user[0]: Refreshing state... [id=56e5b1cf3fb10387213633492993251aed8a2c5c]
module.k3s.tls_private_key.kubernetes_ca[0]: Refreshing state... [id=ac1d6302a69b6a95ceaa0ab3513a0befc52eae1b]
module.k3s.tls_private_key.kubernetes_ca[1]: Refreshing state... [id=fb31ae6ed4b0cda025947155d636f41acc97932a]
module.k3s.tls_cert_request.master_user[0]: Refreshing state... [id=5457fd7b6ff35fb465293f5ac48cf51f73f604da]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Refreshing state... [id=179381637326519421633541131708977756020]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Refreshing state... [id=162007174841491717249967010213281523901]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Refreshing state... [id=240239317385907192679446816919634150104]
module.k3s.tls_locally_signed_cert.master_user[0]: Refreshing state... [id=269975477633753601419630313929603900630]
civo_instance.node_instances[1]: Refreshing state... [id=27cf0c45-25d9-47a8-abda-e0616fa68d9f]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the last "terraform apply":
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"] has been changed
~ resource "tls_self_signed_cert" "kubernetes_ca_certs" {
id = "240239317385907192679446816919634150104"
# (10 unchanged attributes hidden)
~ subject {
+ street_address = []
# (1 unchanged attribute hidden)
}
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"] has been changed
~ resource "tls_self_signed_cert" "kubernetes_ca_certs" {
id = "179381637326519421633541131708977756020"
# (10 unchanged attributes hidden)
~ subject {
+ street_address = []
# (1 unchanged attribute hidden)
}
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"] has been changed
~ resource "tls_self_signed_cert" "kubernetes_ca_certs" {
id = "162007174841491717249967010213281523901"
# (10 unchanged attributes hidden)
~ subject {
+ street_address = []
# (1 unchanged attribute hidden)
}
}
# module.k3s.tls_cert_request.master_user[0] has been changed
~ resource "tls_cert_request" "master_user" {
id = "5457fd7b6ff35fb465293f5ac48cf51f73f604da"
# (3 unchanged attributes hidden)
~ subject {
+ street_address = []
# (2 unchanged attributes hidden)
}
}
# civo_instance.node_instances[1] has been changed
~ resource "civo_instance" "node_instances" {
id = "27cf0c45-25d9-47a8-abda-e0616fa68d9f"
+ tags = []
# (17 unchanged attributes hidden)
}
Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# civo_instance.node_instances[0] will be created
+ resource "civo_instance" "node_instances" {
+ cpu_cores = (known after apply)
+ created_at = (known after apply)
+ disk_gb = (known after apply)
+ disk_image = (known after apply)
+ firewall_id = (known after apply)
+ hostname = "node-1"
+ id = (known after apply)
+ initial_password = (sensitive value)
+ initial_user = "civo"
+ network_id = (known after apply)
+ private_ip = (known after apply)
+ public_ip = (known after apply)
+ public_ip_required = "create"
+ ram_mb = (known after apply)
+ size = "g3.small"
+ source_id = (known after apply)
+ source_type = (known after apply)
+ status = (known after apply)
+ template = "8eb48e20-e5db-49fe-9cdf-cc8f381c61c6"
}
# civo_instance.node_instances[2] will be created
+ resource "civo_instance" "node_instances" {
+ cpu_cores = (known after apply)
+ created_at = (known after apply)
+ disk_gb = (known after apply)
+ disk_image = (known after apply)
+ firewall_id = (known after apply)
+ hostname = "node-3"
+ id = (known after apply)
+ initial_password = (sensitive value)
+ initial_user = "civo"
+ network_id = (known after apply)
+ private_ip = (known after apply)
+ public_ip = (known after apply)
+ public_ip_required = "create"
+ ram_mb = (known after apply)
+ size = "g3.small"
+ source_id = (known after apply)
+ source_type = (known after apply)
+ status = (known after apply)
+ template = "8eb48e20-e5db-49fe-9cdf-cc8f381c61c6"
}
# module.k3s.null_resource.k8s_ca_certificates_install[0] will be created
+ resource "null_resource" "k8s_ca_certificates_install" {
+ id = (known after apply)
}
# module.k3s.null_resource.k8s_ca_certificates_install[1] will be created
+ resource "null_resource" "k8s_ca_certificates_install" {
+ id = (known after apply)
}
# module.k3s.null_resource.k8s_ca_certificates_install[2] will be created
+ resource "null_resource" "k8s_ca_certificates_install" {
+ id = (known after apply)
}
# module.k3s.null_resource.k8s_ca_certificates_install[3] will be created
+ resource "null_resource" "k8s_ca_certificates_install" {
+ id = (known after apply)
}
# module.k3s.null_resource.k8s_ca_certificates_install[4] will be created
+ resource "null_resource" "k8s_ca_certificates_install" {
+ id = (known after apply)
}
# module.k3s.null_resource.k8s_ca_certificates_install[5] will be created
+ resource "null_resource" "k8s_ca_certificates_install" {
+ id = (known after apply)
}
# module.k3s.null_resource.kubernetes_ready will be created
+ resource "null_resource" "kubernetes_ready" {
+ id = (known after apply)
}
# module.k3s.null_resource.servers_drain["node-1"] will be created
+ resource "null_resource" "servers_drain" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_drain["node-2"] will be created
+ resource "null_resource" "servers_drain" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_drain["node-3"] will be created
+ resource "null_resource" "servers_drain" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_install["node-1"] will be created
+ resource "null_resource" "servers_install" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_install["node-2"] will be created
+ resource "null_resource" "servers_install" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_install["node-3"] will be created
+ resource "null_resource" "servers_install" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] will be created
+ resource "null_resource" "servers_label" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] will be created
+ resource "null_resource" "servers_label" {
+ id = (known after apply)
+ triggers = (known after apply)
}
# module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] will be created
+ resource "null_resource" "servers_label" {
+ id = (known after apply)
+ triggers = (known after apply)
}
Plan: 18 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ kube_config = (sensitive value)
╷
│ Warning: Deprecated Resource
│
│ with data.civo_template.ubuntu,
│ on main.tf line 19, in data "civo_template" "ubuntu":
│ 19: data "civo_template" "ubuntu" {
│
│ "civo_template" data source is deprecated. Moving forward, please use "civo_disk_image" data source.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: Attribute is deprecated
│
│ with civo_instance.node_instances,
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
╵
╷
│ Warning: Deprecated Attribute
│
│ with civo_instance.node_instances[0],
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
│
│ (and 2 more similar warnings elsewhere)
╵
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
civo_instance.node_instances[0]: Creating...
civo_instance.node_instances[2]: Creating...
civo_instance.node_instances[2]: Still creating... [10s elapsed]
civo_instance.node_instances[0]: Still creating... [10s elapsed]
civo_instance.node_instances[0]: Still creating... [20s elapsed]
civo_instance.node_instances[2]: Still creating... [20s elapsed]
civo_instance.node_instances[2]: Still creating... [30s elapsed]
civo_instance.node_instances[0]: Still creating... [30s elapsed]
civo_instance.node_instances[0]: Still creating... [40s elapsed]
civo_instance.node_instances[2]: Still creating... [40s elapsed]
civo_instance.node_instances[0]: Still creating... [50s elapsed]
civo_instance.node_instances[2]: Still creating... [50s elapsed]
civo_instance.node_instances[2]: Still creating... [1m0s elapsed]
civo_instance.node_instances[0]: Still creating... [1m0s elapsed]
civo_instance.node_instances[0]: Still creating... [1m10s elapsed]
civo_instance.node_instances[2]: Still creating... [1m10s elapsed]
civo_instance.node_instances[0]: Creation complete after 1m12s [id=078aeab9-89da-46db-995b-d20fc2a026e7]
civo_instance.node_instances[2]: Creation complete after 1m16s [id=7666cd7d-48ad-4703-9253-481d6aa9e7a5]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Creating...
module.k3s.null_resource.k8s_ca_certificates_install[4]: Creating...
module.k3s.null_resource.k8s_ca_certificates_install[3]: Creating...
module.k3s.null_resource.k8s_ca_certificates_install[0]: Creating...
module.k3s.null_resource.k8s_ca_certificates_install[2]: Creating...
module.k3s.null_resource.k8s_ca_certificates_install[1]: Creating...
module.k3s.null_resource.k8s_ca_certificates_install[5]: Provisioning with 'remote-exec'...
module.k3s.null_resource.k8s_ca_certificates_install[1]: Provisioning with 'remote-exec'...
module.k3s.null_resource.k8s_ca_certificates_install[3]: Provisioning with 'remote-exec'...
module.k3s.null_resource.k8s_ca_certificates_install[2]: Provisioning with 'remote-exec'...
module.k3s.null_resource.k8s_ca_certificates_install[0]: Provisioning with 'remote-exec'...
module.k3s.null_resource.k8s_ca_certificates_install[4]: Provisioning with 'remote-exec'...
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[2] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[0] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[4] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[0] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[4] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[2] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[0] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[4] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[2] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[0] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[4] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[2] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): User: root
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Password: true
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Private key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Certificate: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): SSH Agent: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Checking Host Key: false
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Target Platform: unix
module.k3s.null_resource.k8s_ca_certificates_install[0] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[4] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[2] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.k8s_ca_certificates_install[3] (remote-exec): Connected!
module.k3s.null_resource.k8s_ca_certificates_install[1] (remote-exec): Connected!
module.k3s.null_resource.k8s_ca_certificates_install[5] (remote-exec): Connected!
module.k3s.null_resource.k8s_ca_certificates_install[2]: Still creating... [10s elapsed]
module.k3s.null_resource.k8s_ca_certificates_install[4]: Still creating... [10s elapsed]
module.k3s.null_resource.k8s_ca_certificates_install[0]: Still creating... [10s elapsed]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Still creating... [10s elapsed]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Still creating... [10s elapsed]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Still creating... [10s elapsed]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Creation complete after 14s [id=6663274023099510187]
module.k3s.null_resource.k8s_ca_certificates_install[0]: Creation complete after 14s [id=5056326130768808998]
module.k3s.null_resource.k8s_ca_certificates_install[2]: Creation complete after 14s [id=5905791126692194777]
module.k3s.null_resource.k8s_ca_certificates_install[4]: Creation complete after 14s [id=776952705639779336]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Creation complete after 14s [id=7322411793949049427]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Creation complete after 14s [id=5484088503243209669]
module.k3s.null_resource.servers_install["node-2"]: Creating...
module.k3s.null_resource.servers_install["node-3"]: Creating...
module.k3s.null_resource.servers_install["node-1"]: Creating...
module.k3s.null_resource.servers_install["node-2"]: Provisioning with 'file'...
module.k3s.null_resource.servers_install["node-1"]: Provisioning with 'file'...
module.k3s.null_resource.servers_install["node-3"]: Provisioning with 'file'...
module.k3s.null_resource.servers_install["node-1"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"]: Still creating... [10s elapsed]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [10s elapsed]
module.k3s.null_resource.servers_install["node-1"]: Still creating... [10s elapsed]
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"]: Still creating... [20s elapsed]
module.k3s.null_resource.servers_install["node-2"]: Still creating... [20s elapsed]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [20s elapsed]
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"]: Still creating... [30s elapsed]
module.k3s.null_resource.servers_install["node-1"]: Still creating... [30s elapsed]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [30s elapsed]
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"]: Creation complete after 33s [id=4470835761045161343]
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"]: Still creating... [40s elapsed]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [40s elapsed]
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"]: Creation complete after 46s [id=8062377340920100958]
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"]: Still creating... [50s elapsed]
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"]: Still creating... [1m0s elapsed]
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"]: Creation complete after 1m2s [id=4646391725211103674]
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Creating...
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Creating...
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Creating...
module.k3s.null_resource.servers_drain["node-1"]: Creating...
module.k3s.null_resource.servers_drain["node-2"]: Creating...
module.k3s.null_resource.servers_drain["node-3"]: Creating...
module.k3s.null_resource.servers_drain["node-1"]: Creation complete after 0s [id=8691228253726825796]
module.k3s.null_resource.servers_drain["node-2"]: Creation complete after 0s [id=3646547839837205088]
module.k3s.null_resource.servers_drain["node-3"]: Creation complete after 0s [id=7048446702711938358]
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): User: root
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Password: true
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Private key: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): User: root
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Password: true
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Private key: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): User: root
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Password: true
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Private key: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Connected!
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Connected!
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Connected!
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): node/node-3 labeled
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): node/node-2 labeled
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): node/node-1 labeled
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Creation complete after 3s [id=1540572406495879692]
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Creation complete after 3s [id=2100247009859532487]
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Creation complete after 4s [id=8481180087172316037]
module.k3s.null_resource.kubernetes_ready: Creating...
module.k3s.null_resource.kubernetes_ready: Creation complete after 0s [id=2551799405757945470]
╷
│ Warning: Deprecated Attribute
│
│ with civo_instance.node_instances[0],
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
│
│ (and one more similar warning elsewhere)
╵
Apply complete! Resources: 18 added, 0 changed, 0 destroyed.
Outputs:
kube_config = <sensitive>
*note: I ran the apply twice. The first time it failed on a timeout, but the second picked up where it left off and completed successfully as shown above*
We see it completing in the CIVO Dashboard.
builder@DESKTOP-QADGF36:~/Workspaces/terraform-module-k3s/examples/civo-k3s$ terraform output -raw kube_config > ~/.kube/config
builder@DESKTOP-QADGF36:~/Workspaces/terraform-module-k3s/examples/civo-k3s$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-1 Ready control-plane,etcd,master 4m43s v1.21.4+k3s1
node-2 Ready control-plane,etcd,master 4m30s v1.21.4+k3s1
node-3 Ready control-plane,etcd,master 4m15s v1.21.4+k3s1
Next, we can install a sample app:
builder@DESKTOP-QADGF36:~/Workspaces/terraform-module-k3s/examples/civo-k3s$ helm repo add azure-samples https://azure-samples.github.io/helm-charts/
"azure-samples" has been added to your repositories
builder@DESKTOP-QADGF36:~/Workspaces/terraform-module-k3s/examples/civo-k3s$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "datadog" chart repository
Update Complete. ⎈Happy Helming!⎈
builder@DESKTOP-QADGF36:~/Workspaces/terraform-module-k3s/examples/civo-k3s$ helm install azure-samples/azure-vote --generate-name
NAME: azure-vote-1636025969
LAST DEPLOYED: Thu Nov 4 06:39:31 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Azure Vote application has been started on your Kubernetes cluster.
Title: Azure Vote App
Vote 1 value: Cats
Vote 2 value: Dogs
The externally accessible IP address can take a minute or so to provision. Run the following command to monitor the provisioning status. Once an External IP address has been provisioned, brows to this IP address to access the Azure Vote application.
kubectl get service -l name=azure-vote-front -w
and see the Vote app is installed and running:
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
azure-vote-1636025969 default 1 2021-11-04 06:39:31.307844671 -0500 CDT deployed azure-vote-0.1.1
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
svclb-azure-vote-front-5rxrg 1/1 Running 0 2m35s
svclb-azure-vote-front-8qlc6 1/1 Running 0 2m34s
svclb-azure-vote-front-rttmq 1/1 Running 0 2m34s
vote-back-azure-vote-1636025969-8cf645cc4-9x9ck 1/1 Running 0 2m35s
vote-front-azure-vote-1636025969-784d87dbbd-t274z 1/1 Running 0 2m35s
Upgrade
$ cat main.tf | grep k3s_version
k3s_version = "v1.21.4+k3s1"
$ sed -i 's/v1.21.4/v1.22.3-rc1/g' main.tf
$ cat main.tf | grep k3s_version
k3s_version = "v1.22.3-rc1+k3s1"
Do a plan
$ terraform plan -var-file="tf.vars"
module.k3s.tls_private_key.kubernetes_ca[1]: Refreshing state... [id=fb31ae6ed4b0cda025947155d636f41acc97932a]
module.k3s.tls_private_key.kubernetes_ca[0]: Refreshing state... [id=ac1d6302a69b6a95ceaa0ab3513a0befc52eae1b]
module.k3s.tls_private_key.kubernetes_ca[2]: Refreshing state... [id=80ca4a9bebd3fa1437efef27571e4927e873a146]
module.k3s.tls_private_key.master_user[0]: Refreshing state... [id=56e5b1cf3fb10387213633492993251aed8a2c5c]
module.k3s.random_password.k3s_cluster_secret: Refreshing state... [id=none]
module.k3s.tls_cert_request.master_user[0]: Refreshing state... [id=5457fd7b6ff35fb465293f5ac48cf51f73f604da]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Refreshing state... [id=179381637326519421633541131708977756020]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Refreshing state... [id=240239317385907192679446816919634150104]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Refreshing state... [id=162007174841491717249967010213281523901]
module.k3s.tls_locally_signed_cert.master_user[0]: Refreshing state... [id=269975477633753601419630313929603900630]
civo_instance.node_instances[0]: Refreshing state... [id=078aeab9-89da-46db-995b-d20fc2a026e7]
civo_instance.node_instances[2]: Refreshing state... [id=7666cd7d-48ad-4703-9253-481d6aa9e7a5]
civo_instance.node_instances[1]: Refreshing state... [id=27cf0c45-25d9-47a8-abda-e0616fa68d9f]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Refreshing state... [id=5484088503243209669]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Refreshing state... [id=7322411793949049427]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Refreshing state... [id=6663274023099510187]
module.k3s.null_resource.k8s_ca_certificates_install[4]: Refreshing state... [id=776952705639779336]
module.k3s.null_resource.k8s_ca_certificates_install[2]: Refreshing state... [id=5905791126692194777]
module.k3s.null_resource.k8s_ca_certificates_install[0]: Refreshing state... [id=5056326130768808998]
module.k3s.null_resource.servers_install["node-1"]: Refreshing state... [id=4470835761045161343]
module.k3s.null_resource.servers_install["node-3"]: Refreshing state... [id=4646391725211103674]
module.k3s.null_resource.servers_install["node-2"]: Refreshing state... [id=8062377340920100958]
module.k3s.null_resource.servers_drain["node-3"]: Refreshing state... [id=7048446702711938358]
module.k3s.null_resource.servers_drain["node-2"]: Refreshing state... [id=3646547839837205088]
module.k3s.null_resource.servers_drain["node-1"]: Refreshing state... [id=8691228253726825796]
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Refreshing state... [id=2100247009859532487]
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Refreshing state... [id=1540572406495879692]
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Refreshing state... [id=8481180087172316037]
module.k3s.null_resource.kubernetes_ready: Refreshing state... [id=2551799405757945470]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the last "terraform apply":
# civo_instance.node_instances[2] has been changed
~ resource "civo_instance" "node_instances" {
id = "7666cd7d-48ad-4703-9253-481d6aa9e7a5"
+ tags = []
# (17 unchanged attributes hidden)
}
# civo_instance.node_instances[0] has been changed
~ resource "civo_instance" "node_instances" {
id = "078aeab9-89da-46db-995b-d20fc2a026e7"
+ tags = []
# (17 unchanged attributes hidden)
}
Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.k3s.null_resource.servers_install["node-1"] must be replaced
-/+ resource "null_resource" "servers_install" {
~ id = "4470835761045161343" -> (known after apply)
~ triggers = { # forces replacement
~ "on_new_version" = "v1.21.4+k3s1" -> "v1.22.3-rc1+k3s1"
# (1 unchanged element hidden)
}
}
# module.k3s.null_resource.servers_install["node-2"] must be replaced
-/+ resource "null_resource" "servers_install" {
~ id = "8062377340920100958" -> (known after apply)
~ triggers = { # forces replacement
~ "on_new_version" = "v1.21.4+k3s1" -> "v1.22.3-rc1+k3s1"
# (1 unchanged element hidden)
}
}
# module.k3s.null_resource.servers_install["node-3"] must be replaced
-/+ resource "null_resource" "servers_install" {
~ id = "4646391725211103674" -> (known after apply)
~ triggers = { # forces replacement
~ "on_new_version" = "v1.21.4+k3s1" -> "v1.22.3-rc1+k3s1"
# (1 unchanged element hidden)
}
}
Plan: 3 to add, 0 to change, 3 to destroy.
╷
│ Warning: Deprecated Resource
│
│ with data.civo_template.ubuntu,
│ on main.tf line 19, in data "civo_template" "ubuntu":
│ 19: data "civo_template" "ubuntu" {
│
│ "civo_template" data source is deprecated. Moving forward, please use "civo_disk_image" data source.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: Attribute is deprecated
│
│ with civo_instance.node_instances,
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
╵
╷
│ Warning: Deprecated Attribute
│
│ with civo_instance.node_instances[2],
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
│
│ (and 2 more similar warnings elsewhere)
╵
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform
apply" now.
then apply
$ terraform apply -var-file="tf.vars"
module.k3s.tls_private_key.master_user[0]: Refreshing state... [id=56e5b1cf3fb10387213633492993251aed8a2c5c]
module.k3s.tls_private_key.kubernetes_ca[0]: Refreshing state... [id=ac1d6302a69b6a95ceaa0ab3513a0befc52eae1b]
module.k3s.tls_private_key.kubernetes_ca[1]: Refreshing state... [id=fb31ae6ed4b0cda025947155d636f41acc97932a]
module.k3s.tls_private_key.kubernetes_ca[2]: Refreshing state... [id=80ca4a9bebd3fa1437efef27571e4927e873a146]
module.k3s.random_password.k3s_cluster_secret: Refreshing state... [id=none]
module.k3s.tls_cert_request.master_user[0]: Refreshing state... [id=5457fd7b6ff35fb465293f5ac48cf51f73f604da]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Refreshing state... [id=162007174841491717249967010213281523901]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Refreshing state... [id=240239317385907192679446816919634150104]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Refreshing state... [id=179381637326519421633541131708977756020]
module.k3s.tls_locally_signed_cert.master_user[0]: Refreshing state... [id=269975477633753601419630313929603900630]
civo_instance.node_instances[2]: Refreshing state... [id=7666cd7d-48ad-4703-9253-481d6aa9e7a5]
civo_instance.node_instances[1]: Refreshing state... [id=27cf0c45-25d9-47a8-abda-e0616fa68d9f]
civo_instance.node_instances[0]: Refreshing state... [id=078aeab9-89da-46db-995b-d20fc2a026e7]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Refreshing state... [id=7322411793949049427]
module.k3s.null_resource.k8s_ca_certificates_install[4]: Refreshing state... [id=776952705639779336]
module.k3s.null_resource.k8s_ca_certificates_install[0]: Refreshing state... [id=5056326130768808998]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Refreshing state... [id=6663274023099510187]
module.k3s.null_resource.k8s_ca_certificates_install[2]: Refreshing state... [id=5905791126692194777]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Refreshing state... [id=5484088503243209669]
module.k3s.null_resource.servers_install["node-3"]: Refreshing state... [id=4646391725211103674]
module.k3s.null_resource.servers_install["node-2"]: Refreshing state... [id=8062377340920100958]
module.k3s.null_resource.servers_install["node-1"]: Refreshing state... [id=4470835761045161343]
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Refreshing state... [id=8481180087172316037]
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Refreshing state... [id=1540572406495879692]
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Refreshing state... [id=2100247009859532487]
module.k3s.null_resource.servers_drain["node-2"]: Refreshing state... [id=3646547839837205088]
module.k3s.null_resource.servers_drain["node-1"]: Refreshing state... [id=8691228253726825796]
module.k3s.null_resource.servers_drain["node-3"]: Refreshing state... [id=7048446702711938358]
module.k3s.null_resource.kubernetes_ready: Refreshing state... [id=2551799405757945470]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the last "terraform apply":
# civo_instance.node_instances[2] has been changed
~ resource "civo_instance" "node_instances" {
id = "7666cd7d-48ad-4703-9253-481d6aa9e7a5"
+ tags = []
# (17 unchanged attributes hidden)
}
# civo_instance.node_instances[0] has been changed
~ resource "civo_instance" "node_instances" {
id = "078aeab9-89da-46db-995b-d20fc2a026e7"
+ tags = []
# (17 unchanged attributes hidden)
}
Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.k3s.null_resource.servers_install["node-1"] must be replaced
-/+ resource "null_resource" "servers_install" {
~ id = "4470835761045161343" -> (known after apply)
~ triggers = { # forces replacement
~ "on_new_version" = "v1.21.4+k3s1" -> "v1.22.3-rc1+k3s1"
# (1 unchanged element hidden)
}
}
# module.k3s.null_resource.servers_install["node-2"] must be replaced
-/+ resource "null_resource" "servers_install" {
~ id = "8062377340920100958" -> (known after apply)
~ triggers = { # forces replacement
~ "on_new_version" = "v1.21.4+k3s1" -> "v1.22.3-rc1+k3s1"
# (1 unchanged element hidden)
}
}
# module.k3s.null_resource.servers_install["node-3"] must be replaced
-/+ resource "null_resource" "servers_install" {
~ id = "4646391725211103674" -> (known after apply)
~ triggers = { # forces replacement
~ "on_new_version" = "v1.21.4+k3s1" -> "v1.22.3-rc1+k3s1"
# (1 unchanged element hidden)
}
}
Plan: 3 to add, 0 to change, 3 to destroy.
╷
│ Warning: Deprecated Resource
│
│ with data.civo_template.ubuntu,
│ on main.tf line 19, in data "civo_template" "ubuntu":
│ 19: data "civo_template" "ubuntu" {
│
│ "civo_template" data source is deprecated. Moving forward, please use "civo_disk_image" data source.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: Attribute is deprecated
│
│ with civo_instance.node_instances,
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
╵
╷
│ Warning: Deprecated Attribute
│
│ with civo_instance.node_instances[0],
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
│
│ (and 2 more similar warnings elsewhere)
╵
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
module.k3s.null_resource.servers_install["node-3"]: Destroying... [id=4646391725211103674]
module.k3s.null_resource.servers_install["node-2"]: Destroying... [id=8062377340920100958]
module.k3s.null_resource.servers_install["node-1"]: Destroying... [id=4470835761045161343]
module.k3s.null_resource.servers_install["node-3"]: Destruction complete after 0s
module.k3s.null_resource.servers_install["node-2"]: Destruction complete after 0s
module.k3s.null_resource.servers_install["node-1"]: Destruction complete after 0s
module.k3s.null_resource.servers_install["node-2"]: Creating...
module.k3s.null_resource.servers_install["node-1"]: Creating...
module.k3s.null_resource.servers_install["node-3"]: Creating...
module.k3s.null_resource.servers_install["node-3"]: Provisioning with 'file'...
module.k3s.null_resource.servers_install["node-2"]: Provisioning with 'file'...
module.k3s.null_resource.servers_install["node-1"]: Provisioning with 'file'...
module.k3s.null_resource.servers_install["node-1"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"]: Still creating... [10s elapsed]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [10s elapsed]
module.k3s.null_resource.servers_install["node-2"]: Still creating... [10s elapsed]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [20s elapsed]
module.k3s.null_resource.servers_install["node-1"]: Still creating... [20s elapsed]
module.k3s.null_resource.servers_install["node-2"]: Still creating... [20s elapsed]
module.k3s.null_resource.servers_install["node-2"]: Still creating... [30s elapsed]
module.k3s.null_resource.servers_install["node-1"]: Still creating... [30s elapsed]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [30s elapsed]
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-2"]: Creation complete after 39s [id=1951024970189533157]
module.k3s.null_resource.servers_install["node-3"]: Still creating... [40s elapsed]
module.k3s.null_resource.servers_install["node-1"]: Still creating... [40s elapsed]
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-1"]: Creation complete after 49s [id=8440407648830712598]
module.k3s.null_resource.servers_install["node-3"] (remote-exec): (output suppressed due to sensitive value in config)
module.k3s.null_resource.servers_install["node-3"]: Creation complete after 49s [id=7060114474816286895]
Apply complete! Resources: 3 added, 0 changed, 3 destroyed.
Outputs:
kube_config = <sensitive>
We can see the cluster was upgraded:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-1 Ready control-plane,etcd,master 11m v1.22.3-rc1+k3s1
node-2 Ready control-plane,etcd,master 11m v1.22.3-rc1+k3s1
node-3 Ready control-plane,etcd,master 10m v1.22.3-rc1+k3s1
but our content is still there:
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
azure-vote-1636025969 default 1 2021-11-04 06:39:31.307844671 -0500 CDT deployed azure-vote-0.1.1
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
svclb-azure-vote-front-5rxrg 1/1 Running 0 4m56s
svclb-azure-vote-front-8qlc6 1/1 Running 0 4m55s
svclb-azure-vote-front-rttmq 1/1 Running 0 4m55s
vote-back-azure-vote-1636025969-8cf645cc4-9x9ck 1/1 Running 0 4m56s
vote-front-azure-vote-1636025969-784d87dbbd-t274z 1/1 Running 0 4m56s
Cleanup
To remove the cluster, we can destroy it with TF Destroy:
$ terraform destroy -var-file="tf.vars"
module.k3s.random_password.k3s_cluster_secret: Refreshing state... [id=none]
module.k3s.tls_private_key.kubernetes_ca[2]: Refreshing state... [id=80ca4a9bebd3fa1437efef27571e4927e873a146]
module.k3s.tls_private_key.master_user[0]: Refreshing state... [id=56e5b1cf3fb10387213633492993251aed8a2c5c]
module.k3s.tls_private_key.kubernetes_ca[0]: Refreshing state... [id=ac1d6302a69b6a95ceaa0ab3513a0befc52eae1b]
module.k3s.tls_private_key.kubernetes_ca[1]: Refreshing state... [id=fb31ae6ed4b0cda025947155d636f41acc97932a]
module.k3s.tls_cert_request.master_user[0]: Refreshing state... [id=5457fd7b6ff35fb465293f5ac48cf51f73f604da]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Refreshing state... [id=240239317385907192679446816919634150104]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Refreshing state... [id=179381637326519421633541131708977756020]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Refreshing state... [id=162007174841491717249967010213281523901]
module.k3s.tls_locally_signed_cert.master_user[0]: Refreshing state... [id=269975477633753601419630313929603900630]
civo_instance.node_instances[0]: Refreshing state... [id=078aeab9-89da-46db-995b-d20fc2a026e7]
civo_instance.node_instances[2]: Refreshing state... [id=7666cd7d-48ad-4703-9253-481d6aa9e7a5]
civo_instance.node_instances[1]: Refreshing state... [id=27cf0c45-25d9-47a8-abda-e0616fa68d9f]
module.k3s.null_resource.k8s_ca_certificates_install[4]: Refreshing state... [id=776952705639779336]
module.k3s.null_resource.k8s_ca_certificates_install[2]: Refreshing state... [id=5905791126692194777]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Refreshing state... [id=7322411793949049427]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Refreshing state... [id=5484088503243209669]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Refreshing state... [id=6663274023099510187]
module.k3s.null_resource.k8s_ca_certificates_install[0]: Refreshing state... [id=5056326130768808998]
module.k3s.null_resource.servers_install["node-2"]: Refreshing state... [id=1951024970189533157]
module.k3s.null_resource.servers_install["node-3"]: Refreshing state... [id=7060114474816286895]
module.k3s.null_resource.servers_install["node-1"]: Refreshing state... [id=8440407648830712598]
module.k3s.null_resource.servers_drain["node-3"]: Refreshing state... [id=7048446702711938358]
module.k3s.null_resource.servers_drain["node-2"]: Refreshing state... [id=3646547839837205088]
module.k3s.null_resource.servers_drain["node-1"]: Refreshing state... [id=8691228253726825796]
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Refreshing state... [id=8481180087172316037]
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Refreshing state... [id=1540572406495879692]
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Refreshing state... [id=2100247009859532487]
module.k3s.null_resource.kubernetes_ready: Refreshing state... [id=2551799405757945470]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
# civo_instance.node_instances[0] will be destroyed
- resource "civo_instance" "node_instances" {
- cpu_cores = 1 -> null
- created_at = "0001-01-01 00:00:00 +0000 UTC" -> null
- disk_gb = 25 -> null
- firewall_id = "092d473d-c273-4a0b-b4a2-61d0184086bc" -> null
- hostname = "node-1" -> null
- id = "078aeab9-89da-46db-995b-d20fc2a026e7" -> null
- initial_password = (sensitive value)
- initial_user = "civo" -> null
- network_id = "7e5b8507-6a82-495c-ac1d-768d04eaca82" -> null
- private_ip = "192.168.1.3" -> null
- public_ip = "212.2.243.199" -> null
- public_ip_required = "create" -> null
- ram_mb = 2048 -> null
- size = "g3.small" -> null
- source_id = "ubuntu-focal" -> null
- source_type = "diskimage" -> null
- status = "ACTIVE" -> null
- tags = [] -> null
- template = "8eb48e20-e5db-49fe-9cdf-cc8f381c61c6" -> null
}
# civo_instance.node_instances[1] will be destroyed
- resource "civo_instance" "node_instances" {
- cpu_cores = 1 -> null
- created_at = "0001-01-01 00:00:00 +0000 UTC" -> null
- disk_gb = 25 -> null
- firewall_id = "092d473d-c273-4a0b-b4a2-61d0184086bc" -> null
- hostname = "node-2" -> null
- id = "27cf0c45-25d9-47a8-abda-e0616fa68d9f" -> null
- initial_password = (sensitive value)
- initial_user = "civo" -> null
- network_id = "7e5b8507-6a82-495c-ac1d-768d04eaca82" -> null
- private_ip = "192.168.1.2" -> null
- public_ip = "212.2.244.130" -> null
- public_ip_required = "create" -> null
- ram_mb = 2048 -> null
- size = "g3.small" -> null
- source_id = "ubuntu-focal" -> null
- source_type = "diskimage" -> null
- status = "ACTIVE" -> null
- tags = [] -> null
- template = "8eb48e20-e5db-49fe-9cdf-cc8f381c61c6" -> null
}
# civo_instance.node_instances[2] will be destroyed
- resource "civo_instance" "node_instances" {
- cpu_cores = 1 -> null
- created_at = "0001-01-01 00:00:00 +0000 UTC" -> null
- disk_gb = 25 -> null
- firewall_id = "092d473d-c273-4a0b-b4a2-61d0184086bc" -> null
- hostname = "node-3" -> null
- id = "7666cd7d-48ad-4703-9253-481d6aa9e7a5" -> null
- initial_password = (sensitive value)
- initial_user = "civo" -> null
- network_id = "7e5b8507-6a82-495c-ac1d-768d04eaca82" -> null
- private_ip = "192.168.1.4" -> null
- public_ip = "212.2.247.59" -> null
- public_ip_required = "create" -> null
- ram_mb = 2048 -> null
- size = "g3.small" -> null
- source_id = "ubuntu-focal" -> null
- source_type = "diskimage" -> null
- status = "ACTIVE" -> null
- tags = [] -> null
- template = "8eb48e20-e5db-49fe-9cdf-cc8f381c61c6" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[0] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "5056326130768808998" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[1] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "5484088503243209669" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[2] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "5905791126692194777" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[3] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "7322411793949049427" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[4] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "776952705639779336" -> null
}
# module.k3s.null_resource.k8s_ca_certificates_install[5] will be destroyed
- resource "null_resource" "k8s_ca_certificates_install" {
- id = "6663274023099510187" -> null
}
# module.k3s.null_resource.kubernetes_ready will be destroyed
- resource "null_resource" "kubernetes_ready" {
- id = "2551799405757945470" -> null
}
# module.k3s.null_resource.servers_drain["node-1"] will be destroyed
- resource "null_resource" "servers_drain" {
- id = "8691228253726825796" -> null
- triggers = {
- "connection_json" = (sensitive)
- "drain_timeout" = "60s"
- "kubectl_cmd" = "kubectl"
- "server_name" = "node-1"
} -> null
}
# module.k3s.null_resource.servers_drain["node-2"] will be destroyed
- resource "null_resource" "servers_drain" {
- id = "3646547839837205088" -> null
- triggers = {
- "connection_json" = (sensitive)
- "drain_timeout" = "60s"
- "kubectl_cmd" = "kubectl"
- "server_name" = "node-2"
} -> null
}
# module.k3s.null_resource.servers_drain["node-3"] will be destroyed
- resource "null_resource" "servers_drain" {
- id = "7048446702711938358" -> null
- triggers = {
- "connection_json" = (sensitive)
- "drain_timeout" = "60s"
- "kubectl_cmd" = "kubectl"
- "server_name" = "node-3"
} -> null
}
# module.k3s.null_resource.servers_install["node-1"] will be destroyed
- resource "null_resource" "servers_install" {
- id = "8440407648830712598" -> null
- triggers = {
- "on_immutable_changes" = "f924609d11690307b180d3a484ac31815d775f47"
- "on_new_version" = "v1.22.3-rc1+k3s1"
} -> null
}
# module.k3s.null_resource.servers_install["node-2"] will be destroyed
- resource "null_resource" "servers_install" {
- id = "1951024970189533157" -> null
- triggers = {
- "on_immutable_changes" = "f924609d11690307b180d3a484ac31815d775f47"
- "on_new_version" = "v1.22.3-rc1+k3s1"
} -> null
}
# module.k3s.null_resource.servers_install["node-3"] will be destroyed
- resource "null_resource" "servers_install" {
- id = "7060114474816286895" -> null
- triggers = {
- "on_immutable_changes" = "f924609d11690307b180d3a484ac31815d775f47"
- "on_new_version" = "v1.22.3-rc1+k3s1"
} -> null
}
# module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] will be destroyed
- resource "null_resource" "servers_label" {
- id = "8481180087172316037" -> null
- triggers = {
- "connection_json" = (sensitive)
- "kubectl_cmd" = "kubectl"
- "label_name" = "node.kubernetes.io/type"
- "on_value_changes" = "master"
- "server_name" = "node-1"
} -> null
}
# module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] will be destroyed
- resource "null_resource" "servers_label" {
- id = "2100247009859532487" -> null
- triggers = {
- "connection_json" = (sensitive)
- "kubectl_cmd" = "kubectl"
- "label_name" = "node.kubernetes.io/type"
- "on_value_changes" = "master"
- "server_name" = "node-2"
} -> null
}
# module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] will be destroyed
- resource "null_resource" "servers_label" {
- id = "1540572406495879692" -> null
- triggers = {
- "connection_json" = (sensitive)
- "kubectl_cmd" = "kubectl"
- "label_name" = "node.kubernetes.io/type"
- "on_value_changes" = "master"
- "server_name" = "node-3"
} -> null
}
# module.k3s.random_password.k3s_cluster_secret will be destroyed
- resource "random_password" "k3s_cluster_secret" {
- id = "none" -> null
- length = 48 -> null
- lower = true -> null
- min_lower = 0 -> null
- min_numeric = 0 -> null
- min_special = 0 -> null
- min_upper = 0 -> null
- number = true -> null
- result = (sensitive value)
- special = false -> null
- upper = true -> null
}
# module.k3s.tls_cert_request.master_user[0] will be destroyed
- resource "tls_cert_request" "master_user" {
- cert_request_pem = <<-EOT
-----BEGIN CERTIFICATE REQUEST-----
MIIBJzCBrgIBADAvMRcwFQYDVQQKEw5zeXN0ZW06bWFzdGVyczEUMBIGA1UEAxML
bWFzdGVyLXVzZXIwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAARq7MnyYj53drtZ/xd4
GRTsH/myr7uLd3XcRIsjr1JYMMpilvbYlhPI+yVwsnacsiyrhyu0Q/CUaLAThdId
Zrhg0WG0Gx5feXP1dvvafjWwTArkexQwUFSutfFISQQaGo6gADAKBggqhkjOPQQD
AwNoADBlAjEA1G+UAdoH2j7DjrH6mqcgCHHDgpPmP7m7i7e4inyhg2ZLS2Ezy7Ov
y0Uv+IcbgnooAjBea7mQnChRFo3d0xS6Wc2gMHR53lDSO8Pm/xtjHavyVAOnHkFZ
nqWHfN/dG2x2dW0=
-----END CERTIFICATE REQUEST-----
EOT -> null
- id = "5457fd7b6ff35fb465293f5ac48cf51f73f604da" -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- subject {
- common_name = "master-user" -> null
- organization = "system:masters" -> null
- street_address = [] -> null
}
}
# module.k3s.tls_locally_signed_cert.master_user[0] will be destroyed
- resource "tls_locally_signed_cert" "master_user" {
- allowed_uses = [
- "key_encipherment",
- "digital_signature",
- "client_auth",
] -> null
- ca_cert_pem = "e8ba395256719b34d6d1912fab1d5b49b55b49cc" -> null
- ca_key_algorithm = "ECDSA" -> null
- ca_private_key_pem = (sensitive value)
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIB4zCCAWigAwIBAgIRAMsbYrrHOIbL0phasAZRZNYwCgYIKoZIzj0EAwMwHzEd
MBsGA1UEAxMUa3ViZXJuZXRlcy1jbGllbnQtY2EwIBcNMjExMTA0MTEyOTM1WhgP
MjEyMTExMDUxMTI5MzVaMC8xFzAVBgNVBAoTDnN5c3RlbTptYXN0ZXJzMRQwEgYD
VQQDEwttYXN0ZXItdXNlcjB2MBAGByqGSM49AgEGBSuBBAAiA2IABGrsyfJiPnd2
u1n/F3gZFOwf+bKvu4t3ddxEiyOvUlgwymKW9tiWE8j7JXCydpyyLKuHK7RD8JRo
sBOF0h1muGDRYbQbHl95c/V2+9p+NbBMCuR7FDBQVK618UhJBBoajqNWMFQwDgYD
VR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw
HwYDVR0jBBgwFoAUeEIuEge2Mysjw5923f+AlJ8F3dEwCgYIKoZIzj0EAwMDaQAw
ZgIxAL1rtFjJB/7RYmT5v5PhLu6qzgagKkyJR2dMqkbPzdEET873NYXXgXeyQbU1
hmoSbQIxANGD9f+RW36/X84+RtXVhU6OidFKdRVExDTkxAuiUbWUxQbiAVrhRhpr
xWIE0YIDyw==
-----END CERTIFICATE-----
EOT -> null
- cert_request_pem = "fb39f4ca739e340df938df19e57b324a5ca775df" -> null
- early_renewal_hours = 0 -> null
- id = "269975477633753601419630313929603900630" -> null
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-05T05:29:35.236126398-06:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-11-04T06:29:35.236126398-05:00" -> null
}
# module.k3s.tls_private_key.kubernetes_ca[0] will be destroyed
- resource "tls_private_key" "kubernetes_ca" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "ac1d6302a69b6a95ceaa0ab3513a0befc52eae1b" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "7f:c5:83:4d:40:f4:d3:76:38:c1:aa:77:93:89:fc:c2" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBGZeZIJKRU9KkGG7eHWcXSqhnLbN0+Z0uTYyy10zyV3KElyNp7+Y02V6+dFn9w5VBHzeVih4O6FnEXPWPkHhcgTj6XrCXJB5MpmzarWO0ThmoOyVbZBFqf6m/54xQYxl0Q==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEZl5kgkpFT0qQYbt4dZxdKqGcts3T5nS5
NjLLXTPJXcoSXI2nv5jTZXr50Wf3DlUEfN5WKHg7oWcRc9Y+QeFyBOPpesJckHky
mbNqtY7ROGag7JVtkEWp/qb/njFBjGXR
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_private_key.kubernetes_ca[1] will be destroyed
- resource "tls_private_key" "kubernetes_ca" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "fb31ae6ed4b0cda025947155d636f41acc97932a" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "29:df:48:05:0f:64:f5:76:05:02:69:77:b7:57:51:01" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBJ8/W/dakRWkTrI4DeahuvKs8Vw3htOtBvl2EKvLXWL/s2nh2e78eFZzw8nr3QpRxZ/YmHARpozpRCtd9S/nssw+Y8czlDG0ESXWNMT3gMtkNMGFEsYmbZVFM7GfOElWMQ==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEnz9b91qRFaROsjgN5qG68qzxXDeG060G
+XYQq8tdYv+zaeHZ7vx4VnPDyevdClHFn9iYcBGmjOlEK131L+eyzD5jxzOUMbQR
JdY0xPeAy2Q0wYUSxiZtlUUzsZ84SVYx
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_private_key.kubernetes_ca[2] will be destroyed
- resource "tls_private_key" "kubernetes_ca" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "80ca4a9bebd3fa1437efef27571e4927e873a146" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "fb:34:97:86:ba:4b:29:5e:cf:3c:58:ee:8c:23:2e:a5" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIqTcWQ+vCsQc23IWwHFPDxl/8WlEuxJIOISmbcK95Q/yc2B7gm/ZQ8SWQiGUoTur5jkHF0H0WCQ2JhF/2qq73JoZLq13mllt0rb2KM39BBtmn5aUlWnFyKuZU6/r8ZBIQ==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEipNxZD68KxBzbchbAcU8PGX/xaUS7Ekg
4hKZtwr3lD/JzYHuCb9lDxJZCIZShO6vmOQcXQfRYJDYmEX/aqrvcmhkurXeaWW3
StvYozf0EG2aflpSVacXIq5lTr+vxkEh
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_private_key.master_user[0] will be destroyed
- resource "tls_private_key" "master_user" {
- algorithm = "ECDSA" -> null
- ecdsa_curve = "P384" -> null
- id = "56e5b1cf3fb10387213633492993251aed8a2c5c" -> null
- private_key_pem = (sensitive value)
- public_key_fingerprint_md5 = "74:1b:6d:ad:f1:f8:96:c0:8f:2d:86:1f:06:6d:2d:77" -> null
- public_key_openssh = <<-EOT
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBGrsyfJiPnd2u1n/F3gZFOwf+bKvu4t3ddxEiyOvUlgwymKW9tiWE8j7JXCydpyyLKuHK7RD8JRosBOF0h1muGDRYbQbHl95c/V2+9p+NbBMCuR7FDBQVK618UhJBBoajg==
EOT -> null
- public_key_pem = <<-EOT
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEauzJ8mI+d3a7Wf8XeBkU7B/5sq+7i3d1
3ESLI69SWDDKYpb22JYTyPslcLJ2nLIsq4crtEPwlGiwE4XSHWa4YNFhtBseX3lz
9Xb72n41sEwK5HsUMFBUrrXxSEkEGhqO
-----END PUBLIC KEY-----
EOT -> null
- rsa_bits = 2048 -> null
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"] will be destroyed
- resource "tls_self_signed_cert" "kubernetes_ca_certs" {
- allowed_uses = [
- "critical",
- "digitalSignature",
- "keyEncipherment",
- "keyCertSign",
] -> null
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIBrzCCATSgAwIBAgIRAIbzpYzSo2OJ1y5+UDS1t3QwCgYIKoZIzj0EAwMwHzEd
MBsGA1UEAxMUa3ViZXJuZXRlcy1jbGllbnQtY2EwIBcNMjExMTA0MTEyOTM1WhgP
MjEyMTExMDUxMTI5MzVaMB8xHTAbBgNVBAMTFGt1YmVybmV0ZXMtY2xpZW50LWNh
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEZl5kgkpFT0qQYbt4dZxdKqGcts3T5nS5
NjLLXTPJXcoSXI2nv5jTZXr50Wf3DlUEfN5WKHg7oWcRc9Y+QeFyBOPpesJckHky
mbNqtY7ROGag7JVtkEWp/qb/njFBjGXRozIwMDAPBgNVHRMBAf8EBTADAQH/MB0G
A1UdDgQWBBR4Qi4SB7YzKyPDn3bd/4CUnwXd0TAKBggqhkjOPQQDAwNpADBmAjEA
8UbD98Msu/hXrzpeZklGKEqdI370B/sa9sxQSPa66d5d+XT/S67Oa/yN+CcfUGHa
AjEAlHYR4Tonbsn74nb0Xla2u+Yza846DxLpSiwLyjmaGwqhOal9QGPBpWvUy40e
5MI6
-----END CERTIFICATE-----
EOT -> null
- early_renewal_hours = 0 -> null
- id = "179381637326519421633541131708977756020" -> null
- is_ca_certificate = true -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-05T05:29:35.200661558-06:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-11-04T06:29:35.200661558-05:00" -> null
- subject {
- common_name = "kubernetes-client-ca" -> null
- street_address = [] -> null
}
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"] will be destroyed
- resource "tls_self_signed_cert" "kubernetes_ca_certs" {
- allowed_uses = [
- "critical",
- "digitalSignature",
- "keyEncipherment",
- "keyCertSign",
] -> null
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIBrTCCATOgAwIBAgIQeeFyaZBqkqhzDxkQKjXovTAKBggqhkjOPQQDAzAfMR0w
GwYDVQQDExRrdWJlcm5ldGVzLXNlcnZlci1jYTAgFw0yMTExMDQxMTI5MzVaGA8y
MTIxMTEwNTExMjkzNVowHzEdMBsGA1UEAxMUa3ViZXJuZXRlcy1zZXJ2ZXItY2Ew
djAQBgcqhkjOPQIBBgUrgQQAIgNiAASfP1v3WpEVpE6yOA3mobryrPFcN4bTrQb5
dhCry11i/7Np4dnu/HhWc8PJ690KUcWf2JhwEaaM6UQrXfUv57LMPmPHM5QxtBEl
1jTE94DLZDTBhRLGJm2VRTOxnzhJVjGjMjAwMA8GA1UdEwEB/wQFMAMBAf8wHQYD
VR0OBBYEFAOuyJo6npIo3zfnJW/I0Idv2s+zMAoGCCqGSM49BAMDA2gAMGUCMQDS
+ZrflVDA7lap8xWsOCE6HenWoVa5UQdebaIgpg0Q54i/k0Ef/9QOjLy8aDsyZvcC
MBKYFIQShw+gt+YG0j/GQx3TEu8kY1QqdAQ9Du7wuI3bemCwq7aVl9OijBqiwqXA
4Q==
-----END CERTIFICATE-----
EOT -> null
- early_renewal_hours = 0 -> null
- id = "162007174841491717249967010213281523901" -> null
- is_ca_certificate = true -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-05T05:29:35.200303447-06:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-11-04T06:29:35.200303447-05:00" -> null
- subject {
- common_name = "kubernetes-server-ca" -> null
- street_address = [] -> null
}
}
# module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"] will be destroyed
- resource "tls_self_signed_cert" "kubernetes_ca_certs" {
- allowed_uses = [
- "critical",
- "digitalSignature",
- "keyEncipherment",
- "keyCertSign",
] -> null
- cert_pem = <<-EOT
-----BEGIN CERTIFICATE-----
MIIBxzCCAUygAwIBAgIRALS8aNYfo/FjXO72cIOpgtgwCgYIKoZIzj0EAwMwKzEp
MCcGA1UEAxMga3ViZXJuZXRlcy1yZXF1ZXN0LWhlYWRlci1rZXktY2EwIBcNMjEx
MTA0MTEyOTM1WhgPMjEyMTExMDUxMTI5MzVaMCsxKTAnBgNVBAMTIGt1YmVybmV0
ZXMtcmVxdWVzdC1oZWFkZXIta2V5LWNhMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE
ipNxZD68KxBzbchbAcU8PGX/xaUS7Ekg4hKZtwr3lD/JzYHuCb9lDxJZCIZShO6v
mOQcXQfRYJDYmEX/aqrvcmhkurXeaWW3StvYozf0EG2aflpSVacXIq5lTr+vxkEh
ozIwMDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSVibFi29JIU5ojwL4+tsR1
/o3r1zAKBggqhkjOPQQDAwNpADBmAjEA4ZZRkVGOOSUZHCTuT99ixNPECpaQGxWx
LADh+RpHj7Q076EOyAQiY4oMVpFZ6KpyAjEA0RfXA/g+y7fHLlHBf3NDyMr9xZ1d
/tlVW6bQ3qkuvr6uot11AxWykjdK/HfIvKG/
-----END CERTIFICATE-----
EOT -> null
- early_renewal_hours = 0 -> null
- id = "240239317385907192679446816919634150104" -> null
- is_ca_certificate = true -> null
- key_algorithm = "ECDSA" -> null
- private_key_pem = (sensitive value)
- ready_for_renewal = false -> null
- validity_end_time = "2121-11-05T05:29:35.200960081-06:00" -> null
- validity_period_hours = 876600 -> null
- validity_start_time = "2021-11-04T06:29:35.200960081-05:00" -> null
- subject {
- common_name = "kubernetes-request-header-key-ca" -> null
- street_address = [] -> null
}
}
Plan: 0 to add, 0 to change, 29 to destroy.
Changes to Outputs:
- kube_config = (sensitive value)
╷
│ Warning: Deprecated Resource
│
│ with data.civo_template.ubuntu,
│ on main.tf line 19, in data "civo_template" "ubuntu":
│ 19: data "civo_template" "ubuntu" {
│
│ "civo_template" data source is deprecated. Moving forward, please use "civo_disk_image" data source.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: Attribute is deprecated
│
│ with civo_instance.node_instances,
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
╵
╷
│ Warning: Deprecated Attribute
│
│ with civo_instance.node_instances[1],
│ on main.tf line 42, in resource "civo_instance" "node_instances":
│ 42: template = element(data.civo_template.ubuntu.templates, 0).id
│
│ "template" attribute is deprecated. Moving forward, please use "disk_image" attribute.
│
│ (and 2 more similar warnings elsewhere)
╵
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
module.k3s.null_resource.kubernetes_ready: Destroying... [id=2551799405757945470]
module.k3s.null_resource.kubernetes_ready: Destruction complete after 0s
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Destroying... [id=1540572406495879692]
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Destroying... [id=2100247009859532487]
module.k3s.null_resource.servers_drain["node-3"]: Destroying... [id=7048446702711938358]
module.k3s.tls_locally_signed_cert.master_user[0]: Destroying... [id=269975477633753601419630313929603900630]
module.k3s.null_resource.servers_drain["node-2"]: Destroying... [id=3646547839837205088]
module.k3s.null_resource.servers_drain["node-1"]: Destroying... [id=8691228253726825796]
module.k3s.tls_locally_signed_cert.master_user[0]: Destruction complete after 0s
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Destroying... [id=8481180087172316037]
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): User: root
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Password: true
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Private key: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_drain["node-2"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): User: root
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Password: true
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Private key: false
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_drain["node-3"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): User: root
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Password: true
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Private key: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): User: root
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Password: true
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Private key: false
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_drain["node-1"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Provisioning with 'remote-exec'...
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): User: root
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Password: true
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Private key: false
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Target Platform: unix
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Connecting to remote host via SSH...
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Host: 212.2.243.199
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): User: root
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Password: true
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Private key: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Certificate: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): SSH Agent: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Checking Host Key: false
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Target Platform: unix
module.k3s.tls_cert_request.master_user[0]: Destroying... [id=5457fd7b6ff35fb465293f5ac48cf51f73f604da]
module.k3s.tls_cert_request.master_user[0]: Destruction complete after 0s
module.k3s.tls_private_key.master_user[0]: Destroying... [id=56e5b1cf3fb10387213633492993251aed8a2c5c]
module.k3s.tls_private_key.master_user[0]: Destruction complete after 0s
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): Connected!
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): Connected!
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Connected!
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Connected!
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Connected!
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): Connected!
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): node/node-2 cordoned
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): node/node-3 cordoned
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): node/node-1 cordoned
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"] (remote-exec): node/node-1 labeled
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"] (remote-exec): node/node-3 labeled
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"] (remote-exec): node/node-2 labeled
module.k3s.null_resource.servers_label["node-1|node.kubernetes.io/type"]: Destruction complete after 7s
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): WARNING: ignoring DaemonSet-managed Pods: default/svclb-azure-vote-front-m78qp
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): WARNING: ignoring DaemonSet-managed Pods: default/svclb-azure-vote-front-2gqpx
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): WARNING: ignoring DaemonSet-managed Pods: default/svclb-azure-vote-front-gfr8p
module.k3s.null_resource.servers_label["node-3|node.kubernetes.io/type"]: Destruction complete after 7s
module.k3s.null_resource.servers_label["node-2|node.kubernetes.io/type"]: Destruction complete after 7s
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): evicting pod default/vote-front-azure-vote-1636025969-784d87dbbd-t274z
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): evicting pod kube-system/local-path-provisioner-64ffb68fd-4mkqp
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): evicting pod kube-system/coredns-85cb69466-6hlbq
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): evicting pod kube-system/helm-install-traefik-crd--1-s7vzg
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): evicting pod kube-system/metrics-server-9cf544f65-6dtnk
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): evicting pod default/vote-back-azure-vote-1636025969-8cf645cc4-9x9ck
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): evicting pod kube-system/helm-install-traefik--1-2qlsp
module.k3s.null_resource.servers_drain["node-3"]: Still destroying... [id=7048446702711938358, 10s elapsed]
module.k3s.null_resource.servers_drain["node-2"]: Still destroying... [id=3646547839837205088, 10s elapsed]
module.k3s.null_resource.servers_drain["node-1"]: Still destroying... [id=8691228253726825796, 10s elapsed]
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): pod/helm-install-traefik--1-2qlsp evicted
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): pod/vote-back-azure-vote-1636025969-8cf645cc4-9x9ck evicted
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): pod/metrics-server-9cf544f65-6dtnk evicted
module.k3s.null_resource.servers_drain["node-3"] (remote-exec): node/node-3 evicted
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): pod/helm-install-traefik-crd--1-s7vzg evicted
module.k3s.null_resource.servers_drain["node-3"]: Destruction complete after 12s
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): pod/coredns-85cb69466-6hlbq evicted
module.k3s.null_resource.servers_drain["node-1"]: Still destroying... [id=8691228253726825796, 20s elapsed]
module.k3s.null_resource.servers_drain["node-2"]: Still destroying... [id=3646547839837205088, 20s elapsed]
module.k3s.null_resource.servers_drain["node-2"]: Still destroying... [id=3646547839837205088, 30s elapsed]
module.k3s.null_resource.servers_drain["node-1"]: Still destroying... [id=8691228253726825796, 30s elapsed]
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): pod/vote-front-azure-vote-1636025969-784d87dbbd-t274z evicted
module.k3s.null_resource.servers_drain["node-2"] (remote-exec): node/node-2 evicted
module.k3s.null_resource.servers_drain["node-2"]: Destruction complete after 39s
module.k3s.null_resource.servers_drain["node-1"]: Still destroying... [id=8691228253726825796, 40s elapsed]
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): pod/local-path-provisioner-64ffb68fd-4mkqp evicted
module.k3s.null_resource.servers_drain["node-1"] (remote-exec): node/node-1 evicted
module.k3s.null_resource.servers_drain["node-1"]: Destruction complete after 41s
module.k3s.null_resource.servers_install["node-3"]: Destroying... [id=7060114474816286895]
module.k3s.null_resource.servers_install["node-2"]: Destroying... [id=1951024970189533157]
module.k3s.null_resource.servers_install["node-1"]: Destroying... [id=8440407648830712598]
module.k3s.null_resource.servers_install["node-3"]: Destruction complete after 0s
module.k3s.null_resource.servers_install["node-1"]: Destruction complete after 0s
module.k3s.null_resource.servers_install["node-2"]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[2]: Destroying... [id=5905791126692194777]
module.k3s.null_resource.k8s_ca_certificates_install[0]: Destroying... [id=5056326130768808998]
module.k3s.null_resource.k8s_ca_certificates_install[3]: Destroying... [id=7322411793949049427]
module.k3s.null_resource.k8s_ca_certificates_install[1]: Destroying... [id=5484088503243209669]
module.k3s.null_resource.k8s_ca_certificates_install[4]: Destroying... [id=776952705639779336]
module.k3s.null_resource.k8s_ca_certificates_install[5]: Destroying... [id=6663274023099510187]
module.k3s.random_password.k3s_cluster_secret: Destroying... [id=none]
module.k3s.random_password.k3s_cluster_secret: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[2]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[5]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[0]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[4]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[3]: Destruction complete after 0s
module.k3s.null_resource.k8s_ca_certificates_install[1]: Destruction complete after 0s
civo_instance.node_instances[2]: Destroying... [id=7666cd7d-48ad-4703-9253-481d6aa9e7a5]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Destroying... [id=162007174841491717249967010213281523901]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["1"]: Destruction complete after 0s
civo_instance.node_instances[0]: Destroying... [id=078aeab9-89da-46db-995b-d20fc2a026e7]
civo_instance.node_instances[1]: Destroying... [id=27cf0c45-25d9-47a8-abda-e0616fa68d9f]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Destroying... [id=240239317385907192679446816919634150104]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Destroying... [id=179381637326519421633541131708977756020]
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["2"]: Destruction complete after 0s
module.k3s.tls_self_signed_cert.kubernetes_ca_certs["0"]: Destruction complete after 0s
module.k3s.tls_private_key.kubernetes_ca[0]: Destroying... [id=ac1d6302a69b6a95ceaa0ab3513a0befc52eae1b]
module.k3s.tls_private_key.kubernetes_ca[1]: Destroying... [id=fb31ae6ed4b0cda025947155d636f41acc97932a]
module.k3s.tls_private_key.kubernetes_ca[2]: Destroying... [id=80ca4a9bebd3fa1437efef27571e4927e873a146]
module.k3s.tls_private_key.kubernetes_ca[0]: Destruction complete after 0s
module.k3s.tls_private_key.kubernetes_ca[1]: Destruction complete after 0s
module.k3s.tls_private_key.kubernetes_ca[2]: Destruction complete after 0s
civo_instance.node_instances[2]: Destruction complete after 1s
civo_instance.node_instances[0]: Destruction complete after 2s
civo_instance.node_instances[1]: Destruction complete after 2s
Destroy complete! Resources: 29 destroyed.
And of course we will see the compute is removed:
*pro-tip: it’s always worth double checking your volumes in case anything made PVCs. Not all cluster removal systems account for Persistant Volume Claims made outside of terraform*
Azure Kubernetes Service (AKS)
AKS, like most PaaS offerings is a bit easier to upgrade. This cluster was created in our last blog, but you can follow steps for AKS create in the aks cli docs
First we can see our version here:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-34481330-vmss000000 Ready agent 7d23h v1.20.9
aks-nodepool1-34481330-vmss000001 Ready agent 7d23h v1.20.9
aks-nodepool1-34481330-vmss000002 Ready agent 7d23h v1.20.9
Next we find our eligible upgrades:
$ az aks get-upgrades -n basicclusteridj -g mydemoaksrg -o table
Name ResourceGroup MasterVersion Upgrades
------- --------------- --------------- --------------
default mydemoaksrg 1.20.9 1.21.1, 1.21.2
Upgrade the cluster (master and nodes)
$ az aks upgrade -n basicclusteridj -g mydemoaksrg --kubernetes-version 1.21.2 --no-wait
Kubernetes may be unavailable during cluster upgrades.
Are you sure you want to perform this operation? (y/N): y
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version 1.21.2. Continue? (y/N): y
We can then see the nodes get updated
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-34481330-vmss000000 Ready agent 47s v1.21.2
aks-nodepool1-34481330-vmss000001 Ready,SchedulingDisabled agent 8d v1.20.9
aks-nodepool1-34481330-vmss000002 Ready agent 8d v1.20.9
aks-nodepool1-34481330-vmss000003 Ready agent 3m8s v1.21.2
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-34481330-vmss000000 Ready agent 78s v1.21.2
aks-nodepool1-34481330-vmss000002 Ready agent 8d v1.20.9
aks-nodepool1-34481330-vmss000003 Ready agent 3m39s v1.21.2
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-34481330-vmss000000 Ready agent 4m5s v1.21.2
aks-nodepool1-34481330-vmss000001 Ready agent 91s v1.21.2
aks-nodepool1-34481330-vmss000002 Ready,SchedulingDisabled agent 8d v1.20.9
aks-nodepool1-34481330-vmss000003 Ready agent 6m26s v1.21.2
While we are waiting on our nodes to upgrade, we can still see what upgrades we can do next (however we must wait to upgrade)
$ az aks get-upgrades -n basicclusteridj -g mydemoaksrg -o table
Name ResourceGroup MasterVersion Upgrades
------- --------------- --------------- --------------------------------
default mydemoaksrg 1.21.2 1.22.1(preview), 1.22.2(preview)
If you try and upgrade before the last completes, you’ll see an error:
$ az aks upgrade -n basicclusteridj -g mydemoaksrg --kubernetes-version 1.22.2 --no-wait
Kubernetes may be unavailable during cluster upgrades.
Are you sure you want to perform this operation? (y/N): y
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version 1.22.2. Continue? (y/N): y
Operation failed with status: 'Conflict'. Details: Operation is not allowed: Another operation (nodepool1 - Upgrading) is in progress, please wait for it to finish before starting a new operation. See https://aka.ms/aks-pending-operation for more details
If something goes wrong, you can try again:
$ az aks upgrade -n basicclusteridj -g mydemoaksrg --kubernetes-version 1.21.2 --no-wait
Kubernetes may be unavailable during cluster upgrades.
Are you sure you want to perform this operation? (y/N): y
Cluster currently in failed state. Proceeding with upgrade to existing version 1.21.2 to attempt resolution of failed cluster state.
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version 1.21.2. Continue? (y/N): y
When this upgrade has completed:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-34481330-vmss000000 Ready agent 58m v1.21.2
aks-nodepool1-34481330-vmss000003 Ready agent 60m v1.21.2
aks-nodepool1-34481330-vmss000004 Ready agent 5m37s v1.21.2
then we can upgrade again
$ az aks upgrade -n basicclusteridj -g mydemoaksrg --kubernetes-version 1.22.2 --no-wait
Kubernetes may be unavailable during cluster upgrades.
Are you sure you want to perform this operation? (y/N): y
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version 1.22.2. Continue? (y/N): y
If we get to the max level, then we see null as the option
$ az aks get-upgrades -n basicclusteridj -g mydemoaksrg | grep upgrades
"upgrades": null
Cleanup
Removal of the AKS cluster is just as easy
$ az account set --subscription "Visual Studio Enterprise Subscription"
$ az aks list -o table
Name Location ResourceGroup KubernetesVersion ProvisioningState Fqdn
--------------- ---------- --------------- ------------------- ------------------- --------------------------------------------------------------
basicclusteridj centralus mydemoaksrg 1.22.2 Failed basicclust-mydemoaksrg-d4c094-143803e4.hcp.centralus.azmk8s.io
$ az aks delete -n basicclusteridj -g mydemoaksrg
Are you sure you want to perform this operation? (y/n): y
Minikube
Assuming you have Minikube on the current stable version:
C:\WINDOWS\system32>minikube start
* minikube v1.23.2 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
* Automatically selected the hyperv driver
* Starting control plane node minikube in cluster minikube
* Creating hyperv VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
* Preparing Kubernetes v1.22.2 on Docker 20.10.8 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Configuring RBAC rules ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
C:\WINDOWS\system32>kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 40s v1.22.2
You can use a specific version or ‘latest’ to just upgrade the kubernetes image
C:\WINDOWS\system32>minikube start --kubernetes-version latest
* minikube v1.23.2 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
* Using the hyperv driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Downloading Kubernetes v1.22.3-rc.0 preload ...
> preloaded-images-k8s-v13-v1...: 511.71 MiB / 511.71 MiB 100.00% 8.09 MiB
* Updating the running hyperv "minikube" VM ...
* Preparing Kubernetes v1.22.3-rc.0 on Docker 20.10.8 ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
C:\WINDOWS\system32>kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 4m24s v1.22.3-rc.0
We can install a specific version of k8s as well:
C:\WINDOWS\system32>minikube start --kubernetes-version=v1.19.3
* minikube v1.23.2 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
* Automatically selected the hyperv driver
* Starting control plane node minikube in cluster minikube
* Creating hyperv VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
> kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubectl: 41.01 MiB / 41.01 MiB [---------------] 100.00% 3.18 MiB p/s 13s
> kubeadm: 37.30 MiB / 37.30 MiB [---------------] 100.00% 2.43 MiB p/s 16s
> kubelet: 104.91 MiB / 104.91 MiB [-------------] 100.00% 4.79 MiB p/s 22s
- Generating certificates and keys ...
- Booting up control plane ...
- Configuring RBAC rules ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
! C:\ProgramData\chocolatey\bin\kubectl.exe is version 1.22.2, which may have incompatibilites with Kubernetes 1.19.3.
- Want kubectl v1.19.3? Try 'minikube kubectl -- get pods -A'
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
C:\WINDOWS\system32>kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready master 30s v1.19.3
Let’s add a to-be-deprecated API for Ingress:
C:\Users\isaac\.kube>kubectl apply -f C:\Users\isaac\Downloads\ingresstest.yaml
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/ingress-v1betatest created
C:\Users\isaac\.kube>cd ..
C:\Users\isaac>kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-v1betatest <none> * 80 56s
C:\Users\isaac>kubectl get ingress ingress-v1betatest -oyaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"ingress-v1betatest","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"nginx-service","servicePort":80},"path":"/path1"}]}}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-11-04T03:34:25Z"
generation: 1
name: ingress-v1betatest
namespace: default
resourceVersion: "862"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/ingress-v1betatest
uid: a7884204-4f60-44fa-b973-489c33469a4b
spec:
rules:
- http:
paths:
- backend:
serviceName: nginx-service
servicePort: 80
path: /path1
pathType: ImplementationSpecific
status:
loadBalancer: {}
Now upgrade the cluster to the latest k8s version
C:\WINDOWS\system32>minikube start --kubernetes-version latest
* minikube v1.23.2 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
* Using the hyperv driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Updating the running hyperv "minikube" VM ...
! This VM is having trouble accessing https://k8s.gcr.io
* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
* Preparing Kubernetes v1.22.3-rc.0 on Docker 20.10.8 ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
and we can see that the ingress was updated to networking.k8s.io/v1:
C:\WINDOWS\system32>kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-v1betatest <none> * 80 6m56s
C:\WINDOWS\system32>kubectl get ingress ingress-v1betatest -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"ingress-v1betatest","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"nginx-service","servicePort":80},"path":"/path1"}]}}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-11-04T03:34:25Z"
generation: 1
name: ingress-v1betatest
namespace: default
resourceVersion: "862"
uid: a7884204-4f60-44fa-b973-489c33469a4b
spec:
rules:
- http:
paths:
- backend:
service:
name: nginx-service
port:
number: 80
path: /path1
pathType: ImplementationSpecific
status:
loadBalancer: {}
Using Rancher
Next we’ll setup Rancher in our production cluster then use it to create a K3s cluster in Azure.
Adding Rancher
Add the Rancher chart
$ helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
"rancher-latest" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "rancher-latest" chart repository
Update Complete. ⎈Happy Helming!⎈
Create the cattle ns
$ kubectl create namespace cattle-system
namespace/cattle-system created
Next i need to setup an ingress
Since we already have cert-manager, i can skip installing that as Rancher’s helm chart is setup for LE already.
Now we can install
$ helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.freshbrewed.science --set replicas=3 --set ingress.tls.source=letsEncrypt --set letsEncrypt.email=isaac.johnson@gmail.com
NAME: rancher
LAST DEPLOYED: Thu Nov 4 07:16:10 2021
NAMESPACE: cattle-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Rancher Server has been installed.
NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued, Containers are started and the Ingress rule comes up.
Check out our docs at https://rancher.com/docs/
If you provided your own bootstrap password during installation, browse to https://rancher.freshbrewed.science to get started.
If this is the first time you installed Rancher, get started by running this command and clicking the URL it generates:
echo https://rancher.freshbrewed.science/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')
To get just the bootstrap password on its own, run:
kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{ "\n" }}'
Happy Containering!
Now i can login to the dashboard to finish setup
then get the secret it needs
$ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'
sfzgbqasdfasdfasfasdfasfdasdfasdfasdfasdftrzgf
Now we can login and see our current cluster already loaded:
Add k3s on Azure through Rancher
If you setup on Azure you can choose to have different Node templates (sizes and even subscriptions) for each type of node and nodepool. Here you see two pools to be created:
You can also specify the Azure Credentials in the “Cloud Credentials” screen:
However, the easiest path i found was to add the cloud credentials for a SP (that has at least Contrib rights) inline of the Node Template:
The simplest setup just being one node. Here I will also set not-the-latest K8s version:
Once I apply we can see the create in progress:
I can also see this reflected in Azure:
We see that Rancher just names the VM “1”
We can see the state change from Processing:
To Active (done) in the Cluster list:
From there, we can use the “Download KubeConfig” dropdown to get the Kube config file.
With the downloaded Kubeconfig, we can see the cluster locally from WSL:
$ cp /mnt/c/Users/isaac/Downloads/myazurepaygk3s.yaml ~/.kube/config
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
1 Ready controlplane,etcd,worker 37m v1.20.12
And then install a sample chart:
$ helm install azure-samples/azure-vote --generate-name
NAME: azure-vote-1636032387
LAST DEPLOYED: Thu Nov 4 08:26:29 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Azure Vote application has been started on your Kubernetes cluster.
Title: Azure Vote App
Vote 1 value: Cats
Vote 2 value: Dogs
The externally accessible IP address can take a minute or so to provision. Run the following command to monitor the provisioning status. Once an External IP address has been provisioned, brows to this IP address to access the Azure Vote application.
kubectl get service -l name=azure-vote-front -w
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vote-back-azure-vote-1636032387-656b6795d-hn4kc 0/1 ContainerCreating 0 7s
vote-front-azure-vote-1636032387-b7fdbb848-sgfcd 0/1 ContainerCreating 0 7s
I can edit the config of the cluster in Rancher to change versions:
And when we save, we see it’s now updating
It takes a few to see that reflected locally:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
1 Ready controlplane,etcd,worker 41m v1.20.12
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
1 Ready,SchedulingDisabled controlplane,etcd,worker 42m v1.20.12
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
1 Ready controlplane,etcd,worker 43m v1.21.6
and i can see that indeed it’s still running pods:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vote-back-azure-vote-1636032387-656b6795d-hn4kc 1/1 Running 0 4m51s
vote-front-azure-vote-1636032387-b7fdbb848-sgfcd 1/1 Running 0 4m51s
and we see the new version reflected in our cluster list as well.
cleanup
We can delete the cluster from Rancher
And it makes you control-click to confirm:
This then starts the deletion process:
Pretty soon i see things being removed using the kubeconfig:
$ kubectl get pods
error: the server doesn't have a resource type "pods"
$ kubectl get nodes
error: the server doesn't have a resource type "nodes"
and see the RG being cleaned up in Azure
Summary
Hopefully this article took you through the myriad of ways to upgrade Kubernetes with a focus on lighter weight options like k3s. There, of course, can be issues, but by in large, when things are deprecated, they are done so in favour of new APIs or versions. My experience upgrading the above showed it generally goes smooth, though some methods are considerably more time consuming than others.
