Published: Sep 25, 2019 by Isaac Johnson

One question that often comes up is how do we properly create Infrastructure as Code.  While in a way this is akin to asking which political party is best or is cilantro tasty, there are patterns I’ve found useful that you may want to leverage without having to spend a fortune.  And for the record, cilantro tastes terrible…

The tools

First we’ll talk first at a high level in a tools agnostic approach and then dig into the tool chains I like to use; namely Terraform OS and Azure DevOps (VSTS).  The approach works just as well with cloudformation templates and variables, or ARM template parameters or even values passed to the doctl CLI.  so use what works best for your particular environment.

Overview

In general, there are two patterns that seem prevalent today.  

[1] All in One

The first is to store all of your IaC code into a single repository, such as “terraform” or “infrastructure-repo”.  This has the advantage of being able to share modules and keep all code of a type together.  

However, just as most companies don’t have a “java” repo nor “dotnet”, the co-mingling of many products in one space can create maintenance challenges.  Namely, sometimes we don’t want a shared module live to all products that consume it.  

Take for instance a “our_vmss” or “our_asg”.  We may want to move AMIs or VMIDs or change the machine size for one product, but not another (maybe it makes sense to scale up our webserver but not the fleet of build machines).  Another issue is that over time, this large repo has a very big blast radius - a seemingly innocuous change for one infrastructure project could affect many different others that consume from the same code.

[2] Repo Per Thing

The second approach is to break your infrastructure down into different repos for different projects.  This has the advantage of minimizing blast radius - one just affects the infrastructure in the project and code reviews can focus just on the infrastructure in question.  

However, there is the cost that shared modules need to be transmitted between repos.  That said, there are a few strategies that can be employed that we can borrow from standard development practices such as publishing shared modules to an artifact store, or forking our IAC repos and cross merging.

Using AzDO pipelines and Terraform

In the above example we put our Terraform for Azure Kubernetes Service (AKS) and Azure Container Service (ACR) into a common repo.  This will be our k8s infra repo. We can then use branches per environment to control environment specific changes.  These can include having different size clusters (horizontal and vertical) as well as using standard sku for ACR in Dev but the Premium SKU with GRS for Production.

The other advantage you may see is in the use of a secret store such as AKV to store the tokens used to access the cluster.  This can be the kubeconfig or perhaps the URL and token in an RBAC cluster.  This would be transmitted by the IAC CD pipeline into AKV for use in the development environments.

As we see in the diagram above, the CD pipeline for Dev consumes the token/config in the prepare stage.  

The areas of this diagram that are not fully fleshed out are in the test and rollback as that largely depends on what you are deploying and how you would like to rollback.  In a kubernetes environment that can be to do Blue/Green deployments and moving an ILB. That can also be to do canary style deployments with a rollback to the last known good chart.  

Each deployment should have at least values and images unique to that deployment, but they may use a common stable chart.  In many cases I’ve substituted the image name in a deployment.yaml or value.yaml that is passed on to CD.

An example

Let’s take the terraform we used in prior guides to fire off a CI pipeline.  In this case, we will validate and store the zipped up contents as a drop.zip (pipeline artifact).

Next, in our CD process we will add a branch filter to gate the deployments to just the master or develop branch.

At the end of the deployment, we need to capture a few things.  In a non-RBAC cluster, the k8s kubeconfig is sufficient.  However, regardless of RBAC, it’s better to create token user and use the token.  Unlike kubeconfigs, tokens can be revoked and rotated as needed and allow finer grain controls.  In this regards, instead of a kubeconfig (like base64’ed), we’ll instead save the token and URL into our Azure Key Vault (AKV)

Let’s look at those last 3 steps:

Bash: Get SA Token

# Write your commands here

kubectl -n myns get secret $(kubectl -n myns get secret | grep myakssa | awk '{print $1}') -o json | jq -r '.data.token' | base64 --decode > token.txt

Bash: Set token to env var

#!/bin/bash
set +x

export tval=`cat token.txt`
echo "##vso[task.setvariable variable=TOKENVAL]$tval" > t.o

set -x

cat ./t.o

Azure CLI:

az keyvault secret set --vault-name infrakv --name aks-token --value $(TOKENVAL)

az keyvault secret set --vault-name infrakv --name aks-namespace --value myaks

Next, in our development pipeline we can grab those settings in a CD stage via Managed Variables. Because they are pulled in at deployment time, we can be sure the latest values are used.  That is, if the IaC pipeline is fired, it will update the key vault and anytime the dev CD pipeline runs, those latest values will be used.

Further, we can actually chain CD pipelines to launch multiple variants of the same infrastructure for multiple environments.  Below we can see how we can link stages with different settings:

Summary

Terraform OS offer a powerful mechanism to create Infrastructure that needs not to be over complicated.  We don’t need TFE/Terraform Cloud with it’s workspaces just to launch different environments.  However, don’t see that as a statement that TFE/TF Cloud doesn’t have it’s purpose.  One key issue with this system is that the builder group that manages AzDO can ultimately see any of these values.  So likely organizations that wish to scale and control production environments will find great value in the commercial side of Terraform.