Published: Oct 12, 2022 by Isaac Johnson
One of the more interesting announcements from Hashiconf this year was native OPA support in Terraform Cloud. Open Policy Agent has become the standard for policy checks and enforcement.
Today we’ll explore the various ways we can tie OPA in with Terraform, both via TF Cloud and via pipelines.
Terraform Cloud
I’ve had Terraform Cloud since it was announced years ago. I checked my instance and found the last run was years ago when 0.14 was released.
I’m operating on the free plan which means I likely do not get the policy checks:
That said, let’s see what we can do with the free plan and if we may not even need built in checks to Terraform if I can run OPA directly.
Create a Terraform Repo
In the past, I directly engaged with TF Cloud via the CLI. Let’s setup a more normal GIT driven flow.
I’ll make this public so others can easily clone and re-use. I’ll set an MIT license and give Security Tower (one more try)
Now that I have the repo, I can clone and add some content.
Update Terraform / tfenv
My local terraform is out of date. I use tfenv to easily switch between versions.
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ tfenv install 1.3.2
Installing Terraform v1.3.2
Downloading release tarball from https://releases.hashicorp.com/terraform/1.3.2/terraform_1.3.2_linux_amd64.zip
######################################################################################################################################### 100.0%
Downloading SHA hash file from https://releases.hashicorp.com/terraform/1.3.2/terraform_1.3.2_SHA256SUMS
No keybase install found, skipping OpenPGP signature verification
Archive: /tmp/tfenv_download.it1Bov/terraform_1.3.2_linux_amd64.zip
inflating: /home/linuxbrew/.linuxbrew/Cellar/tfenv/2.2.3/versions/1.3.2/terraform
Installation of terraform v1.3.2 successful. To make this your default version, run 'tfenv use 1.3.2'
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ tfenv use 1.3.2
Switching default version to v1.3.2
Switching completed
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$
Now that we have TF 1.3.2
$ terraform version
Terraform v1.3.2
on linux_amd64
Next, I’ll create a basic main.tf that has an AWS ASG and instance using 20.04 LTS Ubuntu (here)
$ cat main.tf
provider "aws" {
region = "us-west-1"
}
resource "aws_instance" "web" {
instance_type = "t2.micro"
ami = "ami-02ef5566632e904f8"
}
resource "aws_autoscaling_group" "my_asg" {
availability_zones = ["us-west-1a"]
name = "my_asg"
max_size = 5
min_size = 1
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 4
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_launch_configuration" "my_web_config" {
name = "my_web_config"
image_id = "ami-02ef5566632e904f8"
instance_type = "t2.micro"
}
I’ll next init my directory
$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v4.34.0...
- Installed hashicorp/aws v4.34.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Lastly, I’ll do a plan
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ terraform plan --out tfplan.binary
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_autoscaling_group.my_asg will be created
+ resource "aws_autoscaling_group" "my_asg" {
+ arn = (known after apply)
+ availability_zones = [
+ "us-west-1a",
]
+ default_cooldown = (known after apply)
+ desired_capacity = 4
+ force_delete = true
+ force_delete_warm_pool = false
+ health_check_grace_period = 300
+ health_check_type = "ELB"
+ id = (known after apply)
+ launch_configuration = "my_web_config"
+ max_size = 5
+ metrics_granularity = "1Minute"
+ min_size = 1
+ name = "my_asg"
+ name_prefix = (known after apply)
+ protect_from_scale_in = false
+ service_linked_role_arn = (known after apply)
+ vpc_zone_identifier = (known after apply)
+ wait_for_capacity_timeout = "10m"
}
# aws_instance.web will be created
+ resource "aws_instance" "web" {
+ ami = "ami-02ef5566632e904f8"
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ disable_api_stop = (known after apply)
+ disable_api_termination = (known after apply)
+ ebs_optimized = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ host_resource_group_arn = (known after apply)
+ id = (known after apply)
+ instance_initiated_shutdown_behavior = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t2.micro"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = (known after apply)
+ monitoring = (known after apply)
+ outpost_arn = (known after apply)
+ password_data = (known after apply)
+ placement_group = (known after apply)
+ placement_partition_number = (known after apply)
+ primary_network_interface_id = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ subnet_id = (known after apply)
+ tags_all = (known after apply)
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
+ capacity_reservation_specification {
+ capacity_reservation_preference = (known after apply)
+ capacity_reservation_target {
+ capacity_reservation_id = (known after apply)
+ capacity_reservation_resource_group_arn = (known after apply)
}
}
+ ebs_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ snapshot_id = (known after apply)
+ tags = (known after apply)
+ throughput = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
+ enclave_options {
+ enabled = (known after apply)
}
+ ephemeral_block_device {
+ device_name = (known after apply)
+ no_device = (known after apply)
+ virtual_name = (known after apply)
}
+ maintenance_options {
+ auto_recovery = (known after apply)
}
+ metadata_options {
+ http_endpoint = (known after apply)
+ http_put_response_hop_limit = (known after apply)
+ http_tokens = (known after apply)
+ instance_metadata_tags = (known after apply)
}
+ network_interface {
+ delete_on_termination = (known after apply)
+ device_index = (known after apply)
+ network_card_index = (known after apply)
+ network_interface_id = (known after apply)
}
+ private_dns_name_options {
+ enable_resource_name_dns_a_record = (known after apply)
+ enable_resource_name_dns_aaaa_record = (known after apply)
+ hostname_type = (known after apply)
}
+ root_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ tags = (known after apply)
+ throughput = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
}
# aws_launch_configuration.my_web_config will be created
+ resource "aws_launch_configuration" "my_web_config" {
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ ebs_optimized = (known after apply)
+ enable_monitoring = true
+ id = (known after apply)
+ image_id = "ami-02ef5566632e904f8"
+ instance_type = "t2.micro"
+ key_name = (known after apply)
+ name = "my_web_config"
+ name_prefix = (known after apply)
+ ebs_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ no_device = (known after apply)
+ snapshot_id = (known after apply)
+ throughput = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
+ metadata_options {
+ http_endpoint = (known after apply)
+ http_put_response_hop_limit = (known after apply)
+ http_tokens = (known after apply)
}
+ root_block_device {
+ delete_on_termination = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ throughput = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
}
Plan: 3 to add, 0 to change, 0 to destroy.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tfplan.binary
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan.binary"
We can run an OPA check against the JSON version of the binary output. Let’s send that plan to JSON
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ terraform show -json tfplan.binary > tfplan.json
We can check the file:
$ cat tfplan.json | jq
{
"format_version": "1.1",
"terraform_version": "1.3.2",
"planned_values": {
"root_module": {
"resources": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_name": "registry.terraform.io/hashicorp/aws",
"schema_version": 0,
"values": {
"availability_zones": [
"us-west-1a"
],
"capacity_rebalance": null,
"context": null,
"default_instance_warmup": null,
"desired_capacity": 4,
"enabled_metrics": null,
"force_delete": true,
"force_delete_warm_pool": false,
"health_check_grace_period": 300,
"health_check_type": "ELB",
"initial_lifecycle_hook": [],
"instance_refresh": [],
"launch_configuration": "my_web_config",
"launch_template": [],
"load_balancers": null,
"max_instance_lifetime": null,
"max_size": 5,
"metrics_granularity": "1Minute",
"min_elb_capacity": null,
"min_size": 1,
"mixed_instances_policy": [],
"name": "my_asg",
"placement_group": null,
"protect_from_scale_in": false,
"suspended_processes": null,
"tag": [],
"tags": null,
"target_group_arns": null,
"termination_policies": null,
"timeouts": null,
"wait_for_capacity_timeout": "10m",
"wait_for_elb_capacity": null,
"warm_pool": []
},
"sensitive_values": {
"availability_zones": [
false
],
"initial_lifecycle_hook": [],
"instance_refresh": [],
"launch_template": [],
"mixed_instances_policy": [],
"tag": [],
"vpc_zone_identifier": [],
"warm_pool": []
}
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_name": "registry.terraform.io/hashicorp/aws",
"schema_version": 1,
"values": {
"ami": "ami-02ef5566632e904f8",
"credit_specification": [],
"get_password_data": false,
"hibernation": null,
"iam_instance_profile": null,
"instance_type": "t2.micro",
"launch_template": [],
"source_dest_check": true,
"tags": null,
"timeouts": null,
"user_data_replace_on_change": false,
"volume_tags": null
},
"sensitive_values": {
"capacity_reservation_specification": [],
"credit_specification": [],
"ebs_block_device": [],
"enclave_options": [],
"ephemeral_block_device": [],
"ipv6_addresses": [],
"launch_template": [],
"maintenance_options": [],
"metadata_options": [],
"network_interface": [],
"private_dns_name_options": [],
"root_block_device": [],
"secondary_private_ips": [],
"security_groups": [],
"tags_all": {},
"vpc_security_group_ids": []
}
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_name": "registry.terraform.io/hashicorp/aws",
"schema_version": 0,
"values": {
"enable_monitoring": true,
"ephemeral_block_device": [],
"iam_instance_profile": null,
"image_id": "ami-02ef5566632e904f8",
"instance_type": "t2.micro",
"name": "my_web_config",
"placement_tenancy": null,
"security_groups": null,
"spot_price": null,
"user_data": null,
"user_data_base64": null,
"vpc_classic_link_id": null,
"vpc_classic_link_security_groups": null
},
"sensitive_values": {
"ebs_block_device": [],
"ephemeral_block_device": [],
"metadata_options": [],
"root_block_device": []
}
}
]
}
},
"resource_changes": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_name": "registry.terraform.io/hashicorp/aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"availability_zones": [
"us-west-1a"
],
"capacity_rebalance": null,
"context": null,
"default_instance_warmup": null,
"desired_capacity": 4,
"enabled_metrics": null,
"force_delete": true,
"force_delete_warm_pool": false,
"health_check_grace_period": 300,
"health_check_type": "ELB",
"initial_lifecycle_hook": [],
"instance_refresh": [],
"launch_configuration": "my_web_config",
"launch_template": [],
"load_balancers": null,
"max_instance_lifetime": null,
"max_size": 5,
"metrics_granularity": "1Minute",
"min_elb_capacity": null,
"min_size": 1,
"mixed_instances_policy": [],
"name": "my_asg",
"placement_group": null,
"protect_from_scale_in": false,
"suspended_processes": null,
"tag": [],
"tags": null,
"target_group_arns": null,
"termination_policies": null,
"timeouts": null,
"wait_for_capacity_timeout": "10m",
"wait_for_elb_capacity": null,
"warm_pool": []
},
"after_unknown": {
"arn": true,
"availability_zones": [
false
],
"default_cooldown": true,
"id": true,
"initial_lifecycle_hook": [],
"instance_refresh": [],
"launch_template": [],
"mixed_instances_policy": [],
"name_prefix": true,
"service_linked_role_arn": true,
"tag": [],
"vpc_zone_identifier": true,
"warm_pool": []
},
"before_sensitive": false,
"after_sensitive": {
"availability_zones": [
false
],
"initial_lifecycle_hook": [],
"instance_refresh": [],
"launch_template": [],
"mixed_instances_policy": [],
"tag": [],
"vpc_zone_identifier": [],
"warm_pool": []
}
}
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_name": "registry.terraform.io/hashicorp/aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"ami": "ami-02ef5566632e904f8",
"credit_specification": [],
"get_password_data": false,
"hibernation": null,
"iam_instance_profile": null,
"instance_type": "t2.micro",
"launch_template": [],
"source_dest_check": true,
"tags": null,
"timeouts": null,
"user_data_replace_on_change": false,
"volume_tags": null
},
"after_unknown": {
"arn": true,
"associate_public_ip_address": true,
"availability_zone": true,
"capacity_reservation_specification": true,
"cpu_core_count": true,
"cpu_threads_per_core": true,
"credit_specification": [],
"disable_api_stop": true,
"disable_api_termination": true,
"ebs_block_device": true,
"ebs_optimized": true,
"enclave_options": true,
"ephemeral_block_device": true,
"host_id": true,
"host_resource_group_arn": true,
"id": true,
"instance_initiated_shutdown_behavior": true,
"instance_state": true,
"ipv6_address_count": true,
"ipv6_addresses": true,
"key_name": true,
"launch_template": [],
"maintenance_options": true,
"metadata_options": true,
"monitoring": true,
"network_interface": true,
"outpost_arn": true,
"password_data": true,
"placement_group": true,
"placement_partition_number": true,
"primary_network_interface_id": true,
"private_dns": true,
"private_dns_name_options": true,
"private_ip": true,
"public_dns": true,
"public_ip": true,
"root_block_device": true,
"secondary_private_ips": true,
"security_groups": true,
"subnet_id": true,
"tags_all": true,
"tenancy": true,
"user_data": true,
"user_data_base64": true,
"vpc_security_group_ids": true
},
"before_sensitive": false,
"after_sensitive": {
"capacity_reservation_specification": [],
"credit_specification": [],
"ebs_block_device": [],
"enclave_options": [],
"ephemeral_block_device": [],
"ipv6_addresses": [],
"launch_template": [],
"maintenance_options": [],
"metadata_options": [],
"network_interface": [],
"private_dns_name_options": [],
"root_block_device": [],
"secondary_private_ips": [],
"security_groups": [],
"tags_all": {},
"vpc_security_group_ids": []
}
}
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_name": "registry.terraform.io/hashicorp/aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"enable_monitoring": true,
"ephemeral_block_device": [],
"iam_instance_profile": null,
"image_id": "ami-02ef5566632e904f8",
"instance_type": "t2.micro",
"name": "my_web_config",
"placement_tenancy": null,
"security_groups": null,
"spot_price": null,
"user_data": null,
"user_data_base64": null,
"vpc_classic_link_id": null,
"vpc_classic_link_security_groups": null
},
"after_unknown": {
"arn": true,
"associate_public_ip_address": true,
"ebs_block_device": true,
"ebs_optimized": true,
"ephemeral_block_device": [],
"id": true,
"key_name": true,
"metadata_options": true,
"name_prefix": true,
"root_block_device": true
},
"before_sensitive": false,
"after_sensitive": {
"ebs_block_device": [],
"ephemeral_block_device": [],
"metadata_options": [],
"root_block_device": []
}
}
}
],
"configuration": {
"provider_config": {
"aws": {
"name": "aws",
"full_name": "registry.terraform.io/hashicorp/aws",
"expressions": {
"region": {
"constant_value": "us-west-1"
}
}
}
},
"root_module": {
"resources": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_config_key": "aws",
"expressions": {
"availability_zones": {
"constant_value": [
"us-west-1a"
]
},
"desired_capacity": {
"constant_value": 4
},
"force_delete": {
"constant_value": true
},
"health_check_grace_period": {
"constant_value": 300
},
"health_check_type": {
"constant_value": "ELB"
},
"launch_configuration": {
"constant_value": "my_web_config"
},
"max_size": {
"constant_value": 5
},
"min_size": {
"constant_value": 1
},
"name": {
"constant_value": "my_asg"
}
},
"schema_version": 0
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_config_key": "aws",
"expressions": {
"ami": {
"constant_value": "ami-02ef5566632e904f8"
},
"instance_type": {
"constant_value": "t2.micro"
}
},
"schema_version": 1
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_config_key": "aws",
"expressions": {
"image_id": {
"constant_value": "ami-02ef5566632e904f8"
},
"instance_type": {
"constant_value": "t2.micro"
},
"name": {
"constant_value": "my_web_config"
}
},
"schema_version": 0
}
]
}
}
}
For our OPA plan, we’ll focus on type and actions:
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ cat tfplan.json | jq '.resource_changes[] | .type'
"aws_autoscaling_group"
"aws_instance"
"aws_launch_configuration"
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ cat tfplan.json | jq '.resource_changes[] | .change.actions'
[
"create"
]
[
"create"
]
[
"create"
]
I’ll make a policy folder and use the policy from the OPA tutorial. I added it to the repo here
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo/policy$ vi terraform.rego
# Consider exactly these resource types in calculations
resource_types := {"aws_autoscaling_group", "aws_instance", "aws_iam", "aws_launch_configuration"}
#########
# Policy
#########
# Authorization holds if score for the plan is acceptable and no changes are made to IAM
default authz := false
authz {
score < blast_radius
not touches_iam
}
# Compute the score for a Terraform plan as the weighted sum of deletions, creations, modifications
score := s {
all := [ x |
some resource_type
crud := weights[resource_type];
del := crud["delete"] * num_deletes[resource_type];
new := crud["create"] * num_creates[resource_type];
mod := crud["modify"] * num_modifies[resource_type];
x := del + new + mod
]
s := sum(all)
}
# Whether there is any change to IAM
touches_iam {
all := resources["aws_iam"]
count(all) > 0
}
####################
# Terraform Library
####################
# list of all resources of a given type
resources[resource_type] := all {
some resource_type
resource_types[resource_type]
all := [name |
name:= tfplan.resource_changes[_]
name.type == resource_type
]
}
# number of creations of resources of a given type
num_creates[resource_type] := num {
some resource_type
resource_types[resource_type]
all := resources[resource_type]
creates := [res | res:= all[_]; res.change.actions[_] == "create"]
num := count(creates)
}
# number of deletions of resources of a given type
num_deletes[resource_type] := num {
some resource_type
resource_types[resource_type]
all := resources[resource_type]
deletions := [res | res:= all[_]; res.change.actions[_] == "delete"]
num := count(deletions)
}
# number of modifications to resources of a given type
num_modifies[resource_type] := num {
some resource_type
resource_types[resource_type]
all := resources[resource_type]
modifies := [res | res:= all[_]; res.change.actions[_] == "update"]
num := count(modifies)
}
Running OPA locally
First, I need to install the Open Policy Agent binary if I havent already.
$ brew install opa
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
Updated 4 taps (codefresh-io/cli, homebrew/core, derailed/k9s and azure/functions).
==> New Formulae
autocorrect curlcpp fastfetch gebug highway json2tsv mycorrhiza nuraft snowball verovio
bazarr dronedb fend git-machete iir1 metview netcdf-cxx rome tbls
cbindgen edencommon fred go-camo jj mxnet netcdf-fortran sambamba tfel
You have 16 outdated formulae installed.
You can upgrade them with brew upgrade
or list them with brew outdated.
==> Downloading https://ghcr.io/v2/homebrew/core/xz/manifests/5.2.7
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/xz/blobs/sha256:dda25f66145c180884d0550a36d68491abd648011b9ac91566773961a1d921aa
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:dda25f66145c180884d0550a36d68491abd648011b9ac91566773961a1d921aa?se=2022-10-11T12%3A15%3A00Z&sig=YPEGqy0nHh4TFT
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/opa/manifests/0.45.0
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/opa/blobs/sha256:769143fb81bc1e78819a1f5e478c73ebfa79cb0d6fd2666c8873d31b0c9b5141
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:769143fb81bc1e78819a1f5e478c73ebfa79cb0d6fd2666c8873d31b0c9b5141?se=2022-10-11T12%3A15%3A00Z&sig=cY%2B9bV8ETuZS
######################################################################## 100.0%
==> Installing dependencies for opa: xz
==> Installing opa dependency: xz
==> Pouring xz--5.2.7.x86_64_linux.bottle.tar.gz
🍺 /home/linuxbrew/.linuxbrew/Cellar/xz/5.2.7: 151 files, 2.5MB
==> Installing opa
==> Pouring opa--0.45.0.x86_64_linux.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
/home/linuxbrew/.linuxbrew/etc/bash_completion.d
==> Summary
🍺 /home/linuxbrew/.linuxbrew/Cellar/opa/0.45.0: 24 files, 25.6MB
==> Running `brew cleanup opa`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Upgrading 3 dependents of upgraded formulae:
...snip...
Validation:
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ opa version
Version: 0.45.0
Build Commit:
Build Timestamp:
Build Hostname:
Go Version: go1.19.2
Platform: linux/amd64
WebAssembly: unavailable
We can now run OPA to check our policy
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ opa exec --decision terraform/analysis/authz --bundle policy/ tfplan.json
{
"result": [
{
"path": "tfplan.json",
"result": true
}
]
}
We can get a score as well (just change authz to score)
$ opa exec --decision terraform/analysis/score --bundle policy/ tfplan.json
{
"result": [
{
"path": "tfplan.json",
"result": 11
}
]
}
I can make a change that would create a rather sizable ASG
provider "aws" {
region = "us-west-1"
}
resource "aws_instance" "web" {
instance_type = "t2.micro"
ami = "ami-02ef5566632e904f8"
}
resource "aws_autoscaling_group" "my_asg" {
availability_zones = ["us-west-1a"]
name = "my_asg"
max_size = 70
min_size = 50
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 50
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_launch_configuration" "my_web_config" {
name = "my_web_config"
image_id = "ami-02ef5566632e904f8"
instance_type = "t2.micro"
}
The run TF and OPA to check
$ terraform init && terraform plan --out tfplan.binary && terraform show -json tfplan.binary > tfplan.json
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.34.0
Terraform has been successfully initialized!
...snip...
OPA Check:
$ opa exec --decision terraform/analysis/authz --bundle policy/ tfplan.json
{
"result": [
{
"path": "tfplan.json",
"result": true
}
]
}
$ opa exec --decision terraform/analysis/score --bundle policy/ tfplan.json
{
"result": [
{
"path": "tfplan.json",
"result": 11
}
]
}
But if I add too many ASGs
$ cat main.tf
provider "aws" {
region = "us-west-1"
}
resource "aws_instance" "web" {
instance_type = "t2.micro"
ami = "ami-02ef5566632e904f8"
}
resource "aws_autoscaling_group" "my_asg" {
availability_zones = ["us-west-1a"]
name = "my_asg"
max_size = 70
min_size = 50
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 50
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_launch_configuration" "my_web_config" {
name = "my_web_config"
image_id = "ami-02ef5566632e904f8"
instance_type = "t2.micro"
}
resource "aws_autoscaling_group" "my_asg2" {
availability_zones = ["us-west-1b"]
name = "my_asg"
max_size = 70
min_size = 50
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 50
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_autoscaling_group" "my_asg3" {
availability_zones = ["us-west-1c"]
name = "my_asg"
max_size = 70
min_size = 50
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 50
force_delete = true
launch_configuration = "my_web_config"
}
Then run a plan
$ terraform init && terraform plan --out tfplan.binary && terraform show -json tfplan.binary > tfplan.json
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.34.0
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_autoscaling_group.my_asg will be created
+ resource "aws_autoscaling_group" "my_asg" {
+ arn = (known after apply)
+ availability_zones = [
+ "us-west-1a",
...snip...
+ volume_type = (known after apply)
}
}
Plan: 5 to add, 0 to change, 0 to destroy.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tfplan.binary
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan.binary"
We can now see it fails the check
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ opa exec --decision terraform/analysis/authz --bundle policy/ tfplan.json | jq -r ".result[] |.result"
false
builder@DESKTOP-QADGF36:~/Workspaces/myTFCloudDemo$ opa exec --decision terraform/analysis/score --bundle policy/ tfplan.json | jq -r ".result[] |.result"
31
Using a Github Action
First, I’ll need some valid AWS creds so we can engage with the AWS Metadata API
Once set, I can then setup a workflow in .github/workflows
$ cat .github/workflows/testing.yml
name: PR And Main Build
on:
push:
branches:
- main
pull_request:
jobs:
build_deploy_test:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v2
- name: tests
run: |
set -x
curl -L -o opa https://github.com/open-policy-agent/opa/releases/download/v0.45.0/opa_linux_amd64
chmod u+x ./opa
terraform --version
./opa version
# Set Up Terraform
terraform init
terraform plan --out tfplan.binary
terraform show -json tfplan.binary > tfplan.json
# OPA Policy Checks
OPACHECK=`./opa exec --decision terraform/analysis/authz --bundle policy/ tfplan.json | jq -r ".result[] |.result"`
OPASCORE=`./opa exec --decision terraform/analysis/score --bundle policy/ tfplan.json | jq -r ".result[] |.result"`
echo $OPACHECK
echo $OPASCORE
# set env vars
echo "OPACHECK=$OPACHECK" >> $GITHUB_ENV
echo "OPASCORE=$OPASCORE" >> $GITHUB_ENV
env:
AWS_ACCESS_KEY_ID: $
AWS_SECRET_ACCESS_KEY: $
- name: tests2
run: |
set -x
echo "OPA Check Passed: $OPACHECK"
echo "OPA Score: $OPASCORE"
# Here we could actuall tf apply
if: env.OPACHECK != 'false'
- name: tests2
run: |
set +x
echo $OPACHECK
echo "OPA Check FAILED Score $OPASCORE"
MINSCORE=`cat policy/terraform.rego | grep 'blast_radius :=' | sed 's/^.* := //'`
echo "OPA Score Limit $MINSCORE"
exit 1
if: env.OPACHECK == 'false'
What we see presently, is with our main.tf having too many ASGs, it exceeds or max blast_radius score and fails the build
We can see that in the output
Now I’ll comment out the last 2 ASG creates and push the commit
Our build passes
And I can see that in the results as well
TF Cloud
Let’s now add to Terraform Cloud to see what checks we can do here
I’ll add a new workspace
I’ll chose Version Control based
Github
which pops up an Auth dialog
I can set the name and optionally add a description
There are some advanced setting there worth pointing out
For instance, if you wish to really auto-apply (and I would only do this with a tightly controlled private repo), then you can set “Auto apply” in the “Apply Method”, otherwise the default is Manual.
The other setting you may chose to update is the “Run Triggers”. If you put your terraform in a subfolder to your repo. Say, for instance, you have an Azure Function and the source is in “/src” you may wish to put the Infrastructure code in “/terraform” or “/iac”. Then it would be worthwhile to limit to that.
e.g.
Now that it’s added, we can set variables and/or start a plan
I’m fairly certain it will fail if I just plan as we need the AWS Secrets set, but let’s give it a try
And, as expected, it fails
That said, I was able to use a sample OPA Rule that would validate tags
Query: data.terraform.policies.policy1.deny
package terraform.policies.policy1
import input.plan as plan
import input.run as run
array_contains(arr, elem) {
arr[_] = elem
}
get_basename(path) = basename{
arr := split(path, "/")
basename:= arr[count(arr) - 1]
}
deny[reason] {
resource := plan.resource_changes[_]
action := resource.change.actions[count(resource.change.actions) - 1]
array_contains(["create", "update"], action)
cloud_tag := get_basename(resource.provider_name)
not run.workspace.tags[cloud_tag]
reason := sprintf("Workspace must be marked with '%s' tag to create resources in %s cloud",
[cloud_tag, cloud_tag])
}
Adding Variables
We can set TF Variables as well as Env Vars in the Variables section of settings.
Here I’ll set the Access Key and mark as sensitive to mask it in any output
I’ll then have both set
I can then start a new run
which detects the resources and would allow me to apply and create them
However, we have no OPA checks.
In fact, in the “Free” tier, it appears we have no options for compliance at all.
As we showed at the start of the blog, that comes with the US$70/user/month plan
And, as you might expect, I need to add CC Details to “try” it
TF Cloud API driven
First, I created a new Workspace set to be API driven
At the top of my main.tf, I added
terraform {
cloud {
organization = "ThePrincessKing"
workspaces {
name = "myTFexampleCLI"
}
}
}
I’ll need to create (or use if already created) a new token, which I can find under my user area
I’ll add that to my Github Actions secrets
Then to my Env Vars used in my Github workflow
env:
TF_TOKEN_app_terraform_io: $
AWS_ACCESS_KEY_ID: $
AWS_SECRET_ACCESS_KEY: $
And as it passes we can see it plan
Though it failed due to missing env vars. Let’s fix that
I can now see the passing plan triggered Terraform Cloud
Now, because we used plan and not apply, we cannot actually complete this plan from within the TF Cloud Workspace
- name: OpaPassed
run: |
set -x
echo "OPA Check Passed: $OPACHECK"
echo "OPA Score: $OPASCORE"
# Here we could actually tf apply
# Uncommnent the first 9 lines
sed -i '1,9 s/^#//' main.tf
terraform init
terraform plan
if: env.OPACHECK != 'false'
env:
TF_TOKEN_app_terraform_io: $
AWS_ACCESS_KEY_ID: $
AWS_SECRET_ACCESS_KEY: $
Now we can change to apply
But it should plan and wait on us to manually apply because we set the workspace to that mode in “Apply Method”
Now that it invokes as an “apply”, we can see it in my Runs area at the top
And now while we see the Github Actions Workflow is completed, it was stuck on an interactive prompt
However, I can complete, if I choose, on the Terraform Cloud Workspace
My choices are to ignore the interactive prompt since I don’t care
terraform init
`terraform apply -input=false` || true
which will queue a plan and move on.
Or I could use terraform apply -auto-approve to auto approve (which I don’t want to do in this case)
Note on TF local vs Cloud
I glossed over a quick fix I did since TF Cloud does not allow me to save a speculative plan locally to fire through OPA.
I just added a sed to nix the Terraform Cloud block at the top of main.tf
jobs:
build_deploy_test:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v2
- name: tests
run: |
set -x
curl -L -o opa https://github.com/open-policy-agent/opa/releases/download/v0.45.0/opa_linux_amd64
chmod u+x ./opa
terraform --version
./opa version
# Not Terraform Cloud for our Test
# Comment out the first 9 lines
sed -i '1,9 s/^/#/' main.tf
# Set Up Terraform
terraform init
terraform plan --out tfplan.binary
terraform show -json tfplan.binary > tfplan.json
# OPA Policy Checks
OPACHECK=`./opa exec --decision terraform/analysis/authz --bundle policy/ tfplan.json | jq -r ".result[] |.result"`
OPASCORE=`./opa exec --decision terraform/analysis/score --bundle policy/ tfplan.json | jq -r ".result[] |.result"`
echo $OPACHECK
echo $OPASCORE
# set env vars
echo "OPACHECK=$OPACHECK" >> $GITHUB_ENV
echo "OPASCORE=$OPASCORE" >> $GITHUB_ENV
env:
AWS_ACCESS_KEY_ID: $
AWS_SECRET_ACCESS_KEY: $
Then in our next sections, I added it back
- name: OpaPassed
run: |
set -x
echo "OPA Check Passed: $OPACHECK"
echo "OPA Score: $OPASCORE"
# Here we could actually tf apply
# Uncommnent the first 9 lines
sed -i '1,9 s/^#//' main.tf
terraform init
`terraform apply -input=false` || true
if: env.OPACHECK != 'false'
env:
TF_TOKEN_app_terraform_io: $
- name: OpaFailed
run: |
set +x
echo $OPACHECK
echo "OPA Check FAILED Score $OPASCORE"
MINSCORE=`cat policy/terraform.rego | grep 'blast_radius :=' | sed 's/^.* := //'`
echo "OPA Score Limit $MINSCORE"
exit 1
if: env.OPACHECK == 'false'
Testing Opa Check
Now that we know we can trigger the plan to apply in TF cloud, let’s change our main.tf to actually try and do that.
I set the main.tf to have the extra ASGs
terraform {
cloud {
organization = "ThePrincessKing"
workspaces {
name = "myTFexampleCLI"
}
}
}
provider "aws" {
region = "us-west-1"
}
resource "aws_instance" "web" {
instance_type = "t2.micro"
ami = "ami-02ef5566632e904f8"
}
resource "aws_autoscaling_group" "my_asg" {
availability_zones = ["us-west-1a"]
name = "my_asg"
max_size = 70
min_size = 50
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 50
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_launch_configuration" "my_web_config" {
name = "my_web_config"
image_id = "ami-02ef5566632e904f8"
instance_type = "t2.micro"
}
resource "aws_autoscaling_group" "my_asg2" {
availability_zones = ["us-west-1b"]
name = "my_asg"
max_size = 70
min_size = 50
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 50
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_autoscaling_group" "my_asg3" {
availability_zones = ["us-west-1c"]
name = "my_asg"
max_size = 70
min_size = 50
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 50
force_delete = true
launch_configuration = "my_web_config"
}
And then pushed. We can see Github skipped the “OpaPassed” block and gave us our failure message
Terraform Cloud Trial
I decided I would be willing to burn my one time trial of “Teams & Governance” and activate the plan
This then informed me I had 30 days then would revert back
It took me a while to realize you have to log out of the UI and back in again to see the Policies integrations. But once I did that, I could go to Workspaces and see a new “Integrations” section”
I can define my policy (same as my existing rego file)
Note: the query matches the format of data.(Our Package Name).(the block that evaluates at the end).
We then tie that to a Policy Set, which applies a policy to destination workspaces
In our case, any based on the GH repo in question
Now when I run a plan on the workspace, we see an OPA Check occur
My run failed with the error “waiting for results”
Which means I likely need to rewrite my checks…
The problem was by tying to VCS it required a path to be set to policies. If I want to use the policy as defined in TF Cloud, I need to make a non-version controlled policy set
Then back in Policies, I can set that set at the bottom
It ran the rule now, but did not like the format of the authz output
I moved from the basic boolean authz to message rule
this gave a passing score
I tried several times, but in all cases it passed
This was my best attempt at creating the right rego policy code on an evaluation of data.terraform.analysis.rule
package terraform.analysis
import future.keywords.in
import input.plan as tfplan
########################
# Parameters for Policy
########################
# acceptable score for automated authorization
blast_radius := 30
# weights assigned for each operation on each resource-type
weights := {
"aws_autoscaling_group": {"delete": 100, "create": 10, "modify": 1},
"aws_instance": {"delete": 10, "create": 1, "modify": 1}
}
# Consider exactly these resource types in calculations
resource_types := {"aws_autoscaling_group", "aws_instance", "aws_iam", "aws_launch_configuration"}
#########
# Policy
#########
# Authorization holds if score for the plan is acceptable and no changes are made to IAM
default authz := false
authz {
score < blast_radius
not touches_iam
}
# Compute the score for a Terraform plan as the weighted sum of deletions, creations, modifications
score := s {
all := [ x |
some resource_type
crud := weights[resource_type];
del := crud["delete"] * num_deletes[resource_type];
new := crud["create"] * num_creates[resource_type];
mod := crud["modify"] * num_modifies[resource_type];
x := del + new + mod
]
s := sum(all)
}
# Whether there is any change to IAM
touches_iam {
all := resources["aws_iam"]
count(all) > 0
}
####################
# Terraform Library
####################
# list of all resources of a given type
resources[resource_type] := all {
some resource_type
resource_types[resource_type]
all := [name |
name:= tfplan.resource_changes[_]
name.type == resource_type
]
}
# number of creations of resources of a given type
num_creates[resource_type] := num {
some resource_type
resource_types[resource_type]
all := resources[resource_type]
creates := [res | res:= all[_]; res.change.actions[_] == "create"]
num := count(creates)
}
# number of deletions of resources of a given type
num_deletes[resource_type] := num {
some resource_type
resource_types[resource_type]
all := resources[resource_type]
deletions := [res | res:= all[_]; res.change.actions[_] == "delete"]
num := count(deletions)
}
# number of modifications to resources of a given type
num_modifies[resource_type] := num {
some resource_type
resource_types[resource_type]
all := resources[resource_type]
modifies := [res | res:= all[_]; res.change.actions[_] == "update"]
num := count(modifies)
}
rule[msg] {
score < blast_radius
msg := sprintf(
"%d %q violation(s) have been detected.",
[score, rego.metadata.rule().custom.severity]
)
}
That said, if I can work out the rules, it should work as such
Summary
We setup a new Github Repo with some basic AWS Terraform to create hosts and Auto Scaling Groups. We then tested locally creating a rego OPA policy, testing our code for both pass and fail conditions.
We next implemented a Github Workflow that would install OPA and test our code for us. At that point, we could have moved on to applying the terraform. However, had we done that, we would have lost our state file as we invoked it in an ephemeral Github provided agent.
Our next step was to tie it to Terraform Cloud. We first tested a basic version controlled system. We could have set to auto-apply then affixed to a controlled branch, leaving our PR to check OPA policies. This would have worked just fine.
Instead, we added an API based Workspace and tied it to our Github Workflow. This did require a bit of a quick sed hack to remove the Terraform Cloud block at the top (as we needed a local speculative plan to run through OPA).
Lastly, we looked at ways to either ignore the planned invokation or auto-approve in the Github workflow.
I enabled a trial of the Teams and Governance Plan to try and run OPA checks available in Terraform Cloud. I could get the rule to run, but not properly evaluate. After several tries, I found a working rego that would test for tags, so I believe the fault lies in my particular rego code.
I’m not sure I would pursue Terraform Cloud for my organization. At $70/user/month, that really becomes rather expensive at any scale. That is, if I were a small shop, I would likely work the pipeline in Github or AzDO as shown above. If I were a large customer, I wouldn’t want to pay, for instance, US$70k/mo for TF Cloud and instead pursue Terraform Enterprise. Terraform Enterprise might include these features, but I lack a TFE (with priveledges anyhow) to check which leads me to believe that the Integrations section is only available for TFE Administrators making it useless for us plebes working the code.
