Published: Jul 19, 2022 by Isaac Johnson

Cloud Custodian is an open-source tool for managing and enforcing policy in the cloud. It’s been around for a while. Releases in Github go back to April 2016. Today c7n can be used for AWS, Azure and GCP (the later two via modules).

There are a few modules, such as Org management and a Mailer that can tie to Slack, SES and Datadog. There is also a PolicyStream module that can be used for PR automation.

Setup

We can check the install page for Cloud Custodian.

Since it’s ideally installed via Python (per Developer Install Docs).

Because in our last blog entry we spoke on how to install AWX, I figured it would be a nice follow-up to use the AWX to do some real work. We’ll create a repo, playbook then add to AWX and use it to install Cloud Custodian.

With Ansible Tower/AWX

I’ll first create a repo for the playbook

then a quick playbook

$ cat cloudcustodian.yaml
---
- name: Install Cloud Custodian
  hosts: all

  tasks:
  - name: Instally Python3
    ansible.builtin.shell: |
      apt-get install python3 python3-venv python3-pip
    become: true
    args:
      chdir: /tmp

  - name: Check Version
    ansible.builtin.shell: |
      python3 -m pip --version
    args:
      chdir: /tmp

  - name: Instally Custodian locally
    ansible.builtin.shell: |
      umask 0002
      python3 -m venv custodian
      source custodian/bin/activate
      pip install c7n  
    become: true
    args:
      chdir: /tmp

  - name: Pull Docker Image
    ansible.builtin.shell: |
      docker pull cloudcustodian/c7n  
    become: true
    args:
      chdir: /tmp

I’ll login to AWX and add a new Project with the repo:

And as is my habit, sync it right away

Going to Templates, I’ll add a new Template

and fill in the details for this project and playbook

Lastly, we just click the Rocket icon to launch

Quick tip: if you get an error about missing sudo password on commands

that usually means your ansible user (in my case ‘builder’) lacks nopassword in the sudoers file.

On the host you can do sudo visudo and change the entry:

builder@builder-HP-EliteBook-745-G5:~$ sudo cat /etc/sudoers | grep builder
builder ALL=(ALL:ALL) NOPASSWD: ALL

Also, if you need to install docker on Ubuntu, you can follow these quick steps

And don’t forget to add your user (in my case builder but for you likely ansible) to the docker group

$ sudo usermod -aG docker builder

When done, we should see AWX installed the Playbook successfully

I can now login to the host to verify both that the docker image is there as well as the binary

$ ssh builder@192.168.1.32
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.13.0-52-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

132 updates can be applied immediately.
8 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable

Your Hardware Enablement Stack (HWE) is supported until April 2025.
*** System restart required ***
Last login: Wed Jul 13 06:52:53 2022 from 192.168.1.214
builder@builder-HP-EliteBook-745-G5:~$ docker images
REPOSITORY           TAG       IMAGE ID       CREATED        SIZE
cloudcustodian/c7n   latest    610f126c34eb   3 weeks ago    895MB
hello-world          latest    feb5d9fea6a5   9 months ago   13.3kB
builder@builder-HP-EliteBook-745-G5:~$ custodian version
0.9.17

Using Custodian

Our first example will be to auto tag resources in Azure.

I’ll be using my Azure account.

The first part is easy, getting the Sub and Tenant

builder@builder-HP-EliteBook-745-G5:~$ az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code E2W38WRF7 to authenticate.
[
  {
    "cloudName": "AzureCloud",
    "id": "d955c0ba-1234-1234-1234-8fed74cbb22d",
    "isDefault": true,
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "28c575f6-1234-1234-1234-7e6d1ba0eb4a",
    "user": {
      "name": "isaac.johnson@gmail.com",
      "type": "user"
    }
  }
]
builder@builder-HP-EliteBook-745-G5:~$ export AZURE_TENANT_ID=28c575f6-1234-1234-1234-7e6d1ba0eb4a
builder@builder-HP-EliteBook-745-G5:~$ export AZURE_SUBSCRIPTION_ID=d955c0ba-1234-1234-1234-8fed74cbb22d

The steps to create an SP have changed a bit in the last year. It used to be just a quick az ad sp create-for-rbac but now they make you check the domain.

You can see what your “domain” is in the Azure Portal under AAD:

$ az ad sp create-for-rbac -n "CloudCustodian" --role Contributor --name cloudcustodian -n cloudcustodian.isaacjohnsongmail.onmicrosoft.com

The password (password) and Client ID (appId) will be shown in the output of the above. We can use that to set the next two variables

builder@builder-HP-EliteBook-745-G5:~$ export AZURE_CLIENT_ID=b**********************************************7
builder@builder-HP-EliteBook-745-G5:~$ export AZURE_CLIENT_SECRET=f******************************************a

My First example uses the VMSS I keep on standby for AzDO

As we can see, they have no meaningful tags

The following policy should tag that VMSS with a Hello World tag

$ cat vmss.policy.yml
policies:
    - name: my-first-policy
      description: |
        Adds a tag to a virtual machine scale sets
      resource: azure.vmss
      filters:
        - type: value
          key: name
          value: azdovmssexample
      actions:
       - type: tag
         tag: Hello
         value: World

If we run this now, we’ll see we are missing the Azure Support (c7n_azure)

builder@builder-HP-EliteBook-745-G5:~$ custodian run --output-dir=. vmss.policy.yml
Traceback (most recent call last):
  File "/usr/local/bin/custodian", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/c7n/cli.py", line 363, in main
    command(config)
  File "/usr/local/lib/python3.8/dist-packages/c7n/commands.py", line 51, in _load_policies
    collection = policy_load(options, fp, validate=validate, vars=vars)
  File "/usr/local/lib/python3.8/dist-packages/c7n/policy.py", line 40, in load
    load_resources(rtypes)
  File "/usr/local/lib/python3.8/dist-packages/c7n/resources/__init__.py", line 20, in load_resources
    load_providers(set(pmap))
  File "/usr/local/lib/python3.8/dist-packages/c7n/resources/__init__.py", line 77, in load_providers
    from c7n_azure.entry import initialize_azure
ModuleNotFoundError: No module named 'c7n_azure'

We could just pip install from the command line the missing module. However, I’ll instead update the playbook

a quick run of the updated template

and now it should work. However, there is a version incompatibility:

$ custodian run --output-dir=. vmss.policy.yml
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/azure/common/credentials.py", line 107, in <module>
    from msrest.authentication import (
  File "/usr/local/lib/python3.8/dist-packages/msrest/__init__.py", line 28, in <module>
    from .configuration import Configuration
  File "/usr/local/lib/python3.8/dist-packages/msrest/configuration.py", line 38, in <module>
    from .universal_http.requests import (
  File "/usr/local/lib/python3.8/dist-packages/msrest/universal_http/__init__.py", line 53, in <module>
    from ..exceptions import ClientRequestError, raise_with_traceback
  File "/usr/local/lib/python3.8/dist-packages/msrest/exceptions.py", line 31, in <module>
    from azure.core.exceptions import SerializationError, DeserializationError
ImportError: cannot import name 'SerializationError' from 'azure.core.exceptions' (/usr/lib/python3/dist-packages/azure/core/exceptions.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/custodian", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/c7n/cli.py", line 363, in main
    command(config)
  File "/usr/local/lib/python3.8/dist-packages/c7n/commands.py", line 51, in _load_policies
    collection = policy_load(options, fp, validate=validate, vars=vars)
  File "/usr/local/lib/python3.8/dist-packages/c7n/policy.py", line 40, in load
    load_resources(rtypes)
  File "/usr/local/lib/python3.8/dist-packages/c7n/resources/__init__.py", line 20, in load_resources
    load_providers(set(pmap))
  File "/usr/local/lib/python3.8/dist-packages/c7n/resources/__init__.py", line 77, in load_providers
    from c7n_azure.entry import initialize_azure
  File "/usr/local/lib/python3.8/dist-packages/c7n_azure/entry.py", line 5, in <module>
    from c7n_azure.provider import Azure  # NOQA
  File "/usr/local/lib/python3.8/dist-packages/c7n_azure/provider.py", line 9, in <module>
    from .session import Session
  File "/usr/local/lib/python3.8/dist-packages/c7n_azure/session.py", line 12, in <module>
    from azure.common.credentials import BasicTokenAuthentication
  File "/usr/lib/python3/dist-packages/azure/common/credentials.py", line 113, in <module>
    raise ImportError("You need to install 'msrest' to use this feature")
ImportError: You need to install 'msrest' to use this feature

basically:

ERROR: c7n-azure 0.7.16 has requirement azure-core==1.24.1; python_version >= "3.6", but you'll have azure-core 1.24.2 which is incompatible.
Installing collected packages: azure-core
  Attempting uninstall: azure-core
    Found existing installation: azure-core 1.24.1
    Uninstalling azure-core-1.24.1:
      Successfully uninstalled azure-core-1.24.1
Successfully installed azure-core-1.24.2

However, this is why we brought down the docker image as well.

Note: I later tried on my laptop without issue

(custodian) builder@DESKTOP-72D2D9T:~/Workspaces/c7n$ vi vmss.policy.yml
(custodian) builder@DESKTOP-72D2D9T:~/Workspaces/c7n$ custodian run --output-dir=. vmss.policy.yml
2022-07-13 21:36:50,719: custodian.azure.session:INFO Authenticated [Principal | d955c0ba-13dc-44cf-a29a-8fed74cbb22d]
2022-07-13 21:36:52,088: custodian.policy:INFO policy:my-first-policy resource:azure.vmss region:AzureCloud count:1 time:1.37
2022-07-13 21:36:52,089: custodian.azure.tagging.Tag:INFO Action 'tag' modified 'azdovmssexample' in resource group 'AZDOVMSSEXAMPLE'.
2022-07-13 21:36:52,090: custodian.policy:INFO policy:my-first-policy action:tag resources:1 execution_time:0.00

Custodian in Docker

We can now run the VMSS policy using the docker image we already have

$ docker run -it -v $(pwd)/output:/home/custodian/output -v $(pwd)/vmss.policy.yml:/home/custodian/policy.yml --env-file <(env | grep "^AWS\|^AZURE\|^GOOGLE") cloudcustodian/c7n run -v -s /home/custodian/output /home/custodian/policy.yml
2022-07-13 12:37:05,659: custodian.commands:DEBUG Loaded file /home/custodian/policy.yml. Contains 1 policies
2022-07-13 12:37:05,659: custodian.azure.session:INFO Authenticated [Principal | d955c0ba-13dc-44cf-a29a-8fed74cbb22d]
2022-07-13 12:37:05,660: custodian.cache:DEBUG Using in-memory cache
2022-07-13 12:37:05,661: custodian.output:DEBUG Storing output with <LogFile file:///home/custodian/output/my-first-policy/custodian-run.log>
2022-07-13 12:37:05,661: custodian.policy:DEBUG Running policy:my-first-policy resource:azure.vmss region:AzureCloud c7n:0.9.17
2022-07-13 12:37:05,868: msal.authority:DEBUG openid_config = {'token_endpoint': 'https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
2022-07-13 12:37:05,869: msal.application:DEBUG Region to be used: None
2022-07-13 12:37:05,869: msal.application:DEBUG Found 0 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': None, 'family_id': '1'}
2022-07-13 12:37:05,869: msal.application:DEBUG Found 0 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': None, 'client_id': 'b8e8d285-0680-4055-b516-96061d2abb67'}
2022-07-13 12:37:05,938: msal.telemetry:DEBUG Generate or reuse correlation_id: df7f0c94-7ee3-4e81-95b9-798aa9de4bf4
2022-07-13 12:37:06,039: msal.token_cache:DEBUG event={
    "client_id": "b8e8d285-0680-4055-b516-96061d2abb67",
    "data": {
        "claims": "{\"access_token\": {\"xms_cc\": {\"values\": [\"CP1\"]}}}",
        "scope": [
            "https://management.core.windows.net/.default"
        ]
    },
    "environment": "login.microsoftonline.com",
    "grant_type": "client_credentials",
    "params": null,
    "response": {
        "access_token": "********",
        "expires_in": 3599,
        "ext_expires_in": 3599,
        "token_type": "Bearer"
    },
    "scope": [
        "https://management.core.windows.net/.default"
    ],
    "token_endpoint": "https://login.microsoftonline.com/28c575f6-ade1-4838-8e7c-7e6d1ba0eb4a/oauth2/v2.0/token"
}
2022-07-13 12:37:06,398: custodian.azure.utils.ServiceClient.send:DEBUG 200
2022-07-13 12:37:06,398: custodian.azure.utils.ServiceClient.send:DEBUG x-ms-ratelimit-remaining-resource:Microsoft.Compute/HighCostGetVMScaleSet3Min;179,Microsoft.Compute/HighCostGetVMScaleSet30Min;887
2022-07-13 12:37:06,398: custodian.azure.utils.ServiceClient.send:DEBUG x-ms-ratelimit-remaining-subscription-reads:11999
2022-07-13 12:37:06,408: custodian.resources.vmscaleset:DEBUG Filtered from 2 to 1 vmscaleset
2022-07-13 12:37:06,408: custodian.policy:INFO policy:my-first-policy resource:azure.vmss region:AzureCloud count:1 time:0.75
2022-07-13 12:37:06,412: msal.application:DEBUG Cache hit an AT
2022-07-13 12:37:06,412: msal.telemetry:DEBUG Generate or reuse correlation_id: b5b54b83-a5a0-48fd-860a-80ebab185f09
2022-07-13 12:37:06,864: custodian.azure.utils.ServiceClient.send:DEBUG 200
2022-07-13 12:37:06,864: custodian.azure.utils.ServiceClient.send:DEBUG x-ms-ratelimit-remaining-subscription-reads:11999
2022-07-13 12:37:09,758: custodian.azure.utils.ServiceClient.send:DEBUG 200
2022-07-13 12:37:09,758: custodian.azure.utils.ServiceClient.send:DEBUG x-ms-ratelimit-remaining-resource:Microsoft.Compute/VMScaleSetActions3Min;239,Microsoft.Compute/VMScaleSetActions30Min;1199,Microsoft.Compute/VmssQueuedVMOperations;0
2022-07-13 12:37:09,758: custodian.azure.utils.ServiceClient.send:DEBUG x-ms-ratelimit-remaining-subscription-writes:1199
2022-07-13 12:37:09,761: custodian.azure.tagging.Tag:INFO Action 'tag' modified 'azdovmssexample' in resource group 'AZDOVMSSEXAMPLE'. {'Hello': 'World'}
2022-07-13 12:37:09,762: custodian.policy:INFO policy:my-first-policy action:tag resources:1 execution_time:3.35
2022-07-13 12:37:09,763: custodian.output:DEBUG metric:ResourceCount Count:1 policy:my-first-policy restype:azure.vmss scope:policy

If I log back into the Azure Portal, we’ll now see the VMSS was updated

But only that one named VMSS

Destructive Operations

Tagging is useful for categorizing resources automatically. However, we can enforce rules by stopping VMs or deleting resources altogether.

First, for a test, let me tag the other VMSS I use

One issue I encountered was that one of my VMSSes had a Spot Instance limit causing the Update Command to fail

2022-07-13 12:45:37,086: custodian.azure.tagging.Tag:ERROR tag failed for 'myvmsslarge'.
Traceback (most recent call last):
  File "/src/tools/c7n_azure/c7n_azure/actions/base.py", line 69, in _process_resources
    result = self._process_resource(r)
  File "/src/tools/c7n_azure/c7n_azure/actions/tagging.py", line 72, in _process_resource
    return TagHelper.add_tags(self, resource, new_tags)
  File "/src/tools/c7n_azure/c7n_azure/tags.py", line 80, in add_tags
    TagHelper.update_resource_tags(tag_action, resource, tags)
  File "/src/tools/c7n_azure/c7n_azure/tags.py", line 39, in update_resource_tags
    client.resources.begin_update_by_id(resource['id'], api_version, tags_patch)
  File "/usr/local/lib/python3.8/site-packages/azure/mgmt/resource/resources/v2020_10_01/operations/_resources_operations.py", line 1461, in begin_update_by_id
    raw_result = self._update_by_id_initial(
  File "/usr/local/lib/python3.8/site-packages/azure/mgmt/resource/resources/v2020_10_01/operations/_resources_operations.py", line 1412, in _update_by_id_initial
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/usr/local/lib/python3.8/site-packages/azure/core/exceptions.py", line 107, in map_error
    raise error
azure.core.exceptions.ResourceExistsError: (OperationNotAllowed) Unable to perform operation 'Update Virtual Machine Scale Set' since the provided max price '0.1 USD' is lower than the current spot price '0.1536 USD' for Azure Spot VM size 'Standard_D8s_v3'. For more information, see http://aka.ms/AzureSpot/errormessages.
Code: OperationNotAllowed
Message: Unable to perform operation 'Update Virtual Machine Scale Set' since the provided max price '0.1 USD' is lower than the current spot price '0.1536 USD' for Azure Spot VM size 'Standard_D8s_v3'. For more information, see http://aka.ms/AzureSpot/errormessages.

I could see the same error if I used the Portal and tried to Tag it

This is the first time I can ever recall seeing inflation reflected in my cloud configurations in such a direct way

I updated the value by a nickel.

When I verified that indeed, the filter on tag wouldn’t match my known good VMSSes

...
2022-07-13 12:51:12,220: custodian.azure.utils.ServiceClient.send:DEBUG 200
2022-07-13 12:51:12,220: custodian.azure.utils.ServiceClient.send:DEBUG x-ms-ratelimit-remaining-resource:Microsoft.Compute/HighCostGetVMScaleSet3Min;173,Microsoft.Compute/HighCostGetVMScaleSet30Min;866
2022-07-13 12:51:12,221: custodian.azure.utils.ServiceClient.send:DEBUG x-ms-ratelimit-remaining-subscription-reads:11999
2022-07-13 12:51:12,231: custodian.resources.vmscaleset:DEBUG Filtered from 2 to 0 vmscaleset
2022-07-13 12:51:12,231: custodian.policy:INFO policy:my-second-policy resource:azure.vmss region:AzureCloud count:0 time:0.62
2022-07-13 12:51:12,232: custodian.output:DEBUG metric:ResourceCount Count:0 policy:my-second-policy restype:azure.vmss scope:policy
....

I then changed the policy to delete

$ cat vmss.policy2.yml
policies:
    - name: my-second-policy
      description: |
        removes untagged VMSSes
      resource: azure.vmss
      filters:
        - tag:Hello: absent
      actions:
       - type: delete

We can now watch and see it in action:

Automating with AzDO

Let’s first create a new Repo to store the pipeline

And give it a reaonsable name

Next, I’ll “Set up build”

And choose a starter pipeline

We’ll change the contents to look like this:

trigger:
- main

pool:
  vmImage: ubuntu-latest


steps:
- script: |
    cat > $(Build.ArtifactStagingDirectory)/vmss.tag.policy.yml << "EOF"
    policies:
        - name: my-first-policy
          description: |
            Adds a tag to a virtual machine scale sets
          resource: azure.vmss
          filters:
            - type: value
              key: name
              value: azdovmssexample
          actions:
          - type: tag
            tag: Hello2
            value: World4
    EOF
    cat $(Build.ArtifactStagingDirectory)/vmss.tag.policy.yml
  displayName: 'Inline VMSS Tag Policy'

- script: |
    mkdir $(Build.ArtifactStagingDirectory)/output || true
    chmod 777 $(Build.ArtifactStagingDirectory)/output
  displayName: 'Create output dir'

- script: |
    docker run -v $(Build.ArtifactStagingDirectory)/output:/home/custodian/output \
       -v $(Build.ArtifactStagingDirectory)/vmss.tag.policy.yml:/home/custodian/policy.yml \
       -e AZURE_CLIENT_ID='$(myazureclientid)' \
       -e AZURE_CLIENT_SECRET='$(myazureclientsecret)' \
       -e AZURE_SUBSCRIPTION_ID='$(myazuresubscription)' \
       -e AZURE_TENANT_ID='$(myazuretenantid)' \
       cloudcustodian/c7n run \
        -v -s /home/custodian/output /home/custodian/policy.yml\
  displayName: 'Run Cloud Custodian'

- script: |
    find $(Build.ArtifactStagingDirectory)/output -type f -exec cat {} \; -print
  displayName: 'Show output files'
  condition: always()

We’ll need to set the secrets as well. You can use just build secret variables in this case

When done, you should have set these four

Then save and run

And if all goes well, you should see output in the build log

And a new tag applied

scheduling

Obviously, running this periodically would be ideal so things are automatically updated.

We can set a schedule two ways in AzDO. The first being via Triggers in the UI

and adding a Schedule

Using this approach we have the advantage of not having to update code (the azure-pipelines.yml) to change schedules, but at the consequence of not seeing who and when the schedule changes.

Instead, we’ll go ahead and use the YAML approach. Here we can set a schedule for daily at midnight

trigger:
- main

pool:
  vmImage: ubuntu-latest

schedules:
- cron: "0 0 * * *"
  displayName: Daily midnight build
  branches:
    include:
    - main
  always: true

steps:
- script: |
    cat > $(Build.ArtifactStagingDirectory)/vmss.tag.policy.yml << "EOF"
    policies:
        - name: my-first-policy
          description: |
            Adds a tag to a virtual machine scale sets
          resource: azure.vmss
          filters:
            - type: value
              key: name
              value: azdovmssexample
          actions:
          - type: tag
            tag: Hello2
            value: World4
    EOF
    cat $(Build.ArtifactStagingDirectory)/vmss.tag.policy.yml
  displayName: 'Inline VMSS Tag Policy'

- script: |
    mkdir $(Build.ArtifactStagingDirectory)/output || true
    chmod 777 $(Build.ArtifactStagingDirectory)/output
  displayName: 'Create output dir'

- script: |
    docker run -v $(Build.ArtifactStagingDirectory)/output:/home/custodian/output \
       -v $(Build.ArtifactStagingDirectory)/vmss.tag.policy.yml:/home/custodian/policy.yml \
       -e AZURE_CLIENT_ID='$(myazureclientid)' \
       -e AZURE_CLIENT_SECRET='$(myazureclientsecret)' \
       -e AZURE_SUBSCRIPTION_ID='$(myazuresubscription)' \
       -e AZURE_TENANT_ID='$(myazuretenantid)' \
       cloudcustodian/c7n run \
        -v -s /home/custodian/output /home/custodian/policy.yml\
  displayName: 'Run Cloud Custodian'

- script: |
    find $(Build.ArtifactStagingDirectory)/output -type f -exec cat {} \; -print
  displayName: 'Show output files'
  condition: always()

Note: Azure Pipelines timezone isn’t always your local timezone. It’s usually best to double check what Azure DevOps thinks “midnight” is. At times I’ve seen this really represent UTC. I believe it leverages the Organization timezone:

Summary

We experimented with C7n on the command line and in Docker. We showed how we could identify resources by tag, tag items that are missing required tags and lastly remove items that are untagged in Azure.

We could have automated by running through scheduled Ansible Jobs, scheduled AzDO Pipeline, Jenkins Job or a myriad of other ways to schedule automated workloads.

To demonstrate an example of that, we created an Azure Repo and Pipeline then automated the running of the docker file with an inline policy that tagged a specific VMSS. Lastly, we discussed schedules and how to automatically run that pipeline on a schedule to enforce policy.

Cloud Custodian is a nice addition to our toolset and partnered with tools like Kubecost and Ternary can track cloud costs and mitigate unapproved spend.