Published: Jun 27, 2023 by Isaac Johnson
The first time out with Configure8 I really enjoyed setting it up and integrating with all of my tooling. I want to keep that going with API access which we’ll use with more tooling, cost savings I managed to get right off the bat, and the setting up of Environments which will give us a way to pair services with cloud resources.
In this post we’ll also dig into the REST API, how to get details and later how to action on users to update values using the API. Additionally, we’ll set up a JIRA integration for displaying and interacting with our Issue Tracking suite.
Cost Savings
I’ve noticed my AWS bills climbing. I get a lot of traffic but that can’t really justify the costs.
All I can see from the AWS Cost metrics is some EC2 Instances and Other
But I have nothing!
No Instances and no Volumes
So what is costing me?!
I figured Configure8 might help here.
I let it index my three main accounts and there is a “Type” column that will prove useful
ECS is under EC2 and I wonder if either one of the VPC has a NAT gateway (which is billed as a little micro VM)
I found the defunct ECS cluster in Oregon region and killed it
I then moved on to killing VPCs
Which showed the IGWs
Some pointed to a phantom EKS
I acknowledge US$9 a month isn’t much - it’s a rounding error for most companies. However, out of pocket, that is a Hulu or other streaming subscription - that is a bargaining chip with the Mrs on keeping my Shudder subscription going.
A quick follow-up
Days later… the bill hadn’t gone down. Yet every VPC was gone. There was nothing in the EC2 list; What gives?
Turns out I had forgotten about a second ‘sandbox’ AWS account I made well over a year ago. It also had VPCs and 1 little t2.micro webserver just sitting there
That forgotten account was just pissing away money every month
I bring this up as a reminder to always double check your AWS Organizations page in case you might have created more than one account
Environments
I want to add an environment to a service.
Let’s set a name and view
Now we want to add some resources (Map Resources)
I’ll filter things to just AWS resources and start to select those items associated to this service (the blog)
When done, I have a mapping to resources
Now I can see all the resources when viewing my service and choosing “Diagram”
API Access
Let’s check out the Configure8 API next.
To do that, we need to setup an API key under API keys
I’ll start with an admin key
Which will show it once when created
I also created a MyReadKey with just Read permission as well.
Troubles
Here is where I encountered a bit of trouble.
I could follow the docs here
However, passing in “pageSize” as a GET param didn’t work
curl -X POST --header "Api-Key: c8xxxxxxxxxxxxxxxxxxxxxxxxxx7" https://app.configure8.io/public/v1/catalog/entities?pageSize=5 | jq '.items[] | .providerResourceType'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13969 100 13969 0 0 39572 0 --:--:-- --:--:-- --:--:-- 39572
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"GitHub:Repository:Manifest"
"AWS:ECR:Registry"
"AWS:ECR:Registry"
and try as a post param also failed
$ curl -X POST --header "Api-Key: c********************7" https://app.configure8.io/public/v1/catalog/entities -d '{ "pageNumber":0, "pageSize": 100 }'
{"statusCode":400,"message":["property { \"pageNumber\":0, \"pageSize\": 100 } should not exist"],"error":"Bad Request"}
$ cat test.json
{
"pageNumber": 0,
"pageSize": 20,
"sort": {
"property": "string",
"order": "ASC"
}
}
$ curl -X POST --header "Api-Key: c***************************7" https://app.configure8.io/public/v1/catalog/entities -d @test.json
{"statusCode":400,"message":["property { \"pageNumber\": 0, \"pageSize\": 20, \"sort\": { \"property\": \"string\", \"order\": \"ASC\" }} should not exist"],"error":"Bad Request"}
Getting the ID of those 20 works, of course. So I can pull in, for instance, the GH Action name from one of those manifest entries we saw earlier
$ curl -X GET --header "Api-Key: c*********************7" https://app.configure8.io/public/v1/catalog/entities/75b0d33b-2ab8-45fa-93fa-df738c9276c6 | jq .n
ame
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 858 100 858 0 0 3864 0 --:--:-- --:--:-- --:--:-- 3864
".github/workflows/azure-static-web-apps-purple-bush-00ab46010.yml"
After I reached out to my contacts at Configure8, the error was actually pretty obvious; I had neglected to add --header "Content-Type: application/json" to the call.
A follow-up showed it working just fine:
$ curl -X POST --header "Content-Type: application/json" --header "Api-Key: c8ak_asdfasfasdfasdfsafd_asdfasdf" https://app.configure8.io/public/v1/catalog/entities -d '{ "pageNumber":0, "pageSize": 100 }' | jq '.items[] | .id' | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 91569 100 91534 100 35 49719 19 0:00:01 0:00:01 --:--:-- 49711
100
Getting details
I can get User details with the REST API
$ curl -X GET --header "Api-Key: sadfasfdsadf" https://app.configure8.io/public/v1/users/
{"items":[{"id":"65a64ee3-asdf-asdf-asdf-657868e30195","firstName":"Isaac","lastName":"Johnson","email":"isaac.johnson@gmail.com","displayName":"Isaac Johnson","status":"ACTIVE","role":"ADMIN","orgId":"c2685afa-asdf-asdf-asdf-9341a5906090","acceptedTermsOfServiceAndPrivacyPolicyAt":"2023-06-13T19:51:05.390Z","lastChangedBy":"isaac.johnson@gmail.com","lastLoginAt":"2023-06-26T00:43:57.081Z"}],"totalFound":1,"pageSize":20,"pageNumber":0}builder@LuiGi17:~/Workspaces/jekyll-blog$ curl -X GET --header "Api-Key: sadfasdfasdf" https://app.configure8.io/public/v1/users/ | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 439 100 439 0 0 1699 0 --:--:-- --:--:-- --:--:-- 1694
{
"items": [
{
"id": "65a64ee3-asdf-asdf-asdf-657868e30195",
"firstName": "Isaac",
"lastName": "Johnson",
"email": "isaac.johnson@gmail.com",
"displayName": "Isaac Johnson",
"status": "ACTIVE",
"role": "ADMIN",
"orgId": "c2685afa-asdf-asdf-asdf-9341a5906090",
"acceptedTermsOfServiceAndPrivacyPolicyAt": "2023-06-13T19:51:05.390Z",
"lastChangedBy": "isaac.johnson@gmail.com",
"lastLoginAt": "2023-06-26T00:43:57.081Z"
}
],
"totalFound": 1,
"pageSize": 20,
"pageNumber": 0
}
Powershell
Bash is fun, but what about Powershell? We can use the iwr (Invoke Web Request) to get the same details:
PS C:\Users\isaac> iwr https://app.configure8.io/public/v1/catalog/entities `
>> -Method 'POST' `
>> -ContentType 'application/json; charset=utf-8' `
>> -Headers @{'Content-Type' = 'application/json; charset=utf-8'; 'Api-Key' = 'c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf'}
StatusCode : 200
StatusDescription : OK
Content : {"totalFound":0,"pageSize":20,"pageNumber":0,"items":[]}
RawContent : HTTP/1.1 200 OK
Connection: keep-alive
Vary: Origin
Access-Control-Allow-Credentials: true
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Resource-Policy: same-origin
Origin-Agent-Cluster:...
Forms : {}
Headers : {[Connection, keep-alive], [Vary, Origin], [Access-Control-Allow-Credentials, true],
[Cross-Origin-Opener-Policy, same-origin]...}
Images : {}
InputFields : {}
Links : {}
ParsedHtml : mshtml.HTMLDocumentClass
RawContentLength : 56
If we have entities, then we should get results
PS C:\Users\isaac> iwr https://app.configure8.io/public/v1/catalog/entities `
>> -Method 'POST' `
>> -ContentType 'application/json; charset=utf-8' `
>> -Headers @{'Content-Type' = 'application/json; charset=utf-8'; 'Api-Key' = 'c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf'}
StatusCode : 200
StatusDescription : OK
Content : {"totalFound":1147,"pageSize":20,"pageNumber":0,"items":[{"id":"75b0d33b-asdf-asfd-asfd-df738c9276c
6","name":".github/workflows/azure-static-web-apps-purple-bush-00ab46010.yml","orgId":"c2685afa-ea1
f-...
RawContent : HTTP/1.1 200 OK
Connection: keep-alive
Vary: Origin
Access-Control-Allow-Credentials: true
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Resource-Policy: same-origin
Origin-Agent-Cluster:...
Forms : {}
Headers : {[Connection, keep-alive], [Vary, Origin], [Access-Control-Allow-Credentials, true],
[Cross-Origin-Opener-Policy, same-origin]...}
Images : {}
InputFields : {}
Links : {}
ParsedHtml : mshtml.HTMLDocumentClass
RawContentLength : 13969
We can pull the full content with Select-Object -expand Content
PS C:\Users\isaac> iwr https://app.configure8.io/public/v1/catalog/entities `
>> -Method 'POST' `
>> -ContentType 'application/json; charset=utf-8' `
>> -Headers @{'Content-Type' = 'application/json; charset=utf-8'; 'Api-Key' = 'c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf'} | Select-Object -Expand Content
{"totalFound":1147,"pageSize":20,"pageNumber":0,"items":[{"id":"75b0d33b-asdf-asdf-asdf-df738c9276c6","name":".github/workflows/azure-static-web-apps-purple-bush-00ab46010.yml","orgId":"c2685afa-asfd-asfd-asdf-9341a5906090","description":"","tags":[],"metaTags":[],"links":[],"parents":[],"createdBy":"C8::CORE::SVC::DISCOVERY","lastChangedBy":"C8::CORE::SVC::DISCOVERY","createDateTime":"2023-06-13T20:29:54.199Z","lastChangedDateTime":"2023-06-13T20:29:54.199Z","type":"manifest","repoPath":".github/workf.... snip
So we can use jq to pull out all the GH Repo Paths
$ curl -X POST --header "Content-Type: application/json" --header "Api-Key: c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf" https://app.configure8.io/public/v1/catalog/entities -d '{ "pageNumber": 0, "pageSize": 100 }' | jq '.items[] | .repoPath' | sort -u
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 79302 100 79266 100 36 44389 20 0:00:01 0:00:01 --:--:-- 44402
".github/workflows/azure-static-web-apps-purple-bush-00ab46010.yml"
".github/workflows/ci.yml"
".github/workflows/codeql-analysis.yml"
".github/workflows/generate-site.yml"
".github/workflows/github-actions.yml"
".github/workflows/main.yaml"
".github/workflows/pr-final.yml"
".github/workflows/primary.yml"
".github/workflows/publish.yml"
".github/workflows/testing-ghrunner.yml"
".github/workflows/testing.yml"
".github/workflows/triggerTest.yml"
"/idjohnson/idjohnson.github.io/blob/main/Gemfile"
"/idjohnson/idjohnson.github.io/blob/main/Gemfile.lock"
"Gemfile"
null
I can also see what cloud resources we have
$ curl -X POST --header "Content-Type: application/json" --header "Api-Key: c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf" https://app.configure8.io/public/v1/catalog/entities -d '{ "pageNumber": 0, "pageSize": 100 }' | jq '.items[] | .providerResourceType' | sort -u
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 79302 100 79266 100 36 46438 21 0:00:01 0:00:01 --:--:-- 46456
"AWS:CloudFront:Distribution"
"AWS:ECR:Registry"
"AWS:LAMBDA:Function"
"AWS:SQS:Queue"
"GITHUB:Repository"
"GitHub:Repository:Library"
"GitHub:Repository:Manifest"
null
Posting data
We can add a person to the catalog (but this is not the same as a user on the account).
$ curl -X POST --header "Content-Type: application/json" --header "Api-Key: c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf" https://app.
configure8.io/public/v1/catalog/entities/person --data "@userDetails.json"
{"id":"fe865c9d-2f92-463f-aa07-dabc03457296","name":"Isaac Freshbrewed","orgId":"c2685afa-ea1f-4474-92c6-9341a5906090","description":"My other id","tags":[],"metaTags":[],"links":[],"parents":[],"createdBy":"MyNewAPIKey","lastChangedBy":"MyNewAPIKey","createDateTime":"2023-06-27T02:52:20.254Z","lastChangedDateTime":"2023-06-27T02:52:20.254Z","type":"person","email":"isaac@freshbrewed.science","userId":"","photoUrl":""}
Gathering Costs
Right now, there are some steps one should to setup costing. The key steps are at the bottom of this guide
We’ll grab the CFN yaml here
Description: Export billing to customer bucket
AWSTemplateFormatVersion: "2010-09-09"
Resources:
C8DailyBillingExport:
Type: AWS::CUR::ReportDefinition
DependsOn:
- C8DailyBillingBucket
- C8DailyBillingBucketPolicy
Properties:
AdditionalSchemaElements:
- RESOURCES
Compression: GZIP
Format: textORcsv
ReportName: c8-daily-report
ReportVersioning: CREATE_NEW_REPORT
RefreshClosedReports: False
S3Bucket: !Ref C8DailyBillingBucket
S3Prefix: reports/
S3Region: us-east-1
TimeUnit: DAILY
C8DailyBillingBucket:
Type: 'AWS::S3::Bucket'
DeletionPolicy: Retain
Properties:
BucketName: !Join
- '-'
- - c8-export-cost
- !Ref AWS::AccountId
C8DailyBillingBucketPolicy:
Type: AWS::S3::BucketPolicy
DependsOn:
- C8DailyBillingBucket
Properties:
Bucket:
Ref: C8DailyBillingBucket
PolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: 'billingreports.amazonaws.com'
Action:
- s3:GetBucketAcl
- s3:GetBucketPolicy
Resource:
- Fn::Join: [
'', [
'arn:aws:s3:::',
{
'Ref': 'C8DailyBillingBucket'
}
]
]
- Effect: Allow
Principal:
Service: 'billingreports.amazonaws.com'
Action:
- s3:PutObject
Resource:
- Fn::Join: [
'', [
'arn:aws:s3:::',
{
'Ref': 'C8DailyBillingBucket'
},
'/*'
]
]
I can use the AWS CLI or Console. Here I’ll just use the AWS console to load the CFN
I’ll give the stack a name
It should create it
This will create a daily billing export we can use later
JIRA
If you had your Issue Management done in Atlassian JIRA, you might want to add that to Configure8 as well.
We’ll need a JIRA API key which, for hosted JIRA, we can get from the ID page of our user
We then can give the API key a name
We can now add it to Configure8 as a JIRA Credential
We can now add JIRA to our service. We add the JIRA plugin and pick the credential we created
In the next screen we can select our project
Now I see JIRA activity in the dashboard
We can also see project activity
Adding users
It’s pretty easy to add more users to our Configure8 account. We just need to invite them:
Once invited, we can see we have an invited user
And once their account is setup, we can see the second user
Let’s use the REST API to fetch our users
$ curl -X GET --header "Content-Type: application/json" --header "Api-Key: c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf" https://app.configure8.io/public/v1/users/ | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 826 100 826 0 0 1493 0 --:--:-- --:--:-- --:--:-- 1493
{
"items": [
{
"id": "65a64ee3-asdf-asfd-asdf-657868e30195",
"firstName": "Isaac",
"lastName": "Johnson",
"email": "isaac.johnson@gmail.com",
"displayName": "Isaac Johnson",
"status": "ACTIVE",
"role": "ADMIN",
"orgId": "c2685afa-ea1f-4474-92c6-9341a5906090",
"acceptedTermsOfServiceAndPrivacyPolicyAt": "2023-06-13T19:51:05.390Z",
"lastChangedBy": "isaac.johnson@gmail.com",
"lastLoginAt": "2023-06-27T02:55:12.458Z"
},
{
"id": "ab8c3e20-asdf-asdf-asdf-eede5e4aab96",
"firstName": "Isaac",
"lastName": "Johnson",
"email": "isaac@freshbrewed.science",
"displayName": "Isaac Johnson",
"status": "ACTIVE",
"role": "USER",
"orgId": "c2685afa-ea1f-4474-92c6-9341a5906090",
"acceptedTermsOfServiceAndPrivacyPolicyAt": "2023-06-27T03:12:41.367Z",
"lastChangedBy": "isaac@freshbrewed.science",
"lastLoginAt": "2023-06-27T03:12:41.772Z"
}
],
"totalFound": 2,
"pageSize": 20,
"pageNumber": 0
}
I can now use the REST API to PATCH the user to Admin
$ curl -X PATCH --header "Content-Type: application/json" --header "Api-Key: c8ak_asdfasdfasdsadfasdfsadsadf_asdfasdf" https://app.configure8.io/public/v1/users/ab8c3e20-asfd-asdf-asdf-eede5e4aab96 -d '{ "role": "ADMIN" }'
{"id":"ab8c3e20-6c8c-4d2a-af0b-eede5e4aab96","firstName":"Isaac","lastName":"Johnson","email":"isaac@freshbrewed.science","displayName":"Isaac Johnson","status":"ACTIVE","role":"ADMIN","photoUrl":null,"orgId":"c2685afa-ea1f-4474-92c6-9341a5906090","acceptedTermsOfServiceAndPrivacyPolicyAt":"2023-06-27T03:12:41.367Z","lastChangedBy":"MyNewAPIKey","lastLoginAt":"2023-06-27T03:12:41.772Z"}
Which we can see reflected in the UI
Summary
We covered some initial cost savings which I just loved finding, especially when it’s me paying the bill. We then explored the REST API before adding JIRA Integration. Lastly, we added an additional user and used the REST API to change the users role.
Also, we did some updates to set up costing details as documented that could be used for Cost details in OpenCost.
The fact is, I’ve had some fantastic conversations with Configure8 and there are some features rolling out in the future that I’m chomping at the bit to dig into. I absolutely will have a follow-up piece later on some of those, but for now, it’s a fantastic suite that I want to explore further with a more complicated service.
