Published: Sep 21, 2023 by Isaac Johnson

I’ve spoken of budgets and cost monitors before such as ConfigU, Kubecost, OpenCost and Cast.ai. In almost all cases, these are designed to look at your overall cloud spend and highlight overspend patterns, cost optimizations and more.

The problem is my dumb butt went and did a demo in a room for a company event. I used my own terraform historical repo to show off some Packer/TF in a pipeline work and forgot about it. The first month, the bill co-mingled with an o365 renewal so I didn’t catch it. But the next month, while chasing down some Home Depot fraud, I saw my Azure spend was nearly $200 again. Why? I left the damn VM running from the packer demo since August.

Now maybe in a company setting we would go “oops” and just delete. But $200 just plain wasted out of my pocket is $200 I won’t be spending on ice fishing gear at Dicks or Cabelas. I just hate that.

I have to go through each cloud this week anyhow, regardless, as some jackhole got hold of my CC and charged up a fortune at Home Depot. It’s marked as fraud but now I’ll need to correct billing across my clouds.

The thing I now realize is email sucks. I mean, it sucks to let me know I better do something. Discord works. Pagerduty works.

Google Cloud (GCP)

I’ve actually had a PagerDuty alert setup for Google Cloud since May

What is more interesting is that it appears I demo’ed that (not in the blog) and never saved the alert.

To date, in my billing account, I only have alerts on Google Map API usage (that goes back to Pebble Watch apps)

The first thing I’m going to do is change this to Cloud Budget, I don’t need alert paths unique to each cloud.

Next, I’ll create a budget in GCP

I definitely want to cover all projects and all services on this one

I feel $20 is a good alert threshold for me.

I’m going to want to link to an email which requires picking another project to house that in

I can see it’s already setup

However, if you need to add it, you can by clicking “Manage Channels”. I’m having a bit of Deja Vu on this, but I recall initially trying to use the PD Service, but the PD Service, as far as a notification channel, did not work with budgets and alerts, which really just wanted email.

You can get the email for your Pager Duty service in Integrations, Email:

I now have enough to go on and can click finish

I can also now see it listed in my GCP budgets

Azure

Let’s go to budgets in Azure

Here, I’ll set up “+ Add” and set a start and end date, as well as an amount. This is embarassing to show the overrun, but I will hide nothing from you, dear reader.

I’ll set it to be 100% of actual and 200% of forecasted. This is because I often spin up a demo example and it will forecast some wild spend initially. But I generally delete it within a day.

In this case, I’ll also send alerts out to my emails.

I can now see it’s set

This may not work, I want to note that. I have a $400 bill coming due by the end of the month and my loving wife will most definatley thump me.

However, at least for last month, I can request a refund. You don’t see this on AWS, I can promise you that.

I almost immedately got an alert and a PagerDuty call on the Azure Budget

I can also see the email alert

AWS

In billing, we can go to Budgets to use a pre-baked template for Budgets

My spend (while growing), is fairely consistant with AWS. I once had a huge spike becuase some legacy AD that was a free for a year to backend VMs all of a sudden kicked in and I had a huge bill. But generally AWS is not my issue.

I’ll create a budget of $25 for AWS and set it to go to all my emails as well

I can now see it listed

Summary

This is not a long post, admittedly, but it is a very important one. One needs to ensure one is alerts on spend by whatever means actually reaches them. In my case, I thought periodically checking would be sufficient, but it wasn’t. The fact that my August bill went unnoticed stinks. I had a work trip and a lot of charges and they just co-mingled.

I will be seeking a refund after September’s bill and corrected Credit Cards have been applied. I might circle back and edit this post at that date to say whether Microsoft was cool or not. I am hopeful.

If you want to ask, why not use your work Azure account - we have all sorts of policies and limits on work accounts, even VSE - which means creating new VMs from base templates by-and-large are broken because you have to use an “approved company image” which doesn’t work well for a demo to make images. I’m not complaining. My dumbass was the one who used his own Azure DevOps account to demo, I cannot blame others. But dammit, such a waste.