Published: Dec 5, 2022 by Isaac Johnson
We’ve covered using Datadog and other tools to monitor and display metrics. Today we’ll wrap our series on Aiven.io by looking at setting up a hosted Grafana instance and tying a custom DNS name to it.
We’ll setup a hosted InfluxDB to collect metrics from Kafka and Redis, then show how to use native Aiven integrations to link InfluxDB to Grafana.
Lastly, we’ll dig in to Grafana queries and dashboards and wrap by setting up Email, Teams and Discord integrations (including setting up a new Discord server)
We first create the service from the Services page. It’s one of the least expensive hosted services from Aiven.io starting (presently) as US$35/mo in GCP
Once running, we can login with the credentials in the service page
And use on the Grafana URL
Of the Options to gather metrics in Aiven, InfluxDB is one of the less expensive at US$60/mo for a startup size instance in US Central
I’ll create and wait on the new service to get past ‘Rebuilding’
Custom DNS for Grafana
creating a custom domain name
Hmm.. that domain has issues; let’s use our primary
I’ll create a CNAME record
$ cat r53-aiven-grafana.json
"Comment": "CREATE awx fb.s CNAME record ",
$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-aiven-grafana.json
"Comment": "CREATE awx fb.s CNAME record "
Then apply it in Aiven.io
In a bit, we’ll see it has a proper cert and responds
Then we can log in
Adding Data Sources
Before we can graph anything, we need to add a data source
Besides setting it up in Grafana, we can setup Data sources from Aiven via the Integrations section
With InfluxDB now running, let’s add it to Kafka Metrics
In Grafana, we’ll then add a new Integration
We’ll want to “Receive Data”
At which point we can choose the InfluxDB
Within Grafana we now see InfluxDB listed in Data Sources
To use it, we can use Influx QL
SELECT * FROM "_internal".."database" LIMIT 10
Or us the Explore query creator. For instance, query the CPU usage
I can then add it to a new Dashboard
and save it with a name
Alerting with Grafana
To send any kind of alert, we first need to define a “contact point”
We can get there from Alerting/Contact Points, then selecting “New contact point”
We can, for instance, define an email address to which we will send alerts.
We can send a test mail
However, this just informed me SMTP was not setup out of the box
In an on-prem install, we would likely change the .ini file used by Grafana to set the
For Aiven.io, we will do it in the Advanced Configuration area
I’ll set the settings for GMAIL
I found using a real 2FA password won’t work.
You need to follow these steps to generate an “App” password we can use for STMP
which I can now see in the list (and revoke later)
Which brings me to a minor issue I have with Aiven - I don’t like seeing my passwords left visible plain text
and we can see it worked
Creating Grafana alerts
Before we being, we’ll create a folder
Then we can create a query
we can set the evaluation behavior
then set our notification label
We can now create a Notification Policy that uses it
and now we can see it has been saved
We can create other contact points such as Teams and Pagerduty
Quick Note on Teams: This has been plaguing me for a while - I can never sign in. I’ve been stymied by this over and over for a long while. I finally figured it out today. There are two (insert profane explicative) Teams apps. One with Blue and one with a White “T” tile.
The White tile, which Windows would always suggest when searching for teams is not going to let you sign into Orgs. The blue tile will. I think only recently did they add “(work or school)” to the App name.
Real Alerts and Metrics
Several days later I actually got a real alert
Had I set a proper description, I might have put in a link to wiki pages, or who to call, perhaps a runbook on remediation.
However as we can see above, 5 minutes later it was resolved.
While the error message would lead me to believe it was from Kafka, the Spike (since I aggregated all the metrics) was actually from Grafana itself (times are in Zulu/GMT)
I’m only sending out Kafka Metrics to Datadog and Cloudwatch
But we can see that in both cases, our Kafka cluster was fine
(The drop at the end is because I just stopped the cluster in the last 5 minutes)
I can see similar (including the drop off of delivered metrics) in Cloudwatch
Setting up Teams
We can go to the Connectors for the Channel
Then we’ll configure an Incoming Webhook
I’ll use an icon (which I’ll leave for you to grab here):
And we can fill out the Webhook details
We’ll then want to copy that URL
Back in Grafana we can setup a Teams Notification Contact Type
And then test it
And see the results
More and more, I’m seeing my colleagues looking to Discord as the ‘new’ Slack. Let’s setup discord (which unlike teams doesn’t require me to endlessly sign out and in again to swap orgs).
I’ll go ahead and “Add a server”
Then I’ll choose to “Create My Own”
I’ll likely just say for “Me and my friends” on the next prompt. I don’t think it matters.
Then we can give it name and icon
We can right-click the server icon to get to Integrations so we can add the webhook
which brings us to the Integrations page
Just like in Teams, we can the Grafana icon
I can “Copy Webhook URL” which puts it in the clipboard. Something like
We can now use that as a Contact Point in Grafana
As before, we can test the alert
and see it fires in our ‘#general’ channel
Today we set up hosted instances of Grafana and Influx DB. We ingested metrics to InfluxDB via native Aiven.io integrations, then turned around and tied the InfluxDB to Grafana. In Grafana we verified we could see these metrics then worked through adding Contact Points, Labels and Alert Policies. We tested sending out alerts to Email, Teams and Discord. Later we received a proper alert from Grafana and looked at Metrics gathered in Aiven, Datadog and AWS to identify the time and service.
Arguably, using InfluxDB and Grafana is a pretty simple setup for those looking for a straightforward hosted APM system - we have InfluxDB and Grafana, together for just under $100/mo with our only limitation on server space. We can easily tie these to Aiven.io services or external services. We can even use the private VPC option if we wish to ingest metrics without traversing the public internet using the Aiven.io VPC private endpoint option
Looking at all Aiven.io offers, one can see it isn’t a replacement for our clouds, but a natural complimentary augmentation - creating turnkey supported hosting solutions in an easy to use way. For instance, this vendor first came on my radar when looking for an MSK solution in GCP (where no Managed Kafka option presently exists).