Published: Aug 15, 2024 by Isaac Johnson
Having created a new .NET 8 MVC app in the post earlier this week, it is time for us to instrument it.
Today we will set up New Relic Instrumentation via Docker and the rich local agent approach, then look at gathering data from our PostgreSQL databases. I’ll also show some example Synthetic Monitors in New Relic including Scripted Browser checks.
We’ll pivot and switch to OpenTelemetry and collecting Logs, Metrics and Traces using .NET and the OpenTelemetry libraries. We’ll roll that into both our Dockerfile and helm charts.
With New Relic sorted, we’ll also look at using an OpenTelemetry collector running on a container to send traces out to GCP Cloud Trace as well as a local Zipkin.
Back into the Dockerfile
I’m going to start with the .NET installation and generate a new key
I’ll note that it’s running on Linux and give it a name (MvcPatients)
I next will see the updated Dockerfile
# Install the agent
RUN apt-get update && apt-get install -y wget ca-certificates gnupg \
&& echo 'deb http://apt.newrelic.com/debian/ newrelic non-free' | tee /etc/apt/sources.list.d/newrelic.list \
&& wget https://download.newrelic.com/548C16BF.gpg \
&& apt-key add 548C16BF.gpg \
&& apt-get update \
&& apt-get install -y 'newrelic-dotnet-agent' \
&& rm -rf /var/lib/apt/lists/*
# Enable the agent
ENV CORECLR_ENABLE_PROFILING=1 \
CORECLR_PROFILER={36032161-FFC0-4B61-B559-F6C5D41BAE5A} \
CORECLR_NEWRELIC_HOME=/usr/local/newrelic-dotnet-agent \
CORECLR_PROFILER_PATH=/usr/local/newrelic-dotnet-agent/libNewRelicProfiler.so \
NEW_RELIC_LICENSE_KEY=ba22692************************************** \
NEW_RELIC_APP_NAME="MvcPatients"
Note: When copied to the clipboard, it does show the actual key in the output.
I can now add that to my Dockerfile and save
Now this next part is quite important. We are about to build a container with a given key baked in. That means we really should not push this to Dockerhub.
As such, I’ll build and push to my private Harbor registry. If you pay for Dockerhub, you can use a private registry. There are also options using things like GAR or GCR in GCP.
We can now build
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ docker build -t harbor.freshbrewed.science/freshbrewedprivate/mvcp
aitentswnr:0.1 .
[+] Building 23.9s (16/16) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.55kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:8.0 0.2s
=> [internal] load metadata for mcr.microsoft.com/dotnet/sdk:8.0 0.2s
=> [internal] load build context 0.1s
=> => transferring context: 77.88kB 0.1s
=> [build 1/6] FROM mcr.microsoft.com/dotnet/sdk:8.0@sha256:7d0ba26469267b563120456557e38eccef9972cb6b9cfbbd 0.0s
=> [runtime 1/4] FROM mcr.microsoft.com/dotnet/aspnet:8.0@sha256:3deda593cf10581cbacfa16a1fbb090353d14beaa65 0.0s
=> CACHED [build 2/6] WORKDIR /app 0.0s
=> CACHED [build 3/6] COPY *.csproj ./ 0.0s
=> CACHED [build 4/6] RUN dotnet restore 0.0s
=> [build 5/6] COPY . ./ 1.2s
=> [build 6/6] RUN dotnet publish MvcPatients.generated.sln -c Release -o out 9.7s
=> CACHED [runtime 2/4] WORKDIR /app 0.0s
=> [runtime 3/4] COPY --from=build /app/out . 0.4s
=> [runtime 4/4] RUN apt-get update && apt-get install -y wget ca-certificates gnupg && echo 'deb http://ap 11.2s
=> exporting to image 0.4s
=> => exporting layers 0.4s
=> => writing image sha256:739568d04ecbdf974a5605f1160034ecb4814355192e0048aa780ae89dd4b422 0.0s
=> => naming to harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr:0.1 0.0s
and push
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ docker push harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr:0.1
aitentswnr:0.1: command not found
The push refers to repository [harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr]shbrewedprivate/mvcpaitentswnr:0.1
70bdaffd26dd: Pushed
096acb12ddb6: Pushed
cbd8bd8f6f0e: Pushed
698640980ef8: Pushed
58fa834ef12a: Pushed
855e51907d3e: Pushed ad8af893343b: Pushed
2ea3b529cc6c: Pushed
e0781bc8667f: Pushed
0.1: digest: sha256:5d070b59aa1fcc6f51559e9f90ad0406f733c9f2ad56a7558c899120c8b747e9 size: 2209
I’ll then make sure to add a valid DockerConfig image pull secret to the namespace
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl apply -f ./t.o -n patientsmvc
secret/myharborreg created
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl get secrets -n patientsmvc
NAME TYPE DATA AGE
db-secret Opaque 1 144m
sh.helm.release.v1.mypatientmvc.v1 helm.sh/release.v1 1 144m
myharborreg kubernetes.io/dockerconfigjson 1 10s
With an updated helm chart, I’ll now use that secret with an upgrade while also swapping container images to harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr:0.1
$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=pg-295c4a9c-isaac-1040.aivencloud.com --set env.dbName=defaultdb --set env.dbPort=11996 --set env.dbUser=avnadmin --set env.dbPassword='xxxxxxxxxxxxx' --set image.repository=harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr --set image.tag=0.1 --set imagePullSecrets[0].name=myharborreg ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sat Aug 10 15:14:24 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 2
TEST SUITE: None
I did notice an error right off
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl get pods -n patientsmvc
NAME READY STATUS RESTARTS AGE
mypatientmvc-deployment-6b9d8f65d8-7gtjs 1/1 Running 0 46s
mypatientmvc-deployment-6b9d8f65d8-xp7xp 1/1 Running 0 35s
mypatientmvc-deployment-6b9d8f65d8-98289 0/1 CrashLoopBackOff 2 (16s ago) 40s
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl logs mypatientmvc-deployment-6b9d8f65d8-98289 -n patientsmvc
Unhandled exception. System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
at System.IO.FileSystemWatcher.StartRaisingEvents()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.ApplyDefaultAppConfiguration(HostBuilderContext hostingContext, IConfigurationBuilder appConfigBuilder, String[] args)
at Microsoft.Extensions.Hosting.HostApplicationBuilder..ctor(HostApplicationBuilderSettings settings)
at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)
at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder(String[] args)
at Program.<Main>$(String[] args) in /app/Program.cs:line 6
I’m going to add to the Dockerfile, before the expose port line an update to increase the limits
# Set the inotify limits
RUN echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.conf && \
echo "fs.inotify.max_user_watches=524288" >> /etc/sysctl.conf
Then build and push
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ docker build -t harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr:0.2 . && docker push harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr:0.2
[+] Building 10.8s (17/17) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.72kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:8.0 0.2s
=> [internal] load metadata for mcr.microsoft.com/dotnet/sdk:8.0 0.2s
=> [build 1/6] FROM mcr.microsoft.com/dotnet/sdk:8.0@sha256:7d0ba26469267b563120456557e38eccef9972cb6b9cfbbd47a50d1218fa7b30 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 58.87kB 0.0s
=> [runtime 1/5] FROM mcr.microsoft.com/dotnet/aspnet:8.0@sha256:3deda593cf10581cbacfa16a1fbb090353d14beaa65adca4611c7c7a458d66b0 0.0s
=> CACHED [build 2/6] WORKDIR /app 0.0s
=> CACHED [build 3/6] COPY *.csproj ./ 0.0s
=> CACHED [build 4/6] RUN dotnet restore 0.0s
=> [build 5/6] COPY . ./ 0.7s
=> [build 6/6] RUN dotnet publish MvcPatients.generated.sln -c Release -o out 8.7s
=> CACHED [runtime 2/5] WORKDIR /app 0.0s
=> CACHED [runtime 3/5] COPY --from=build /app/out . 0.0s
=> CACHED [runtime 4/5] RUN apt-get update && apt-get install -y wget ca-certificates gnupg && echo 'deb http://apt.newrelic.com/deb 0.0s
=> [runtime 5/5] RUN echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.conf && echo "fs.inotify.max_user_watches=524288" 0.6s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1a8ee037c61126942617f84d8174827aa3aa082a05ef0e52ac87ad9fae87a57a 0.0s
=> => naming to harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr:0.2 0.0s
The push refers to repository [harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr]
3808dadd3e10: Pushed
70bdaffd26dd: Layer already exists
096acb12ddb6: Layer already exists
cbd8bd8f6f0e: Layer already exists
698640980ef8: Layer already exists
58fa834ef12a: Layer already exists
855e51907d3e: Layer already exists
ad8af893343b: Layer already exists
2ea3b529cc6c: Layer already exists
e0781bc8667f: Layer already exists
0.2: digest: sha256:8d80ac6b9afca7b5a937bd9d7c93a0f1089ecd37d141137cfb504c465efa1f70 size: 2416
Since a rotate didn’t help
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl get pods -n patientsmvc
NAME READY STATUS RESTARTS AGE
mypatientmvc-deployment-cf945646c-knd5p 1/1 Running 0 16s
mypatientmvc-deployment-cf945646c-mp678 1/1 Running 0 13s
mypatientmvc-deployment-cf945646c-mvqcb 0/1 CrashLoopBackOff 1 (4s ago) 8s
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl logs mypatientmvc-deployment-cf945646c-mvqcb -n patientsmvc
Unhandled exception. System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
at System.IO.FileSystemWatcher.StartRaisingEvents()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.ApplyDefaultAppConfiguration(HostBuilderContext hostingContext, IConfigurationBuilder appConfigBuilder, String[] args)
at Microsoft.Extensions.Hosting.HostApplicationBuilder..ctor(HostApplicationBuilderSettings settings)
at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)
at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder(String[] args)
at Program.<Main>$(String[] args) in /app/Program.cs:line 6
I suspect I have an ailing Kubernetes node that might need a reboot or update. Looks like it’s the master node
$ kubectl describe pod mypatientmvc-deployment-cf945646c-mvqcb -n patientsmvc | grep Node:
Node: builder-hp-elitebook-745-g5/192.168.1.33
Our Service will load balance so for now, I can ignore this, but I will hop to that host to update for the next reboot:
builder@builder-HP-EliteBook-745-G5:~$ ulimit -n 1000000
builder@builder-HP-EliteBook-745-G5:~$ sysctl -w fs.file-max=1000000
sysctl: permission denied on key "fs.file-max", ignoring
builder@builder-HP-EliteBook-745-G5:~$ sudo sysctl -w fs.file-max=1000000
fs.file-max = 1000000
builder@builder-HP-EliteBook-745-G5:~$ sudo vi /etc/security/limits.conf
builder@builder-HP-EliteBook-745-G5:~$ sudo cat /etc/security/limits.conf | grep file-max
fs.file-max = 1000000
Because I’m not running privileged, we will not get host metrics so I am just expecting data from the .NET agent for now
I did a port forward
$ kubectl port-forward svc/mypatientmvc-service -n patientsmvc 8888:80
Forwarding from 127.0.0.1:8888 -> 8080
Forwarding from [::1]:8888 -> 8080
Handling connection for 8888
Handling connection for 8888
Handling connection for 8888
Handling connection for 8888
and added some data
I can now see details reflected in New Relic APM
With Distributed Tracing I can see what was being requested from that container
It becomes easy to see that a lot of my slowness comes from the slow database in a Droplet in NYC. But it is using a free-tier PostgreSQL database from Aiven.io at this point
The logs are also pulled in from the NewRelic instrumentation showing our SQL queries executed
Ingress and Test Frameworks
Use Kube-proxy and pointing and clicking might generate a little bit of data, but it might be challenging to do that all the time. My mantra is: do, do, then write a script.
First, I’ll create a A record in CloudDNS
$ gcloud dns --project=myanthosproject2 record-sets create patientsmvc.steeped.space --zone="steepedspace" --type="A"
--ttl="300" --rrdatas="75.73.224.240"
NAME TYPE TTL DATA
patientsmvc.steeped.space. A 300 75.73.224.240
Next, I’ll create the Kubernetes Ingress YAML that will use it.
$ cat patientsmvc.ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: gcpleprod2
ingress.kubernetes.io/proxy-body-size: "0"
ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.org/client-max-body-size: "0"
nginx.org/proxy-connect-timeout: "3600"
nginx.org/proxy-read-timeout: "3600"
nginx.org/websocket-services: mypatientmvc-service
name: patientmvcingress
spec:
rules:
- host: patientsmvc.steeped.space
http:
paths:
- backend:
service:
name: mypatientmvc-service
port:
number: 80
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- patientsmvc.steeped.space
secretName: patientsmvc-tls
Note: We created the gcpleprod2 cluster-issuer in the blog post about GCP CloudDNS.
I’ll apply it
$ kubectl apply -f ./patientsmvc.ingress.yaml -n patientsmvc
ingress.networking.k8s.io/patientmvcingress created
Once the cert is fetched
$ kubectl get cert -n patientsmvc
NAME READY SECRET AGE
patientsmvc-tls True patientsmvc-tls 73s
Which now loads on https://patientsmvc.steeped.space/Patients
New Relic Synthetics
We can use New Relic to actually do some of the testing.
Our first step is a page crawler. We can get there from “Synthetic Monitoring” and “Create monitor”
I can then add the URL and period. Because I’m in a free tier, I just have 500 checks I can do in a month so I’ll set the period to 5 minutes for the moment, then scale back to weekly
This is a bit of a bummer. It won’t let me create based on the estimated monthly which is really a future problem.
I went back and changed to daily and then I could save
I can see the check created but not sure when it will run
Interestingly enough I can edit and change to an hour and even though it would be over, it lets me save
I’ll just wait an hour to see it run. We can see the activity stream on the upper right
After a bit, I could see data in our graphs
New Relic Scripted Browser
Let’s now create a scripted browser test. We can find that in Synthetic Monitors as well
I then select the type of browser and some device emulation options as well as period
Like before, I’ll select a location. To be different, I’ll select Montreal:
We then go to the “Write Script” section.
For the script, I’ll use
var assert = require('assert');
$browser.get('https://patientsmvc.steeped.space/Patients').then(function(){
// Check the H1 title matches "Example Domain"
return $browser.findElement($driver.By.css('h1')).then(function(element){
return element.getText().then(function(text){
assert.equal('Index', text, 'Page H1 title did not match');
});
});
}).then(function(){
// Check that the external link matches "https://www.iana.org/domains/example"
return $browser.findElement($driver.By.linkText('Create New')).then(function(element){
element.click();
});
});
I can then click “Validate” and see the output
APM Results
Because we instrumented already, we can see some data now in our APM Services page for the MvcPatients app
You’ll note that now in the lower right pane for “Related entities”, we see linked User Experience.
These can highlight issues from our monitors. Mouse over the icon and we can see the names of those monitors we created
If I click “See full map”, we actually can see a nice visualization of all that New Relic is now aware of related to this service
PostgreSQL instrumentation
NewRelic is aware that we have an uninstrumented database
If I click the link we see, indeed, it is the database
Since this is using an Aiven.io instance I can’t pick a SaaS option like “Google Cloud SQL”, but I can choose PostgreSQL
Next, I need to pick a host that can run our agent that will engage with PostgreSQL. The problem, as we see, is that I don’t have an infrastructure agent on this New Relic instance yet
I’ll go ahead and choose to instrument a new host and create a new key
The next step provides the instructions I’ll need to install
I can run that on a utility host
Once completed
I can click Next in New Relic and see we have data from the agent
Our next step reminds us we need psql installed
However, in this refreshed host, it is absent
isaac@isaac-MacBookAir:~$ psql --version
Command 'psql' not found, but can be installed with:
sudo apt install postgresql-client-common
I can install with a simple one-liner
isaac@isaac-MacBookAir:~$ sudo apt update && sudo apt install postgresql-client -y
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 https://download.newrelic.com/infrastructure_agent/linux/apt jammy InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:5 https://debian.neo4j.com stable InRelease
Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
58 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: https://download.newrelic.com/infrastructure_agent/linux/apt/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
app-install-data-partner bsdmainutils g++-9 gcc-10-base gir1.2-clutter-1.0 gir1.2-clutter-gst-3.0 gir1.2-cogl-1.0 gir1.2-coglpango-1.0
gir1.2-gnomebluetooth-1.0 gir1.2-gtkclutter-1.0 gnome-getting-started-docs gnome-screenshot ippusbxd libamtk-5-0 libamtk-5-common
libaom0 libasn1-8-heimdal libboost-date-time1.71.0 libboost-filesystem1.71.0 libboost-iostreams1.71.0 libboost-locale1.71.0
libboost-thread1.71.0 libbrlapi0.7 libcamel-1.2-62 libcbor0.6 libcdio18 libcmis-0.5-5v5 libcodec2-0.9 libdns-export1109 libdvdnav4
libdvdread7 libedataserver-1.2-24 libedataserverui-1.2-2 libextutils-pkgconfig-perl libfftw3-double3 libfprint-2-tod1 libfuse2
libfwupdplugin1 libgdk-pixbuf-xlib-2.0-0 libgdk-pixbuf2.0-0 libgssapi3-heimdal libgupnp-1.2-0 libhandy-0.0-0 libhcrypto4-heimdal
libheimbase1-heimdal libheimntlm0-heimdal libhogweed5 libhx509-5-heimdal libicu66 libidn11 libigdgmm11 libisl22 libjson-c4 libjuh-java
libjurt-java libkrb5-26-heimdal libldap-2.4-2 liblibreoffice-java libllvm10 libllvm12 libmozjs-68-0 libmpdec2 libmysqlclient21
libneon27-gnutls libnettle7 libntfs-3g883 liborcus-0.15-0 libperl5.30 libpgm-5.2-0 libphonenumber7 libpoppler97 libprotobuf17
libpython3.8 libpython3.8-minimal libpython3.8-stdlib libqpdf26 libraw19 libreoffice-style-tango libridl-java libroken18-heimdal
libsane libsnmp35 libssl1.1 libstdc++-9-dev libtepl-4-0 libtracker-control-2.0-0 libtracker-miner-2.0-0 libtracker-sparql-2.0-0
libunoloader-java libvpx6 libwebp6 libwind0-heimdal libwmf0.2-7 libx264-155 libx265-179 libxmlb1 linux-hwe-5.15-headers-5.15.0-100
ltrace lz4 mysql-common ncal perl-modules-5.30 pkg-config popularity-contest python3-entrypoints python3-requests-unixsocket
python3-simplejson python3.8 python3.8-minimal syslinux syslinux-common syslinux-legacy ure-java vino xul-ext-ubufox
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
postgresql-client-14 postgresql-client-common
Suggested packages:
postgresql-14 postgresql-doc-14
The following NEW packages will be installed:
postgresql-client postgresql-client-14 postgresql-client-common
0 upgraded, 3 newly installed, 0 to remove and 58 not upgraded.
Need to get 1,256 kB of archives.
After this operation, 4,185 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.12-0ubuntu0.22.04.1 [1,223 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B]
Fetched 1,256 kB in 4s (326 kB/s)
Selecting previously unselected package postgresql-client-common.
(Reading database ... 271678 files and directories currently installed.)
Preparing to unpack .../postgresql-client-common_238_all.deb ...
Unpacking postgresql-client-common (238) ...
Selecting previously unselected package postgresql-client-14.
Preparing to unpack .../postgresql-client-14_14.12-0ubuntu0.22.04.1_amd64.deb ...
Unpacking postgresql-client-14 (14.12-0ubuntu0.22.04.1) ...
Selecting previously unselected package postgresql-client.
Preparing to unpack .../postgresql-client_14+238_all.deb ...
Unpacking postgresql-client (14+238) ...
Setting up postgresql-client-common (238) ...
Setting up postgresql-client-14 (14.12-0ubuntu0.22.04.1) ...
update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up postgresql-client (14+238) ...
Processing triggers for man-db (2.10.2-1) ...
and verify I have it installed
isaac@isaac-MacBookAir:~$ psql --version
psql (PostgreSQL) 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1
The other check was to ensure we had up-to-date ciphers
$ openssl ciphers -v | awk '{print " - " $2}' | sort | uniq
- SSLv3
- TLSv1
- TLSv1.2
- TLSv1.3
The next page is mostly right, but we’ll have to do some tweaks
For instance, I’ll need to connect as the “main” user, which is usually ‘postgres’, but not in this instance. The same might be true for CloudSQL where we have a different ‘admin’ user.
builder@DESKTOP-QADGF36:~$ export PGPASSWORD=Axxxxxxxxxxxxxxxxxxxx
builder@DESKTOP-QADGF36:~$ psql -U avnadmin -h pg-295c4a9c-isaac-1040.aivencloud.com --dbname=defaultdb --port=11996
psql (12.19 (Ubuntu 12.19-0ubuntu0.20.04.1), server 15.7)
WARNING: psql major version 12, server major version 15.
Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
defaultdb=>
I’ll then use the template, but use a different password, of course
-- Create the 'newrelic' user
CREATE USER newrelic WITH LOGIN PASSWORD 'MyNewRelicPassword';
-- Grant SELECT privileges
GRANT SELECT ON pg_stat_database, pg_stat_database_conflicts, pg_stat_bgwriter TO newrelic;
I’m a bit worried as I see no priveldges warnings when I run the steps
defaultdb=> CREATE USER newrelic WITH LOGIN PASSWORD 'NotTheRealPassword';
CREATE ROLE
defaultdb=> GRANT SELECT ON pg_stat_database, pg_stat_database_conflicts, pg_stat_bgwriter TO newrelic;
WARNING: no privileges were granted for "pg_stat_database"
WARNING: no privileges were granted for "pg_stat_database_conflicts"
WARNING: no privileges were granted for "pg_stat_bgwriter"
GRANT
The test seems to pass
builder@DESKTOP-QADGF36:~$ export PGPASSWORD=NotTheRealPassword
builder@DESKTOP-QADGF36:~$ psql -U newrelic -h pg-295c4a9c-isaac-1040.aivencloud.com --port=11996 --dbname=defaultdb --command '\conninfo'
| grep -ie "^You are connected"
You are connected to database "defaultdb" as user "newrelic" on host "pg-295c4a9c-isaac-1040.aivencloud.com" (address "159.89.38.30") at port "11996".
Our next page in the steps
I first install the ‘nri-postgresql’ integration
isaac@isaac-MacBookAir:~$ sudo apt-get install nri-postgresql -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
app-install-data-partner bsdmainutils g++-9 gcc-10-base gir1.2-clutter-1.0 gir1.2-clutter-gst-3.0 gir1.2-cogl-1.0 gir1.2-coglpango-1.0
gir1.2-gnomebluetooth-1.0 gir1.2-gtkclutter-1.0 gnome-getting-started-docs gnome-screenshot ippusbxd libamtk-5-0 libamtk-5-common
libaom0 libasn1-8-heimdal libboost-date-time1.71.0 libboost-filesystem1.71.0 libboost-iostreams1.71.0 libboost-locale1.71.0
libboost-thread1.71.0 libbrlapi0.7 libcamel-1.2-62 libcbor0.6 libcdio18 libcmis-0.5-5v5 libcodec2-0.9 libdns-export1109 libdvdnav4
libdvdread7 libedataserver-1.2-24 libedataserverui-1.2-2 libextutils-pkgconfig-perl libfftw3-double3 libfprint-2-tod1 libfuse2
libfwupdplugin1 libgdk-pixbuf-xlib-2.0-0 libgdk-pixbuf2.0-0 libgssapi3-heimdal libgupnp-1.2-0 libhandy-0.0-0 libhcrypto4-heimdal
libheimbase1-heimdal libheimntlm0-heimdal libhogweed5 libhx509-5-heimdal libicu66 libidn11 libigdgmm11 libisl22 libjson-c4 libjuh-java
libjurt-java libkrb5-26-heimdal libldap-2.4-2 liblibreoffice-java libllvm10 libllvm12 libmozjs-68-0 libmpdec2 libmysqlclient21
libneon27-gnutls libnettle7 libntfs-3g883 liborcus-0.15-0 libperl5.30 libpgm-5.2-0 libphonenumber7 libpoppler97 libprotobuf17
libpython3.8 libpython3.8-minimal libpython3.8-stdlib libqpdf26 libraw19 libreoffice-style-tango libridl-java libroken18-heimdal
libsane libsnmp35 libssl1.1 libstdc++-9-dev libtepl-4-0 libtracker-control-2.0-0 libtracker-miner-2.0-0 libtracker-sparql-2.0-0
libunoloader-java libvpx6 libwebp6 libwind0-heimdal libwmf0.2-7 libx264-155 libx265-179 libxmlb1 linux-hwe-5.15-headers-5.15.0-100
ltrace lz4 mysql-common ncal perl-modules-5.30 pkg-config popularity-contest python3-entrypoints python3-requests-unixsocket
python3-simplejson python3.8 python3.8-minimal syslinux syslinux-common syslinux-legacy ure-java vino xul-ext-ubufox
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
nri-postgresql
0 upgraded, 1 newly installed, 0 to remove and 58 not upgraded.
Need to get 2,508 kB of archives.
After this operation, 5,836 kB of additional disk space will be used.
Get:1 https://download.newrelic.com/infrastructure_agent/linux/apt jammy/main amd64 nri-postgresql amd64 2.13.6 [2,508 kB]
Fetched 2,508 kB in 0s (7,210 kB/s)
Selecting previously unselected package nri-postgresql.
(Reading database ... 271981 files and directories currently installed.)
Preparing to unpack .../nri-postgresql_2.13.6_amd64.deb ...
Unpacking nri-postgresql (2.13.6) ...
Setting up nri-postgresql (2.13.6) ...
Note: if you see no provider, you might have, as i did, forgot you are in a different window and are on WSL.. however, if you do see this but know you have NRI installed
builder@DESKTOP-QADGF36:~$ sudo apt-get install nri-postgresql -y
[sudo] password for builder:
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nri-postgresql
You can fix with adding the NRI repos
builder@DESKTOP-QADGF36:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
builder@DESKTOP-QADGF36:~$ curl -fsSL https://download.newrelic.com/infrastructure_agent/gpg/newrelic-infra.gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/newrelic-infra.gpg
builder@DESKTOP-QADGF36:~$ echo "deb https://download.newrelic.com/infrastructure_agent/linux/apt focal main" | sudo tee -a /etc/apt/sources.list.d/newrelic-infra.list
deb https://download.newrelic.com/infrastructure_agent/linux/apt focal main
builder@DESKTOP-QADGF36:~$ sudo apt-get update
... snip ...
I’ll then go to the integrations directory and update the postgresql-config.yaml
isaac@isaac-MacBookAir:/etc/newrelic-infra/integrations.d$ sudo cp postgresql-config.yml.sample postgresql-config.yml
isaac@isaac-MacBookAir:/etc/newrelic-infra/integrations.d$ sudo vi postgresql-config.yml
Sadly, this isn’t working when i test
isaac@isaac-MacBookAir:/etc/newrelic-infra/integrations.d$ sudo /usr/bin/newrelic-infra -dry_run -integration_config_path /etc/newrelic-infra/integrations.d/postgresql-config.yml
time="2024-08-11T11:32:27-05:00" level=error msg="integration log" component=integration-errors env=production fields.level=error fields.msg="Failed to build schema list for database 'defaultdb': pq: no pg_hba.conf entry for host \"75.73.224.240\", user \"newrelic\", database \"defaultdb\", no encryption" integration_name=nri-postgresql role=postgresql runner_uid=43bba91259
time="2024-08-11T11:32:27-05:00" level=error msg="integration log" component=integration-errors env=production fields.level=error fields.msg="Error creating list of entities to collect: no database to collect data" integration_name=nri-postgresql role=postgresql runner_uid=43bba91259
time="2024-08-11T11:32:27-05:00" level=warning msg="integration exited with error state" component=integrations.runner.Runner env=production error="exit status 1" integration_name=nri-postgresql role=postgresql runner_uid=43bba91259 stderr="[ERR] Failed to build schema list for database 'defaultdb': pq: no pg_hba.conf entry for host \"75.73.224.240\", user \"newrelic\", database \"defaultdb\", no encryption\n[ERR] Error creating list of entities to collect: no database to collect data"
I had to mess with it a bit, ultimately using my admin user instead
isaac@isaac-MacBookAir:/etc/newrelic-infra/integrations.d$ sudo cat ./postgresql-config.yml | grep -v '^\s*#'
integrations:
- name: nri-postgresql
env:
USERNAME: avnadmin
PASSWORD: 'xxxxxx not the real password xxxxxxx'
HOSTNAME: pg-295c4a9c-isaac-1040.aivencloud.com
PORT: "11996"
COLLECTION_LIST: '["defaultdb"]'
COLLECT_DB_LOCK_METRICS: "false"
COLLECT_BLOAT_METRICS: "true"
ENABLE_SSL: "true"
TRUST_SERVER_CERTIFICATE: "true"
TIMEOUT: "10"
interval: 15s
labels:
env: production
role: postgresql
inventory_source: config/postgresql
which now works
$ sudo /usr/bin/newrelic-infra -dry_run -integration_config_path /etc/newrelic-infra/integrations.d/postgresql-config.yml | grep -wo "Integration health check finished with success"
Integration health check finished with success
I’ll skip logging for now as this is a remote server
The test was timing out because we did the validation but didn’t make it live
I restarted the agent
isaac@isaac-MacBookAir:/etc/newrelic-infra/integrations.d$ sudo systemctl restart newrelic-infra
isaac@isaac-MacBookAir:/etc/newrelic-infra/integrations.d$ sudo systemctl status newrelic-infra
● newrelic-infra.service - New Relic Infrastructure Agent
Loaded: loaded (/etc/systemd/system/newrelic-infra.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2024-08-11 11:41:10 CDT; 36s ago
Main PID: 744514 (newrelic-infra-)
Tasks: 33 (limit: 9327)
Memory: 33.8M (limit: 1.0G)
CPU: 1.419s
CGroup: /system.slice/newrelic-infra.service
├─744514 /usr/bin/newrelic-infra-service
├─744522 /usr/bin/newrelic-infra
├─744711 /opt/fluent-bit/bin/fluent-bit -c /tmp/fb/nr_fb_config1844940112 -e /var/db/newrelic-infra/newrelic-integrations/lo>
└─744733 /var/db/newrelic-infra/newrelic-integrations/bin/nri-postgresql
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=se>
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=sy>
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=ke>
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=ke>
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=se>
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=co>
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=pa>
Aug 11 11:41:16 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:16-05:00" level=info msg="Agent plugin" plugin=co>
Aug 11 11:41:17 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:17-05:00" level=info msg="connect got id" agent-g>
Aug 11 11:41:17 isaac-MacBookAir newrelic-infra-service[744522]: time="2024-08-11T11:41:17-05:00" level=info msg="Integration health chec>
lines 1-23/23 (END)
But still it did not seem to like that
I might have to pivot to a local PostgreSQL as there just is no root postgres database that collects stats
builder@DESKTOP-QADGF36:~$ psql -U avnadmin -h pg-295c4a9c-isaac-1040.aivencloud.com --dbname=postgres --port=11996
psql: error: FATAL: database "postgres" does not exist
That said, we can see some database operations as collected by our APM agent in the Services details
As well as Performance
Local PostgreSQL
Now that I think on it, I could just use this utility box as the database host for our app.
I just need to install postgresql (the server)
isaac@isaac-MacBookAir:~$ sudo apt install postgresql
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
app-install-data-partner bsdmainutils g++-9 gcc-10-base gir1.2-clutter-1.0 gir1.2-clutter-gst-3.0 gir1.2-cogl-1.0 gir1.2-coglpango-1.0
gir1.2-gnomebluetooth-1.0 gir1.2-gtkclutter-1.0 gnome-getting-started-docs gnome-screenshot ippusbxd libamtk-5-0 libamtk-5-common
libaom0 libasn1-8-heimdal libboost-date-time1.71.0 libboost-filesystem1.71.0 libboost-iostreams1.71.0 libboost-locale1.71.0
libboost-thread1.71.0 libbrlapi0.7 libcamel-1.2-62 libcbor0.6 libcdio18 libcmis-0.5-5v5 libcodec2-0.9 libdns-export1109 libdvdnav4
libdvdread7 libedataserver-1.2-24 libedataserverui-1.2-2 libextutils-pkgconfig-perl libfftw3-double3 libfprint-2-tod1 libfuse2
libfwupdplugin1 libgdk-pixbuf-xlib-2.0-0 libgdk-pixbuf2.0-0 libgssapi3-heimdal libgupnp-1.2-0 libhandy-0.0-0 libhcrypto4-heimdal
libheimbase1-heimdal libheimntlm0-heimdal libhogweed5 libhx509-5-heimdal libicu66 libidn11 libigdgmm11 libisl22 libjson-c4 libjuh-java
libjurt-java libkrb5-26-heimdal libldap-2.4-2 liblibreoffice-java libllvm10 libllvm12 libmozjs-68-0 libmpdec2 libmysqlclient21
libneon27-gnutls libnettle7 libntfs-3g883 liborcus-0.15-0 libperl5.30 libpgm-5.2-0 libphonenumber7 libpoppler97 libprotobuf17
libpython3.8 libpython3.8-minimal libpython3.8-stdlib libqpdf26 libraw19 libreoffice-style-tango libridl-java libroken18-heimdal
libsane libsnmp35 libssl1.1 libstdc++-9-dev libtepl-4-0 libtracker-control-2.0-0 libtracker-miner-2.0-0 libtracker-sparql-2.0-0
libunoloader-java libvpx6 libwebp6 libwind0-heimdal libwmf0.2-7 libx264-155 libx265-179 libxmlb1 linux-hwe-5.15-headers-5.15.0-100
ltrace lz4 mysql-common ncal perl-modules-5.30 pkg-config popularity-contest python3-entrypoints python3-requests-unixsocket
python3-simplejson python3.8 python3.8-minimal syslinux syslinux-common syslinux-legacy ure-java vino xul-ext-ubufox
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
libcommon-sense-perl libjson-perl libjson-xs-perl libllvm14 libtypes-serialiser-perl postgresql-14 postgresql-common sysstat
Suggested packages:
postgresql-doc isag
The following NEW packages will be installed:
libcommon-sense-perl libjson-perl libjson-xs-perl libllvm14 libtypes-serialiser-perl postgresql postgresql-14 postgresql-common
sysstat
0 upgraded, 9 newly installed, 0 to remove and 58 not upgraded.
Need to get 41.0 MB of archives.
After this operation, 157 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
... snip ...
I then need to enable it to listen to any addresses
isaac@isaac-MacBookAir:~$ sudo vi /etc/postgresql/14/main/postgresql.conf
isaac@isaac-MacBookAir:~$ cat /etc/postgresql/14/main/postgresql.conf | grep listen
listen_addresses = '*' # what IP address(es) to listen on;
I’ll set a password
isaac@isaac-MacBookAir:~$ sudo -u postgres psql template1
psql (14.12 (Ubuntu 14.12-0ubuntu0.22.04.1))
Type "help" for help.
template1=# ALTER USER postgres with encrypted password 'AVNS_DGv7-vTV1RBNuwjATg5';
ALTER ROLE
template1=# \q
In the /etc/postgresql/14/main/pg_hba.conf, I want to add
# Allow local network
host all all 192.168.1.1/24 scram-sha-256
I’ll test from WSL
builder@DESKTOP-QADGF36:~$ psql -h 192.168.1.78 -U postgres --password --dbname template1
Password:
psql (12.19 (Ubuntu 12.19-0ubuntu0.20.04.1), server 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1))
WARNING: psql major version 12, server major version 14.
Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
template1=# \list
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
template1=
I’m now going to create a new database and table.
First, the database and user
builder@DESKTOP-QADGF36:~/Workspaces/exampleDotNetPatientApp$ psql -h 192.168.1.78 -U postgres --password --dbname postgres
Password:
psql (12.19 (Ubuntu 12.19-0ubuntu0.20.04.1), server 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1))
WARNING: psql major version 12, server major version 14.
Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
postgres=# CREATE DATABASE patientsdb;
CREATE DATABASE
postgres=# CREATE USER patientsuser WITH PASSWORD 'patientPassword1';
CREATE ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE patientsdb TO patientsuser;
GRANT
postgres=#
I can then test connecting to it as the user
builder@DESKTOP-QADGF36:~/Workspaces/exampleDotNetPatientApp$ psql -h 192.168.1.78 -U patientsuser --password --dbname patientsdb
Password:
psql (12.19 (Ubuntu 12.19-0ubuntu0.20.04.1), server 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1))
WARNING: psql major version 12, server major version 14.
Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
patientsdb=>
I can now create the Table (with the casing .NET expects)
CREATE TABLE "Patient" (
"Id" SERIAL PRIMARY KEY,
"FirstName" VARCHAR(50),
"LastName" VARCHAR(50),
"DateOfBirth" DATE,
"SocialSecurityNumber" CHAR(11),
"CreatedDate" TIMESTAMP DEFAULT CURRENT_TIMESTAMP);
Applied and verified
patientsdb=> CREATE TABLE "Patient" (
KEY,
"Firstpatientsdb(> "Id" SERIAL PRIMARY KEY,
patientsdb(> "FirstName" VARCHAR(50),
patientsdb(> "LastName" VARCHAR(50),
patientsdb(> "DateOfBirth" DATE,
patientsdb(> "SocialSecurityNumber" CHAR(11),
patientsdb(> "CreatedDate" TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE
patientsdb=> \dt;
List of relations
Schema | Name | Type | Owner
--------+---------+-------+--------------
public | Patient | table | patientsuser
(1 row)
I can then insert the sample data
patientsdb=> INSERT INTO "Patient" ("FirstName", "LastName", "DateOfBirth", "SocialSecurityNumber") VALUES
patientsdb-> ('John', 'Doe', '1985-05-15', '123-45-6789'),
patientsdb-> ('Jane', 'Smith', '1990-07-22', '987-65-4321'),
patientsdb-> ('Alice', 'Johnson', '1978-03-10', '111-22-3333'),
patientsdb-> ('Bob', 'Brown', '1982-11-30', '444-55-6666'),
patientsdb-> ('Charlie', 'Davis', '1995-01-25', '777-88-9999'),
patientsdb-> ('Diana', 'Miller', '1988-09-14', '222-33-4444'),
patientsdb-> ('Eve', 'Wilson', '1992-06-18', '555-66-7777'),
patientsdb-> ('Frank', 'Moore', '1980-12-05', '888-99-0000'),
patientsdb-> ('Grace', 'Taylor', '1983-04-27', '333-44-5555'),
patientsdb-> ('Hank', 'Anderson', '1975-08-19', '666-77-8888');
INSERT 0 10
I’ll follow the steps to add the PostgreSQL integration with New Relic. This time, the GRANT steps work
isaac@isaac-MacBookAir:~$ sudo -u postgres psql --dbname=postgres --port=5432
psql (14.12 (Ubuntu 14.12-0ubuntu0.22.04.1))
Type "help" for help.
postgres=# CREATE USER newrelic WITH LOGIN PASSWORD 'xxxxxxxx';
CREATE ROLE
postgres=# GRANT SELECT ON pg_stat_database, pg_stat_database_conflicts, pg_stat_bgwriter TO newrelic;
GRANT
postgres=#
The test works
isaac@isaac-MacBookAir:~$ psql --username=newrelic --host=localhost --port=5432 --dbname=postgres --command '\conninfo' | grep -ie "^You are connected"
Password for user newrelic:
You are connected to database "postgres" as user "newrelic" on host "localhost" (address "127.0.0.1") at port "5432".
I updated the integration yaml and tested with success this time
isaac@isaac-MacBookAir:~$ sudo vi /etc/newrelic-infra/integrations.d/postgresql-config.yml
isaac@isaac-MacBookAir:~$ sudo /usr/bin/newrelic-infra -dry_run -integration_config_path /etc/newrelic-infra/integrations.d/postgresql-config.yml | grep -wo "Integration health check finished with success"
Integration health check finished with success
This time I can add the PSQL logging as well (since it is local)
isaac@isaac-MacBookAir:/etc/newrelic-infra/logging.d$ sudo vi postgresql-log.yml
isaac@isaac-MacBookAir:/etc/newrelic-infra/logging.d$ cat ./postgresql-log.yml
# Log forwarder configuration for PostgreSQL logs
# NOTE: PostgreSQL installations can vary. Use the appropriate file
# log location that matches your environment/installation and version.
# Source: file
# Available customization parameters: attributes, max_line_kb, pattern.
logs:
- name: postgresql
file: /var/lib/pgsql/14/data/log/postgresql*.log
attributes:
logtype: postgresql-error
I then restart the NRI agent to make it live
isaac@isaac-MacBookAir:/etc/newrelic-infra/logging.d$ sudo systemctl restart newrelic-infra
isaac@isaac-MacBookAir:/etc/newrelic-infra/logging.d$ sudo systemctl status newrelic-infra
● newrelic-infra.service - New Relic Infrastructure Agent
Loaded: loaded (/etc/systemd/system/newrelic-infra.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2024-08-11 13:31:26 CDT; 6s ago
Main PID: 759092 (newrelic-infra-)
Tasks: 24 (limit: 9327)
Memory: 26.1M (limit: 1.0G)
CPU: 891ms
CGroup: /system.slice/newrelic-infra.service
├─759092 /usr/bin/newrelic-infra-service
├─759100 /usr/bin/newrelic-infra
├─759109 /var/db/newrelic-infra/newrelic-integrations/bin/nri-postgresql
└─759290 /sbin/modinfo cdc_ether
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=se>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=se>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=se>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=sy>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=ke>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=ke>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=se>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=co>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=pa>
Aug 11 13:31:32 isaac-MacBookAir newrelic-infra-service[759100]: time="2024-08-11T13:31:32-05:00" level=info msg="Agent plugin" plugin=co>
isaac@isaac-MacBookAir:/etc/newrelic-infra/logging.d$
This time the test connection works
Let’s now upgrade and use our local database
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=192.168.1.78 --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr --set image.tag=0.2 --set imagePullSecrets[0].name=myharborreg ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sun Aug 11 13:36:16 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 4
TEST SUITE: None
And after the pods cycle I can see it is working
I can see instrumentation on the host
I still see an error about “uninstrumented” because my Helm uses “192.168.1.78” but our NRI client sees it as “isaac-MacBookAir”.
I double checked my KubeDNS is working
root@mypatientmvc-deployment-7c46bf465f-cxlpd:/app# nslookup isaac-MacBookAir
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53
Name: isaac-MacBookAir
Address: 192.168.1.78
I’ll swap my pods to use the name instead of the IPv4
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=isaac-MacBookAir --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=harbor.freshbrewed.science/freshbrewedprivate/mvcpaitentswnr --set image.tag=0.2 --set imagePullSecrets[0].name=myharborreg ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sun Aug 11 13:47:33 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 6
TEST SUITE: None
Now we see the connection in the full view
Let’s hit the Index page 100 times real fast
$ for i in {1..100}; do wget https://patientsmvc.steeped.space/Patients; done
which we can see the 100 in the Throughput
Pivot to Open Telemetry
However, here is a challenge. I cannot even commit my updated Chart because my plaintext key is baked in to the code
Let alone that the container, with built-in key, must be not shared publicly because it will send NR data until I void the API Key.
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ git branch --show-current
newrelic
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ git add -A
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ git commit -m "Key Exposed - do not push"
[newrelic b624562] Key Exposed - do not push
5 files changed, 37 insertions(+), 1 deletion(-)
create mode 100644 t.o
How can we address this? - the key is to use OTLP with New Relic’s OTLP endpoint
First I’ll need to add the OpenTelemetry NuGet package
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
Determining projects to restore...
Writing /tmp/tmpERqvcA.tmp
info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.303/trustedroots/codesignctl.pem'.
info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.303/trustedroots/timestampctl.pem'.
info : Adding PackageReference for package 'OpenTelemetry.Exporter.OpenTelemetryProtocol' into project '/home/builder/Workspaces/MvcPatients/MvcPatients.csproj'.
info : GET https://api.nuget.org/v3/registration5-gz-semver2/opentelemetry.exporter.opentelemetryprotocol/index.json
info : OK https://api.nuget.org/v3/registration5-gz-semver2/opentelemetry.exporter.opentelemetryprotocol/index.json 45ms
info : Restoring packages for /home/builder/Workspaces/MvcPatients/MvcPatients.csproj...
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry.exporter.opentelemetryprotocol/index.json
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry.exporter.opentelemetryprotocol/index.json 47ms
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry.exporter.opentelemetryprotocol/1.9.0/opentelemetry.exporter.opentelemetryprotocol.1.9.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry.exporter.opentelemetryprotocol/1.9.0/opentelemetry.exporter.opentelemetryprotocol.1.9.0.nupkg 25ms
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry/index.json
info : GET https://api.nuget.org/v3-flatcontainer/google.protobuf/index.json
info : GET https://api.nuget.org/v3-flatcontainer/grpc.net.client/index.json
info : GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.configuration.binder/index.json
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry/index.json 30ms
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry/1.9.0/opentelemetry.1.9.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/google.protobuf/index.json 58ms
info : GET https://api.nuget.org/v3-flatcontainer/google.protobuf/3.22.5/google.protobuf.3.22.5.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/grpc.net.client/index.json 92ms
info : GET https://api.nuget.org/v3-flatcontainer/grpc.net.client/2.52.0/grpc.net.client.2.52.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.configuration.binder/index.json 122ms
info : GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.configuration.binder/8.0.1/microsoft.extensions.configuration.binder.8.0.1.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry/1.9.0/opentelemetry.1.9.0.nupkg 122ms
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry.api.providerbuilderextensions/index.json
info : GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.configuration/index.json
info : OK https://api.nuget.org/v3-flatcontainer/google.protobuf/3.22.5/google.protobuf.3.22.5.nupkg 105ms
info : OK https://api.nuget.org/v3-flatcontainer/grpc.net.client/2.52.0/grpc.net.client.2.52.0.nupkg 88ms
info : OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.configuration.binder/8.0.1/microsoft.extensions.configuration.binder.8.0.1.nupkg 94ms
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry.api.providerbuilderextensions/index.json 61ms
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry.api.providerbuilderextensions/1.9.0/opentelemetry.api.providerbuilderextensions.1.9.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry.api.providerbuilderextensions/1.9.0/opentelemetry.api.providerbuilderextensions.1.9.0.nupkg 28ms
info : OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.configuration/index.json 91ms
info : GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.configuration/8.0.0/microsoft.extensions.logging.configuration.8.0.0.nupkg
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry.api/index.json
info : OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.configuration/8.0.0/microsoft.extensions.logging.configuration.8.0.0.nupkg 27ms
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry.api/index.json 34ms
info : GET https://api.nuget.org/v3-flatcontainer/opentelemetry.api/1.9.0/opentelemetry.api.1.9.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/opentelemetry.api/1.9.0/opentelemetry.api.1.9.0.nupkg 23ms
info : GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options.configurationextensions/index.json
info : OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options.configurationextensions/index.json 29ms
info : GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options.configurationextensions/8.0.0/microsoft.extensions.options.configurationextensions.8.0.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options.configurationextensions/8.0.0/microsoft.extensions.options.configurationextensions.8.0.0.nupkg 27ms
info : GET https://api.nuget.org/v3-flatcontainer/grpc.net.common/index.json
info : OK https://api.nuget.org/v3-flatcontainer/grpc.net.common/index.json 28ms
info : GET https://api.nuget.org/v3-flatcontainer/grpc.net.common/2.52.0/grpc.net.common.2.52.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/grpc.net.common/2.52.0/grpc.net.common.2.52.0.nupkg 27ms
info : GET https://api.nuget.org/v3-flatcontainer/grpc.core.api/index.json
info : OK https://api.nuget.org/v3-flatcontainer/grpc.core.api/index.json 32ms
info : GET https://api.nuget.org/v3-flatcontainer/grpc.core.api/2.52.0/grpc.core.api.2.52.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/grpc.core.api/2.52.0/grpc.core.api.2.52.0.nupkg 29ms
info : Installed OpenTelemetry.Api 1.9.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/opentelemetry.api/1.9.0 with content hash Xz8ZvM1Lm0m7BbtGBnw2JlPo++YKyMp08zMK5p0mf+cIi5jeMt2+QsYu9X6YEAbjCxBQYwEak5Z8sY6Ig2WcwQ==.
info : Installed OpenTelemetry.Api.ProviderBuilderExtensions 1.9.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/opentelemetry.api.providerbuilderextensions/1.9.0 with content hash L0D4LBR5JFmwLun5MCWVGapsJLV0ANZ+XXu9NEI3JE/HRKkRuUO+J2MuHD5DBwiU//QMYYM4B22oev1hVLoHDQ==.
info : Installed Microsoft.Extensions.Options.ConfigurationExtensions 8.0.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/microsoft.extensions.options.configurationextensions/8.0.0 with content hash 0f4DMRqEd50zQh+UyJc+/HiBsZ3vhAQALgdkcQEalSH1L2isdC7Yj54M3cyo5e+BeO5fcBQ7Dxly8XiBBcvRgw==.
info : Installed Microsoft.Extensions.Logging.Configuration 8.0.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/microsoft.extensions.logging.configuration/8.0.0 with content hash ixXXV0G/12g6MXK65TLngYN9V5hQQRuV+fZi882WIoVJT7h5JvoYoxTEwCgdqwLjSneqh1O+66gM8sMr9z/rsQ==.
info : Installed Grpc.Net.Common 2.52.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/grpc.net.common/2.52.0 with content hash di9qzpdx525IxumZdYmu6sG2y/gXJyYeZ1ruFUzB9BJ1nj4kU1/dTAioNCMt1VLRvNVDqh8S8B1oBdKhHJ4xRg==.
info : Installed Grpc.Core.Api 2.52.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/grpc.core.api/2.52.0 with content hash SQiPyBczG4vKPmI6Fd+O58GcxxDSFr6nfRAJuBDUNj+PgdokhjWJvZE/La1c09AkL2FVm/jrDloG89nkzmVF7A==.
info : Installed OpenTelemetry 1.9.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/opentelemetry/1.9.0 with content hash 7scS6BUhwYeSXEDGhCxMSezmvyCoDU5kFQbmfyW9iVvVTcWhec+1KIN33/LOCdBXRkzt2y7+g03mkdAB0XZ9Fw==.
info : Installed OpenTelemetry.Exporter.OpenTelemetryProtocol 1.9.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/opentelemetry.exporter.opentelemetryprotocol/1.9.0 with content hash qzFOP3V2eYIVbug3U4BJzzidHe9JhAJ42WZ/H8pUp/45Ry3MQQg/+e/ZieClJcxKnpbkXi7dUq1rpvuNp+yBYA==.
info : Installed Grpc.Net.Client 2.52.0 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/grpc.net.client/2.52.0 with content hash hWVH9g/Nnjz40ni//2S8UIOyEmhueQREoZIkD0zKHEPqLxXcNlbp4eebXIOicZtkwDSx0TFz9NpkbecEDn6rBw==.
info : Installed Microsoft.Extensions.Configuration.Binder 8.0.1 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/microsoft.extensions.configuration.binder/8.0.1 with content hash 2UKFJnLiBt7Od6nCnTqP9rTIUNhzmn9Hv1l2FchyKbz8xieB9ULwZTbQZMw+M24Qw3F5dzzH1U9PPleN0LNLOQ==.
info : Installed Google.Protobuf 3.22.5 from https://api.nuget.org/v3/index.json to /home/builder/.nuget/packages/google.protobuf/3.22.5 with content hash tTMtDZPbLxJew8pk7NBdqhLqC4OipfkZdwPuCEUNr2AoDo1siUGcxFqJK0wDewTL8ge5Cjrb16CToMPxBUHMGA==.
info : GET https://api.nuget.org/v3/vulnerabilities/index.json
info : OK https://api.nuget.org/v3/vulnerabilities/index.json 23ms
info : GET https://api.nuget.org/v3-vulnerabilities/2024.08.09.12.03.59/vulnerability.base.json
info : GET https://api.nuget.org/v3-vulnerabilities/2024.08.09.12.03.59/2024.08.11.18.04.04/vulnerability.update.json
info : OK https://api.nuget.org/v3-vulnerabilities/2024.08.09.12.03.59/vulnerability.base.json 23ms
info : OK https://api.nuget.org/v3-vulnerabilities/2024.08.09.12.03.59/2024.08.11.18.04.04/vulnerability.update.json 27ms
info : Package 'OpenTelemetry.Exporter.OpenTelemetryProtocol' is compatible with all the specified frameworks in project '/home/builder/Workspaces/MvcPatients/MvcPatients.csproj'.
info : PackageReference for package 'OpenTelemetry.Exporter.OpenTelemetryProtocol' version '1.9.0' added to file '/home/builder/Workspaces/MvcPatients/MvcPatients.csproj'.
info : Generating MSBuild file /home/builder/Workspaces/MvcPatients/obj/MvcPatients.csproj.nuget.g.targets.
info : Writing assets file to disk. Path: /home/builder/Workspaces/MvcPatients/obj/project.assets.json
log : Restored /home/builder/Workspaces/MvcPatients/MvcPatients.csproj (in 1.37 sec).
In fact, I ended up installing the following packages
$ dotnet add package OpenTelemetry
$ dotnet add package OpenTelemetry.Extensions.Hosting
$ dotnet add package OpenTelemetry.Exporter.Console
$ dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
$ dotnet add package OpenTelemetry.Instrumentation.AspNetCore
Program.cs had the most changes.
But it did compile and I pushed ‘1.0’ to Dockerhhub
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ docker build -t idjohnson/freshmvcapp:1.0 .
[+] Building 12.1s (16/16) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:8.0 0.3s
=> [internal] load metadata for mcr.microsoft.com/dotnet/sdk:8.0 0.3s
=> [internal] load build context 0.0s
=> => transferring context: 57.70kB 0.0s
=> [build 1/6] FROM mcr.microsoft.com/dotnet/sdk:8.0@sha256:7d0ba26469267b563120456557e38eccef9972cb6b9cfbbd47a50d1218fa7b30 0.0s
=> [runtime 1/4] FROM mcr.microsoft.com/dotnet/aspnet:8.0@sha256:3deda593cf10581cbacfa16a1fbb090353d14beaa65adca4611c7c7a458d66b0 0.0s
=> CACHED [build 2/6] WORKDIR /app 0.0s
=> CACHED [build 3/6] COPY *.csproj ./ 0.0s
=> CACHED [build 4/6] RUN dotnet restore 0.0s
=> [build 5/6] COPY . ./ 0.5s
=> [build 6/6] RUN dotnet publish MvcPatients.generated.sln -c Release -o out 9.7s
=> CACHED [runtime 2/4] WORKDIR /app 0.0s
=> [runtime 3/4] COPY --from=build /app/out . 0.3s
=> [runtime 4/4] RUN echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.conf && echo "fs.inotify.max_user_watches=524288" 0.4s
=> exporting to image 0.4s
=> => exporting layers 0.4s
=> => writing image sha256:4e06252d734fabac03f6d167db6579d68dd9d8cb776faf2a163f59818b94e7c6 0.0s
=> => naming to docker.io/idjohnson/freshmvcapp:1.0 0.0s
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ docker push idjohnson/freshmvcapp:1.0
The push refers to repository [docker.io/idjohnson/freshmvcapp]
49f45979fa76: Pushed
db1268234da5: Pushed
cbd8bd8f6f0e: Layer already exists
698640980ef8: Layer already exists
58fa834ef12a: Layer already exists
855e51907d3e: Layer already exists
ad8af893343b: Layer already exists
2ea3b529cc6c: Layer already exists
e0781bc8667f: Layer already exists
1.0: digest: sha256:f50437c01ee5d299aef95fd4543ac73ddd5db7ad392dfe3ee82fe04ce0b2e0e9 size: 2205
My next step is to test the changes with the updated helm chart.
Here I added a new otel block
That should pull them in as environment variables to the pods
Since this is all new code, I updated the App version and Chart Version in Chart.yaml
apiVersion: v2
name: mvcpatients-chart
description: A Helm chart for deploying the MVC Patients application
version: 0.1.5
appVersion: "1.1"
Then fired a helm update to make it live
$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=isaac-MacBookAir --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=idjohnson/freshmvcapp --set image.tag=1.0 --set imagePullSecrets[0].name=myharborreg --set otel.headers="api-key=xxxxxxxxxxxxxxxxxxxxxxFNRAL" ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sun Aug 11 18:45:03 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 8
TEST SUITE: None
It looks good so far
$ kubectl get pods -n patientsmvc
NAME READY STATUS RESTARTS AGE
mypatientmvc-deployment-bbfdff4c4-f4sw2 1/1 Running 0 5s
mypatientmvc-deployment-bbfdff4c4-rzj2w 1/1 Running 0 5s
mypatientmvc-deployment-bbfdff4c4-dxd5p 1/1 Running 0 5s
One thing I do realize is that I might need to move the Header to a secret as it exposes my key in the Pod definition
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl describe pod mypatientmvc-deployment-bbfdff4c4-f4sw2 -n patientsmvc
Name: mypatientmvc-deployment-bbfdff4c4-f4sw2
Namespace: patientsmvc
Priority: 0
Service Account: default
Node: hp-hp-elitebook-850-g2/192.168.1.57
Start Time: Sun, 11 Aug 2024 18:46:25 -0500
Labels: app=mvcpatients-chart
pod-template-hash=bbfdff4c4
Annotations: <none>
Status: Running
IP: 10.42.2.160
IPs:
IP: 10.42.2.160
Controlled By: ReplicaSet/mypatientmvc-deployment-bbfdff4c4
Containers:
mvcpatients-chart-container:
Container ID: containerd://762ce1890960b96203b974b7021a4cd868341f6fdf14250bd009f515bb2cbbd7
Image: idjohnson/freshmvcapp:1.0
Image ID: docker.io/idjohnson/freshmvcapp@sha256:f50437c01ee5d299aef95fd4543ac73ddd5db7ad392dfe3ee82fe04ce0b2e0e9
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 11 Aug 2024 18:46:26 -0500
Ready: True
Restart Count: 0
Environment:
DB_HOST: isaac-MacBookAir
DB_NAME: patientsdb
DB_USER: patientsuser
DB_PORT: 5432
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.nr-data.net:4318/v1/traces
OTEL_SERVICE_NAME: MvcPatients
OTEL_EXPORTER_OTLP_HEADERS: api-key=xxxxxxxxxxxxxxxxxxxxxxxxxFFNRAL
OTEL_RESOURCE_ATTRIBUTES: app=MvcPatients
DB_PASSWORD: <set to the key 'DB_PASSWORD' in secret 'db-secret'> Optional: false
... snip ...
I’ll be honest. I usually iterate for a few hours - this time it actually worked the first go of it. I can immediately see we have switched from “Services - APM” to “Services - OpenTelemetry”
I’m now seeing spans
However, at least with my tracing so far, I am not seeing the full service map I did when I instrumented with New Relic natively
So basically at this point I can easily see spans
and span details
The lack of Metrics made me realize I neglected to actually add the “AddOtlpExporter” line to the metrics collector
I’ll also add a bit of humour to the landing page before I rebuild.
I’ll then build and push to Dockerhub
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ docker build -t idjohnson/freshmvcapp:1.1 .
[+] Building 13.0s (16/16) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:8.0 0.3s
=> [internal] load metadata for mcr.microsoft.com/dotnet/sdk:8.0 0.2s
=> [runtime 1/4] FROM mcr.microsoft.com/dotnet/aspnet:8.0@sha256:3deda593cf10581cbacfa16a1fbb090353d14beaa65adca4611c7c7a458d66b0 0.0s
=> [build 1/6] FROM mcr.microsoft.com/dotnet/sdk:8.0@sha256:7d0ba26469267b563120456557e38eccef9972cb6b9cfbbd47a50d1218fa7b30 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 60.64kB 0.1s
=> CACHED [build 2/6] WORKDIR /app 0.0s
=> CACHED [build 3/6] COPY *.csproj ./ 0.0s
=> CACHED [build 4/6] RUN dotnet restore 0.0s
=> [build 5/6] COPY . ./ 0.9s
=> [build 6/6] RUN dotnet publish MvcPatients.generated.sln -c Release -o out 9.9s
=> CACHED [runtime 2/4] WORKDIR /app 0.0s
=> [runtime 3/4] COPY --from=build /app/out . 0.4s
=> [runtime 4/4] RUN echo "fs.inotify.max_user_instances=8192" >> /etc/sysctl.conf && echo "fs.inotify.max_user_watches=524288" 0.4s
=> exporting to image 0.4s
=> => exporting layers 0.4s
=> => writing image sha256:157bdd0e38936e0c2e3ed6fe8c2faf802007dcdfabbfe5878e67a8eba2e9496e 0.0s
=> => naming to docker.io/idjohnson/freshmvcapp:1.1 0.0s
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ docker push idjohnson/freshmvcapp:1.1
The push refers to repository [docker.io/idjohnson/freshmvcapp]
72081a048a86: Pushed
c63cfe31d753: Pushed
cbd8bd8f6f0e: Layer already exists
698640980ef8: Layer already exists
58fa834ef12a: Layer already exists
855e51907d3e: Layer already exists
ad8af893343b: Layer already exists
2ea3b529cc6c: Layer already exists
e0781bc8667f: Layer already exists
1.1: digest: sha256:42b7fb6897837607470c89b5a3fbf45c0e4e592aee14fc15d95cf878fc97d6f4 size: 2205
I can then upgrade to use it
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=isaac-MacBookAir --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=idjohnson/freshmvcapp --set image.tag=1.1 --set imagePullSecrets[0].name=myharborreg --set otel.headers="api-key=xxxxxxxxxxxxxxxxxxFFNRAL" ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sun Aug 11 19:19:13 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 2
TEST SUITE: None
I can now see Metrics
But still no logs
I realized I had added the ConsoleExporter but NOT the Otlp one. I then added the missing exporter
I then commented out the Console Exporter as it was being noisy
var otelResources = ResourceBuilder.CreateEmpty()
.AddTelemetrySdk()
.AddEnvironmentVariableDetector();
// Add Metrics for ASP.NET Core and our custom metrics and export to Prometheus
otel.WithMetrics(metrics => metrics
// Metrics provider from OpenTelemetry
.AddAspNetCoreInstrumentation()
.SetResourceBuilder(otelResources)
// Metrics provides by ASP.NET Core in .NET 8
.AddMeter("Microsoft.AspNetCore.Hosting")
.AddMeter("Microsoft.AspNetCore.Server.Kestrel")
.AddOtlpExporter());
// Configure tracing
otel.WithTracing(tracing =>
{
tracing.AddAspNetCoreInstrumentation();
tracing.SetResourceBuilder(otelResources);
tracing.AddOtlpExporter();
// Just for troubleshooting purposes to see if spans are generated and printed out to logs.
//tracing.AddConsoleExporter();
});
// Send Logs
builder.Logging.AddOpenTelemetry(options =>
{
options
.SetResourceBuilder(otelResources)
.AddOtlpExporter();
//.AddConsoleExporter();
});
Here is just a portion of some of the container logs with the Console Exporters turned on:
Resource associated with LogRecord:
service.name: MvcPatients
app: MvcPatients
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.9.0
Activity.TraceId: 2825dfb5e389a32d4912ded3a456605c
Activity.SpanId: 84bf8bde3a32df3b
Activity.TraceFlags: Recorded
Activity.ActivitySourceName: Microsoft.AspNetCore
Activity.DisplayName: GET {controller=Home}/{action=Index}/{id?}
Activity.Kind: Server
Activity.StartTime: 2024-08-12T00:22:56.8425503Z
Activity.Duration: 00:00:00.0452189
Activity.Tags:
server.address: patientsmvc.steeped.space
http.request.method: GET
url.scheme: http
url.path: /Patients
network.protocol.version: 1.1
user_agent.original: Wget/1.20.3 (linux-gnu)
http.route: {controller=Home}/{action=Index}/{id?}
http.response.status_code: 200
Resource associated with Activity:
service.name: MvcPatients
app: MvcPatients
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.9.0
LogRecord.Timestamp: 2024-08-12T00:22:57.9836354Z
LogRecord.TraceId: ce4821edee99cd63d78639a4c708f5e2
LogRecord.SpanId: 5b28aecef39c6c06
LogRecord.TraceFlags: Recorded
LogRecord.CategoryName: Microsoft.EntityFrameworkCore.Database.Command
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
Executed DbCommand (2ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT p."Id", p."CreatedDate", p."DateOfBirth", p."FirstName", p."LastName", p."SocialSecurityNumber"
FROM "Patient" AS p
LogRecord.Severity: Info
LogRecord.SeverityText: Information
LogRecord.Body: Executed DbCommand ({elapsed}ms) [Parameters=[{parameters}], CommandType='{commandType}', CommandTimeout='{commandTimeout}']{newLine}{commandText}
LogRecord.Attributes (Key:Value):
elapsed: 2
parameters:
commandType: Text
commandTimeout: 30
newLine:
commandText: SELECT p."Id", p."CreatedDate", p."DateOfBirth", p."FirstName", p."LastName", p."SocialSecurityNumber"
FROM "Patient" AS p
OriginalFormat (a.k.a Body): Executed DbCommand ({elapsed}ms) [Parameters=[{parameters}], CommandType='{commandType}', CommandTimeout='{commandTimeout}']{newLine}{commandText}
LogRecord.EventId: 20101
LogRecord.EventName: Microsoft.EntityFrameworkCore.Database.Command.CommandExecuted
Resource associated with LogRecord:
service.name: MvcPatients
app: MvcPatients
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.9.0
Activity.TraceId: ce4821edee99cd63d78639a4c708f5e2
Activity.SpanId: 5b28aecef39c6c06
Activity.TraceFlags: Recorded
Activity.ActivitySourceName: Microsoft.AspNetCore
Activity.DisplayName: GET {controller=Home}/{action=Index}/{id?}
Activity.Kind: Server
Activity.StartTime: 2024-08-12T00:22:57.9806093Z
Activity.Duration: 00:00:00.0285474
Activity.Tags:
server.address: patientsmvc.steeped.space
http.request.method: GET
url.scheme: http
url.path: /Patients
network.protocol.version: 1.1
user_agent.original: Wget/1.20.3 (linux-gnu)
http.route: {controller=Home}/{action=Index}/{id?}
http.response.status_code: 200
Resource associated with Activity:
service.name: MvcPatients
app: MvcPatients
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.9.0
Another build and push
$ docker build -t idjohnson/freshmvcapp:1.2 . && docker push idjohnson/freshmvcapp:1.2
Another helm upgrade to use it
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=isaac-MacBookAir --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=idjohnson/freshmvcapp --set image.tag=1.2 --set imagePullSecrets[0].name=myharborreg --set otel.headers="api-key=xxxxxxxxxxxxxxxxxxxFFFFNRA
L" ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sun Aug 11 19:29:33 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 3
TEST SUITE: None
Though still no go.
Now I’m realizing endpoint in the chart is fixed to traces
endpoint: "https://otlp.nr-data.net:4318/v1/traces"
Since it is a value I can override, I’ll do so with a helm set
$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=isaac-MacBookAir --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=idjohnson/freshmvcapp --set image.tag=1.3 --set imagePullSecrets[0].name=myharborreg --set otel.headers="api-key=xxxxxxxxxxxxxxxxxxxxxFFNRAL" --set otel.endpoint="https://otlp.nr-data.net:4318/" ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sun Aug 11 19:47:53 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 5
TEST SUITE: None
Now I can see logs!
And I can see more metrics
Demo:
Disabling
Let’s say we want to ditch the Trace data. Can we set the value to just a localhost:4317?
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=isaac-MacBookAir --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=idjohnson/freshmvcapp --set image.tag=1.3 --set imagePullSecrets[0].name=myharborreg --set otel.headers="" --set otel.endpoint="http://localhost:4318/"
./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Sun Aug 11 19:56:56 2024
NAMESPACE: patientsmvc
STATUS: deployed
REVISION: 6
TEST SUITE: None
It’s not erroring
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl logs mypatientmvc-deployment-dc97c9c5b-dbdvp -n patientsmvc | head -n 20
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
No XML encryptor configured. Key {d4a355f0-5a54-488a-9120-d2430fbdca1e} may be persisted to storage in unencrypted form.
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://[::]:8080
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: /app
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
Failed to determine the https port for redirect.
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
Executed DbCommand (19ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT p."Id", p."CreatedDate", p."DateOfBirth", p."FirstName", p."LastName", p."SocialSecurityNumber"
FROM "Patient" AS p
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
Executed DbCommand (12ms) [Parameters=[@__id_0='?' (DbType = Int32)], CommandType='Text', CommandTimeout='30']
I fired many queries and see it worked without issues
GCP Cloud Trace
First, let’s create a GCP Service account we can use for the OTel Collector:
$ gcloud iam service-accounts create otelcollector \
--display-name "otelcollector Service Account" \
--project myanthosproject2
Created service account [otelcollector].
Next I’ll want to save a JSON key locally
$ gcloud iam service-accounts keys create otelcollector-key.json \
nt otelcollector> --iam-account otelcollector@myanthosproject2.iam.gserviceaccount.com \
> --project myanthosproject2
created key [43f81aec334fd47c0cc0e2e97e1655a89c72c59f] of type [json] as [otelcollector-key.json] for [otelcollector@myanthosproject2.iam.gserviceaccount.com]
Lastly, we need three roles added - LogWriter, CloudTrace.agent and metricWriter
$ gcloud projects add-iam-policy-binding myanthosproject2 \
--member serviceAccount:otelcollector@myanthosproject2.iam.gserviceaccount.com \
--role roles/monitoring.metricWriter
$ gcloud projects add-iam-policy-binding myanthosproject2 \
--member serviceAccount:otelcollector@myanthosproject2.iam.gserviceaccount.com \
--role roles/cloudtrace.agent
$ gcloud projects add-iam-policy-binding myanthosproject2 \
--member serviceAccount:otelcollector@myanthosproject2.iam.gserviceaccount.com \
--role roles/logging.logWriter
I copied the JSON SA key to my docker host, saving in /home/builder/otel/otelcollector-key.json
Next, I’m going to create a containerized instance that can push to GoogleCloud
The OpenTelemetry Config.yaml:
builder@builder-T100:~/otel$ cat config.yaml
receivers:
otlp:
protocols:
grpc:
http:
exporters:
googlecloud:
log:
default_log_name: opentelemetry.io/collector-exported-log
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20
batch:
resourcedetection:
detectors: [gcp]
timeout: 10s
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [googlecloud]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [googlecloud]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [googlecloud]
I can now launch with my DD API Key and path to the Service Account JSON we created
$ docker run -d --name opentelemetry-collector --volume /home/builder/otel/otelcollector-key.json:/etc/otelcol-contrib/key.json --hostname $(hostname) -v /home/builder/otel/config.yaml:/etc/otelcol-contrib/config.yaml --env GOOGLE_APPLICATION_CREDENTIALS=/etc/otelcol-contrib/key.json -p 55681:55681 -p 54318:4318 -p 54317:4317 otel/opentelemetry-collector-contrib
I can see it running
builder@builder-T100:~/otel$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
d20d4a00cb48 otel/opentelemetry-collector-contrib "/otelcol-contrib --…" 19 seconds ago Up 17 seconds 55678-55679/tcp, 0.0.0.0:55681->55681/tcp, :::55681->55681/tcp, 0.0.0.0:54317->4317/tcp, :::54317->4317/tcp, 0.0.0.0:54318->4318/tcp, :::54318->4318/tcp opentelemetry-collector
Over in my MvcPatients app, I can now update to use it
$ helm upgrade mypatientmvc -n patientsmvc --set env.dbHost=isaac-MacBookAir --set env.dbName=patientsdb --set env.dbPort=5432 --set env.dbUser=patientsuser --set env.dbPassword=patientPassword1 --set image.repository=idjohnson/freshmvcapp --set image.tag=1.3 --set imagePullSecrets[0].name=myharborreg --set otel.headers="" --set otel.endpoint="http://192.168.1.100:54318" ./Deployment/chart/
Release "mypatientmvc" has been upgraded. Happy Helming!
NAME: mypatientmvc
LAST DEPLOYED: Mon Aug 12 19:09:49 2024
Here we can see it in action:
Zipkin
Next, let’s install a local Zipkin from their helm chart
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ helm repo add zipkin https://zipkin.io/zipkin-helm
"zipkin" has been added to your repositories
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "adwerx" chart repository
...Successfully got an update from the "zipkin" chart repository
...Successfully got an update from the "confluentinc" chart repository
...Successfully got an update from the "jfelten" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "opencost-charts" chart repository
...Successfully got an update from the "opencost" chart repository
...Successfully got an update from the "rhcharts" chart repository
...Successfully got an update from the "kube-state-metrics" chart repository
...Successfully got an update from the "dapr" chart repository
...Successfully got an update from the "portainer" chart repository
...Successfully got an update from the "bitwarden" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "sonarqube" chart repository
...Successfully got an update from the "lifen-charts" chart repository
...Successfully got an update from the "akomljen-charts" chart repository
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "kubecost" chart repository
...Successfully got an update from the "castai-helm" chart repository
...Successfully got an update from the "longhorn" chart repository
...Successfully got an update from the "uptime-kuma" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "signoz" chart repository
...Unable to get an update from the "epsagon" chart repository (https://helm.epsagon.com):
Get "https://helm.epsagon.com/index.yaml": dial tcp: lookup helm.epsagon.com on 10.255.255.254:53: server misbehaving
...Successfully got an update from the "crossplane-stable" chart repository
...Successfully got an update from the "argo-cd" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "rancher-latest" chart repository
...Successfully got an update from the "gitlab" chart repository
...Unable to get an update from the "myharbor" chart repository (https://harbor.freshbrewed.science/chartrepo/library):
failed to fetch https://harbor.freshbrewed.science/chartrepo/library/index.yaml : 404 Not Found
...Unable to get an update from the "freshbrewed" chart repository (https://harbor.freshbrewed.science/chartrepo/library):
failed to fetch https://harbor.freshbrewed.science/chartrepo/library/index.yaml : 404 Not Found
...Successfully got an update from the "nfs" chart repository
...Successfully got an update from the "ngrok" chart repository
...Successfully got an update from the "zabbix-community" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "backstage" chart repository
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "novum-rgi-helm" chart repository
...Successfully got an update from the "btungut" chart repository
...Successfully got an update from the "openfunction" chart repository
...Successfully got an update from the "kiwigrid" chart repository
...Successfully got an update from the "sumologic" chart repository
...Successfully got an update from the "gitea-charts" chart repository
...Successfully got an update from the "openzipkin" chart repository
...Successfully got an update from the "spacelift" chart repository
...Successfully got an update from the "open-telemetry" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "elastic" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "skm" chart repository
...Successfully got an update from the "openproject" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "ananace-charts" chart repository
...Successfully got an update from the "newrelic" chart repository
...Successfully got an update from the "makeplane" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
I can then install Zipkin
$ helm install myzipkin zipkin/zipkin
NAME: myzipkin
LAST DEPLOYED: Tue Aug 13 06:19:00 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=zipkin,app.kubernetes.io/instance=myzipkin" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
I port-forward to the Zipkin port on the pod
$ kubectl --namespace default port-forward `kubectl get pods --namespace default -l "app.kubernetes.io/name=zipkin,app.kubernetes.io/instance=myzipkin" -o jsonpath="{.items[0].metadata.name}"` 9411:9411
Forwarding from 127.0.0.1:9411 -> 9411
Forwarding from [::1]:9411 -> 9411
And can see a blank, but functional Zipkin running
I now need a LoadBalancer my OTel collector could reach. I’m still using the one for GCP CloudTrace running on a different DockerHost
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ cat zipkin.lb.yaml
apiVersion: v1
kind: Service
metadata:
finalizers:
- service.kubernetes.io/load-balancer-cleanup
name: zipkinlb
spec:
allocateLoadBalancerNodePorts: true
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: PreferDualStack
ports:
- name: myzipkin
port: 9411
protocol: TCP
targetPort: 9411
selector:
app.kubernetes.io/instance: myzipkin
app.kubernetes.io/name: zipkin
sessionAffinity: None
type: LoadBalancer
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ kubectl apply -f ./zipkin.lb.yaml
service/zipkinlb created
Which looks to have pulled NodePort 31822
$ kubectl get svc zipkinlb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
zipkinlb LoadBalancer 10.43.102.173 192.168.1.215,192.168.1.33,192.168.1.36,192.168.1.57 9411:31557/TCP 10s
Which I can see accessible on the NodePort and listen Port
I can now go to the DockerHost which is already running the ‘contrib’ variant and add a block for the zipkin exporter
builder@builder-T100:~/otel$ cat config.yaml
receivers:
otlp:
protocols:
grpc:
http:
exporters:
zipkin/nontls:
endpoint: "http://192.168.1.33:31557/api/v2/spans"
format: proto
googlecloud:
log:
default_log_name: opentelemetry.io/collector-exported-log
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20
batch:
resourcedetection:
detectors: [gcp]
timeout: 10s
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [googlecloud,"zipkin/nontls"]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [googlecloud]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [googlecloud]
I’ll restart it
builder@builder-T100:~/otel$ docker stop 03b83631cfe8
03b83631cfe8
builder@builder-T100:~/otel$ docker rm 03b83631cfe8
03b83631cfe8
builder@builder-T100:~/otel$ !1644
docker run -d --name opentelemetry-collector --volume /home/builder/otel/otelcollector-key.json:/etc/otelcol-contrib/key.json --hostname $(hostname) -v /home/builder/otel/config.yaml:/etc/otelcol-contrib/config.yaml --env GOOGLE_APPLICATION_CREDENTIALS=/etc/otelcol-contrib/key.json -p 55681:55681 -p 54318:4318 -p 54317:4317 otel/opentelemetry-collector-contrib
a4451a0cd6dab53e446c17684804b9d1415ad23d65cdd376654b910e4a34186b
Double check I’m still sending traces to the Docker Otel collector
builder@DESKTOP-QADGF36:~/Workspaces/MvcPatients$ !2293
helm get values mypatientmvc -n patientsmvc
USER-SUPPLIED VALUES:
env:
dbHost: isaac-MacBookAir
dbName: patientsdb
dbPassword: patientPassword1
dbPort: 5432
dbUser: patientsuser
image:
repository: idjohnson/freshmvcapp
tag: "1.3"
imagePullSecrets:
- name: myharborreg
otel:
endpoint: http://192.168.1.100:54318
headers: ""
I can now see data both in Zipkin running in the cluster
and GCP Cloud Trace
Here we can see a demo in action
Summary
Today we tackled quite a bit. We instrumented the .NET 8 MVC app with NewRelic in the very fast Out-of-the-box fashion. This worked great but we wanted to have some more options so we moved to OpenTelemetry but not before showing how to create New Relic Synthetic Checks including Selenium-based Scripted Browser monitors.
I showed how to update the .NET Code to use OpenTelemetry as well as updating the Dockerfile and Helm Charts. We looked at setting up the OpenTelemetry Contrib Collector in Docker to send logs, traces and metrics to GCP including CloudTrace and Log Monitoring. Lastly, we showed adding Zipkin to the stack to see local trace data as sent via OTLP to the Collector and transformed back to Zipkin v2 spans.
Hopefully you found this useful and can inspire you for new and unique ways to gather and measure applications.
I did push the updated App, Charts and Dockerfile to the repo with this commit.
Thanks for reading!
