Fresh/Brewed

Instana: Part 1: Setup, K8s, Website and Open Telemetry

Published: Jan 10, 2023 by Isaac Johnson

Today we will dig into creating an Instana account and setting up Host and Kubernetes monitors. We’ll look into serverless (AWS Lambdas) and tracing through Open Telemetry.

History

Instana was a German/American company created in 2015 having spun off of German consulting firm, Codecentric. Founded by Mirko Novakovic, Pete Abrams, Fabian Lange, and Pavlo Baron, they wanted to focus their “APM solution” on containers and microservice-based architecture. Around the time I became familiar with them in 2018, they had already raised US$15m and had 225 customers. In his 5th Anniversary Blog Post, co-founder Mirko Novakovic summarized the DNA of Instana as “One Agent Per Host, Real-Time Processing Platform with Streaming Data and High Semantic Understanding built into the product”. He summarized the growth thus far:

/content/images/2023/01/instana-190.png

By the time Covid-19 took over the world, Instana was up to 200 employees and a global player. There is a great blog with photos https://the0x00.dev/instana/History-of-Instana/. But it essentially ends with “On December 28, 2020, Instana was acquired by IBM”. We can see that soon after, the then CEO Mirko Novakovic went to be VP of Watson AI for 3 months before leaving IBM. The COO Pete Abrams left within 3 months of acquisition. However, it would seem Fabian Lange stayed on as did Pavlo Baron whose now listed as “Chief Innovation Officer”.

Since being acquired, evidence suggests IBM is continuing to invest in Instana. The crunchbase page shows 101-250 employees and that it’s now based in Chicago. It’s in the 2022 Garnter Magic Quandrant for APM along with Datadog, Dynatrace, New Relic and Honeycomb.

The interesting thing to me is how I’ve seen how companies like IBM (and Oracle as well) acquire large players then radically raise prices. However, from what I personally am aware, the price of Instana has not gone up radically. Perhaps IBM is realizing the consequences of dramatic prices increases (I’m thinking of BuildForge, ClearCase, and other tools that shot up in price and were subsequently abandoned by the market).

Signup

Let’s go to the IBM Instana page and “Start a free trial”.

/content/images/2023/01/instana-01.png

I’ll fill in the details and signup

/content/images/2023/01/instana-02.png

I’ll then select my region and click “Build my Instana instance”

/content/images/2023/01/instana-03.png

While I appreciate the proactive nature of asking my thoughts, sending me 4 emails right away on signup raises a flag

/content/images/2023/01/instana-04.png

Upon logging in, it asks for my email preferences (interestingly “Marketing” was deselected already) then my role. The fonts on this page (or lack thereof) caught my eye.

/content/images/2023/01/instana-05.png

My next page shows the agent launch page. I was really surprised to see my “key” match the URL GET parameter (I masked above).

/content/images/2023/01/instana-06.png

Let’s start with Kubernetes. I have a few options for install type (default is Operator). I’ll use helm

/content/images/2023/01/instana-07.png

$ helm install instana-agent \
   --repo https://agents.instana.io/helm \
   --namespace instana-agent \
   --create-namespace \
   --set agent.key=asdfsafd \
   --set agent.downloadKey=undefined \
   --set agent.endpointHost=ingress-coral-saas.instana.io \
   --set agent.endpointPort=443 \
   --set cluster.name='mac81cluster' \
   --set zone.name='onprem' \
   instana-agent
NAME: instana-agent
LAST DEPLOYED: Thu Dec 29 18:46:30 2022
NAMESPACE: instana-agent
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
It may take a few moments for the agents to fully deploy. You can see what agents are running by listing resources in the instana-agent namespace:

    kubectl get all -n instana-agent

You can get the logs for all of the agents with `kubectl logs`:

    kubectl logs -l app.kubernetes.io/name=instana-agent -n instana-agent -c instana-agent

I can check the deployment status

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get all -n instana-agent
NAME                      READY   STATUS              RESTARTS   AGE
pod/instana-agent-pc9c5   0/2     ContainerCreating   0          24s
pod/instana-agent-rpbbc   0/2     ContainerCreating   0          24s
pod/instana-agent-x7lw5   0/2     ContainerCreating   0          24s

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/instana-agent   3         3         0       3            0           <none>          25s

$ kubectl get all -n instana-agent
NAME                      READY   STATUS    RESTARTS   AGE
pod/instana-agent-pc9c5   2/2     Running   0          2m44s
pod/instana-agent-rpbbc   2/2     Running   0          2m44s
pod/instana-agent-x7lw5   2/2     Running   0          2m44s

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/instana-agent   3         3         3       3            3           <none>          2m45s

About this time, I got my Password set email

/content/images/2023/01/instana-08.png

I set my password and now I have an instance ID

/content/images/2023/01/instana-09.png

However, I see nothing show up… the logs of the agent seem to be caused by 401 unauthorized error on their own maven repo

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl logs instana-agent-pc9c5 -n instana-agent
Defaulted container "instana-agent" out of: instana-agent, leader-elector
Docker socket not found at /var/run/docker.sock
Starting Instana Agent ...
Found 'JAVA_OPTS' settings: '-XX:+ExitOnOutOfMemoryError '.
The amount of memory to be used by the agent is 805306368 bytes (from relocated: /sys/fs/cgroup/memory/memory.limit_in_bytes).
2022-12-30T00:52:46.722+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Bootstrapping agent
2022-12-30T00:52:46.736+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Running on anna-MacBookAir, Linux 5.15.0-46-generic amd64
2022-12-30T00:52:46.737+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Using PATH set to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-30T00:52:46.737+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Using Java 1.8.0_352 OpenJDK 64-Bit Server VM 25.352-b08 (Azul Systems, Inc.) from /opt/instana/agent/jvm
2022-12-30T00:52:46.738+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Using JAVA_OPTS -XX:+ExitOnOutOfMemoryError -Xmx410204K -XX:MaxDirectMemorySize=45578K -XX:+UseSerialGC -Xmn32m -Xss640k -XX:InitialCodeCacheSize=8m -XX:CodeCacheExpansionSize=64k -XX:ReservedCodeCacheSize=16m -XX:-TieredCompilation -XX:CICompilerCount=1 -XX:CompressedClassSpaceSize=24m -XX:MinMetaspaceExpansion=1M -XX:MaxMetaspaceExpansion=8M -XX:MaxMetaspaceSize=128m -XX:MaxDirectMemorySize=128m -XX:+PerfDisableSharedMem -Dcom.ibm.tools.attach.enable=no -Xverify:none -Xms64m -Dcom.ibm.jsse2.overrideDefaultTLS=true -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2,TLSv1.3 -Dlog4j2.formatMsgNoLookups=true -XX:+ExitOnOutOfMemoryError -Djava.endorsed.dirs=/opt/instana/agent/jvm/jre/lib/endorsed:/opt/instana/agent/jvm/lib/endorsed:/opt/instana/agent/lib/endorsed -Djava.ext.dirs=/opt/instana/agent/jvm/jre/lib/ext:/opt/instana/agent/jvm/lib/ext:/opt/instana/agent/lib/ext -Dkaraf.instances=/opt/instana/agent/instances -Dkaraf.home=/opt/instana/agent -Dkaraf.base=/opt/instana/agent -Dkaraf.data=/opt/instana/agent/data -Dkaraf.etc=/opt/instana/agent/etc -Dkaraf.log=/opt/instana/agent/data/log -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/opt/instana/agent/data/tmp -Djava.util.logging.config.file=/opt/instana/agent/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -agentpath:/opt/dynatrace/oneagent/agent/bin/1.255.179.20221213-180126/linux-x86-64/liboneagentloader.so=loglevelcon=none,datastorage=/var/lib/dynatrace/oneagent/datastorage,logdir=/var/log/dynatrace/oneagent
2022-12-30T00:52:46.738+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Agent installed to /opt/instana/agent and started by root
2022-12-30T00:52:46.739+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Dynamic agent running latest version and installed from public_docker
2022-12-30T00:52:46.828+00:00 | INFO  | agent-bootstrap                  | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Installing agent
2022-12-30T00:52:46.832+00:00 | INFO  | features-3-thread-1              | pdateManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Automatic updates scheduled every DAY at 4:30
2022-12-30T00:52:48.511+00:00 | ERROR | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Error when adding repository mvn:com.instana/agent-feature/1.0.0-SNAPSHOT/xml/features
java.lang.RuntimeException: Error resolving artifact com.instana:agent-feature:xml:features:1.0.0-SNAPSHOT: [Could not transfer artifact com.instana:agent-feature:xml:features:1.0.0-SNAPSHOT from/to features (https://artifact-public.instana.io/artifactory/features-public/): authentication failed for https://artifact-public.instana.io/artifactory/features-public/com/instana/agent-feature/1.0.0-SNAPSHOT/agent-feature-1.0.0-SNAPSHOT-features.xml, status: 401 Unauthorized, Could not transfer artifact com.instana:agent-feature:xml:features:1.0.0-SNAPSHOT from/to shared (https://artifact-public.instana.io/artifactory/shared/): authentication failed for https://artifact-public.instana.io/artifactory/shared/com/instana/agent-feature/1.0.0-SNAPSHOT/agent-feature-1.0.0-SNAPSHOT-features.xml, status: 401 Unauthorized] : mvn:com.instana/agent-feature/1.0.0-SNAPSHOT/xml/features
        at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:116) ~[?:?]
        at org.apache.karaf.features.internal.service.RepositoryImpl.<init>(RepositoryImpl.java:50) ~[?:?]
        at org.apache.karaf.features.internal.service.RepositoryCacheImpl.create(RepositoryCacheImpl.java:51) ~[?:?]
        at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:380) ~[?:?]
        at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:375) ~[?:?]
        at com.instana.agent.bootstrap.impl.FeaturesManagerImpl.addRepository(FeaturesManagerImpl.java:56) ~[?:?]
        at com.instana.agent.bootstrap.AgentBootstrap.lambda$activate$0(AgentBootstrap.java:102) ~[?:?]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]
Caused by: java.io.IOException: Error resolving artifact com.instana:agent-feature:xml:features:1.0.0-SNAPSHOT: [Could not transfer artifact com.instana:agent-feature:xml:features:1.0.0-SNAPSHOT from/to features (https://artifact-public.instana.io/artifactory/features-public/): authentication failed for https://artifact-public.instana.io/artifactory/features-public/com/instana/agent-feature/1.0.0-SNAPSHOT/agent-feature-1.0.0-SNAPSHOT-features.xml, status: 401 Unauthorized, Could not transfer artifact com.instana:agent-feature:xml:features:1.0.0-SNAPSHOT from/to shared (https://artifact-public.instana.io/artifactory/shared/): authentication failed for https://artifact-public.instana.io/artifactory/shared/com/instana/agent-feature/1.0.0-SNAPSHOT/agent-feature-1.0.0-SNAPSHOT-features.xml, status: 401 Unauthorized]
        at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.configureIOException(AetherBasedResolver.java:803) ~[?:?]
        at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:774) ~[?:?]
        at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
        at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
        at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
        at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:555) ~[?:?]
        at org.ops4j.pax.url.mvn.internal.Connection.getInputStream(Connection.java:123) ~[?:?]
        at java.net.URL.openStream(URL.java:1093) ~[?:1.8.0_352]

Indeed, they have a maven issue

/content/images/2023/01/instana-10.png

Let me try YAML instead

$ helm delete instana-agent -n instana-agent
release "instana-agent" uninstalled

Then we can down the YAML

/content/images/2023/01/instana-11.png

Which I can apply directly

$ kubectl apply -f /mnt/c/Users/isaac/Downloads/configuration.yaml
Warning: resource namespaces/instana-agent is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/instana-agent configured
serviceaccount/instana-agent created
secret/instana-agent created
configmap/instana-agent created
daemonset.apps/instana-agent created
clusterrole.rbac.authorization.k8s.io/instana-agent created
clusterrolebinding.rbac.authorization.k8s.io/instana-agent created

$ kubectl get pods -n instana-agent
NAME                  READY   STATUS    RESTARTS   AGE
instana-agent-bhxss   2/2     Running   0          42s
instana-agent-dwgvb   2/2     Running   0          42s
instana-agent-hvxd4   2/2     Running   0          42s

We can see this worked just fine

$ kubectl logs instana-agent-bhxss -n instana-agent
Defaulted container "instana-agent" out of: instana-agent, leader-elector
Docker socket not found at /var/run/docker.sock
Starting Instana Agent ...
Found 'JAVA_OPTS' settings: '-XX:+ExitOnOutOfMemoryError '.
The amount of memory to be used by the agent is 805306368 bytes (from relocated: /sys/fs/cgroup/memory/memory.limit_in_bytes).
2022-12-30T00:59:52.128+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Bootstrapping agent
2022-12-30T00:59:52.147+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Running on anna-MacBookAir, Linux 5.15.0-46-generic amd64
2022-12-30T00:59:52.148+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Using PATH set to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-30T00:59:52.148+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Using Java 1.8.0_352 OpenJDK 64-Bit Server VM 25.352-b08 (Azul Systems, Inc.) from /opt/instana/agent/jvm
2022-12-30T00:59:52.149+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Using JAVA_OPTS -XX:+ExitOnOutOfMemoryError -Xmx410204K -XX:MaxDirectMemorySize=45578K -XX:+UseSerialGC -Xmn32m -Xss640k -XX:InitialCodeCacheSize=8m -XX:CodeCacheExpansionSize=64k -XX:ReservedCodeCacheSize=16m -XX:-TieredCompilation -XX:CICompilerCount=1 -XX:CompressedClassSpaceSize=24m -XX:MinMetaspaceExpansion=1M -XX:MaxMetaspaceExpansion=8M -XX:MaxMetaspaceSize=128m -XX:MaxDirectMemorySize=128m -XX:+PerfDisableSharedMem -Dcom.ibm.tools.attach.enable=no -Xverify:none -Xms64m -Dcom.ibm.jsse2.overrideDefaultTLS=true -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2,TLSv1.3 -Dlog4j2.formatMsgNoLookups=true -XX:+ExitOnOutOfMemoryError -Djava.endorsed.dirs=/opt/instana/agent/jvm/jre/lib/endorsed:/opt/instana/agent/jvm/lib/endorsed:/opt/instana/agent/lib/endorsed -Djava.ext.dirs=/opt/instana/agent/jvm/jre/lib/ext:/opt/instana/agent/jvm/lib/ext:/opt/instana/agent/lib/ext -Dkaraf.instances=/opt/instana/agent/instances -Dkaraf.home=/opt/instana/agent -Dkaraf.base=/opt/instana/agent -Dkaraf.data=/opt/instana/agent/data -Dkaraf.etc=/opt/instana/agent/etc -Dkaraf.log=/opt/instana/agent/data/log -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/opt/instana/agent/data/tmp -Djava.util.logging.config.file=/opt/instana/agent/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -agentpath:/opt/dynatrace/oneagent/agent/bin/1.255.179.20221213-180126/linux-x86-64/liboneagentloader.so=loglevelcon=none,datastorage=/var/lib/dynatrace/oneagent/datastorage,logdir=/var/log/dynatrace/oneagent
2022-12-30T00:59:52.149+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Agent installed to /opt/instana/agent and started by root
2022-12-30T00:59:52.150+00:00 | INFO  | features-3-thread-1              | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Dynamic agent running latest version and installed from public_docker
2022-12-30T00:59:52.252+00:00 | INFO  | agent-bootstrap                  | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Installing agent
2022-12-30T00:59:52.256+00:00 | INFO  | features-3-thread-1              | pdateManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Automatic updates scheduled every DAY at 4:30
2022-12-30T01:00:03.169+00:00 | INFO  | features-3-thread-1              | Agent            | com.instana.agent - 1.1.669 | Agent Mode: apm
2022-12-30T01:00:03.461+00:00 | INFO  | features-3-thread-1              | SchedulerImpl    | com.instana.agent - 1.1.669 | Started sensor scheduler with 4 threads on 2 CPUs.
2022-12-30T01:00:04.119+00:00 | INFO  | features-3-thread-1              | ntainerdUtilImpl | com.instana.agent - 1.1.669 | ContainerD detected
2022-12-30T01:00:04.123+00:00 | INFO  | features-3-thread-1              | ntainerdUtilImpl | com.instana.agent - 1.1.669 | Agent is running in containerd container f213d1f88d3975c90fd4d71d8878f97adbc9ea0d6d2dcfcd77a7e0093bec5b45
2022-12-30T01:00:04.125+00:00 | INFO  | features-3-thread-1              | ntainerdUtilImpl | com.instana.agent - 1.1.669 | Discovered container IP 192.168.1.81 which resolves to inet address /192.168.1.81
2022-12-30T01:00:05.636+00:00 | INFO  | features-3-thread-1              | SensorHelperImpl | com.instana.agent - 1.1.669 | started AgentSensorHelperImpl
2022-12-30T01:00:06.143+00:00 | INFO  | instana-http-service             | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on 127.0.0.1:42699
2022-12-30T01:00:06.144+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Started agent http service with 4 threads.
2022-12-30T01:00:06.147+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:470:a6ff:fe58:dd88%veth3d6263bf:42699
2022-12-30T01:00:06.151+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:78cd:3aff:fe92:189c%vethc1d893b6:42699
2022-12-30T01:00:06.152+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:18f4:1eff:fead:2ea8%veth35e711c2:42699
2022-12-30T01:00:06.154+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:8cad:3dff:fe80:144b%veth64f217ee:42699
2022-12-30T01:00:06.155+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:801f:12ff:fed4:701c%veth0a26485d:42699
2022-12-30T01:00:06.156+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on 10.42.0.0:42699
2022-12-30T01:00:06.158+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on 10.42.0.1:42699
2022-12-30T01:00:06.159+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:cb3:14ff:fef0:4dd0%veth5a1d57b0:42699
2022-12-30T01:00:06.163+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:c78:3dff:fe5c:62e5%veth814cc878:42699
2022-12-30T01:00:06.164+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:ac53:bff:fe49:ea2c%veth77e5ad6e:42699
2022-12-30T01:00:06.211+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:4c1f:87ff:fe10:c919%veth670efb43:42699
2022-12-30T01:00:06.213+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:84c5:8eff:fed1:a62e%vethcf809f69:42699
2022-12-30T01:00:06.219+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on 192.168.1.81:42699
2022-12-30T01:00:06.220+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:d447:aeff:febb:973c%vethad442497:42699
2022-12-30T01:00:06.221+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:fca7:9aff:fe16:afe2%vethcdd4cfa9:42699
2022-12-30T01:00:06.225+00:00 | INFO  | features-3-thread-1              | tHttpServiceImpl | com.instana.agent - 1.1.669 | Listening on fe80:0:0:0:a43f:ebff:fec9:8e19%veth4b7dd3a4:42699
2022-12-30T01:00:06.443+00:00 | INFO  | features-3-thread-1              | AgentSensor      | com.instana.agent - 1.1.669 | Activated Sensor
2022-12-30T01:00:06.518+00:00 | INFO  | features-3-thread-1              | Backend          | com.instana.agent - 1.1.669 | Not verifying certificate fingerprint of connection
2022-12-30T01:00:06.567+00:00 | INFO  | features-3-thread-1              | AgentImpl        | com.instana.agent - 1.1.669 | Cleaning heap.
2022-12-30T01:00:06.698+00:00 | INFO  | features-3-thread-1              | AgentImpl        | com.instana.agent - 1.1.669 | Agent configured with repositories https://artifact-public.instana.io/artifactory/features-public@id=features@snapshots@snapshotsUpdate=always,https://artifact-public.instana.io/artifactory/shared@id=shared@snapshots@snapshotsUpdate=never
2022-12-30T01:00:06.708+00:00 | INFO  | features-3-thread-1              | AgentImpl        | com.instana.agent - 1.1.669 | Maven server configured: id: 'features', username: '_', password: 'n8I0U7mlTIqAdoAiwd1Dmw'
2022-12-30T01:00:06.709+00:00 | INFO  | features-3-thread-1              | AgentImpl        | com.instana.agent - 1.1.669 | Maven server configured: id: 'shared', username: '_', password: 'n8I0U7mlTIqAdoAiwd1Dmw'
2022-12-30T01:00:06.710+00:00 | INFO  | features-3-thread-1              | AgentImpl        | com.instana.agent - 1.1.669 | Started agent scheduler.
2022-12-30T01:00:06.711+00:00 | INFO  | features-3-thread-1              | AgentImpl        | com.instana.agent - 1.1.669 | Starting Instana agent.
2022-12-30T01:00:06.720+00:00 | INFO  | features-3-thread-1              | AgentImpl        | com.instana.agent - 1.1.669 | Instana agent started.
2022-12-30T01:00:06.728+00:00 | INFO  | agent-starter                    | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Installing discoveries
2022-12-30T01:00:06.930+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-init/1.0.7
2022-12-30T01:00:06.931+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-netty-anti-leak/1.0.5
2022-12-30T01:00:06.932+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-jctools2/2.1.6
2022-12-30T01:00:06.932+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-api/1.2.118
2022-12-30T01:00:06.933+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-unix-socket-util/1.2.7
2022-12-30T01:00:06.933+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-agent-api/1.0.5
2022-12-30T01:00:06.933+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-agent-optional-api/1.0.5
2022-12-30T01:00:06.934+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-agent-internal-api/1.0.5
2022-12-30T01:00:06.934+00:00 | INFO  | agent-bootstrap                  | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-agent/1.1.669
2022-12-30T01:00:08.095+00:00 | INFO  | ss-coral-saas.instana.io:443-4-1 | irectMemoryGuard | com.instana.agent - 1.1.669 | Initializing DirectMemoryGuard with 1077947/134217728 bytes of direct memory and 81517577 permits
2022-12-30T01:00:08.096+00:00 | INFO  | ss-coral-saas.instana.io:443-4-1 | Backend          | com.instana.agent - 1.1.669 | Connected using HTTP/2 to ingress-coral-saas.instana.io:443 with id '28:cf:e9:ff:fe:4b:10:91' and key 'n8I0U7mlTIqAdoAiwd1Dmw'
2022-12-30T01:00:09.168+00:00 | INFO  | tana-sensor-scheduler-thread-2-2 | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-process-sensor
2022-12-30T01:00:09.229+00:00 | INFO  | c00-c74c-45b0-bb92-6cbfbda93191) | Process          | com.instana.sensor-process - 1.1.46 | Activated Sensor for PID 655084
2022-12-30T01:00:14.834+00:00 | ERROR | tana-global-scheduler-thread-3-4 | chineFactoryImpl | com.instana.agent - 1.1.669 | Get java version for '4114751' failed: timeout calling 'java.lang.UNIXProcess@24d8ed8a'
2022-12-30T01:00:53.833+00:00 | INFO  | features-3-thread-1              | PackageInstaller | com.instana.discovery-python - 1.9.9 | Using pip (/usr/bin/pip3) version: pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
2022-12-30T01:00:55.442+00:00 | INFO  | tana-sensor-scheduler-thread-2-3 | Python           | com.instana.discovery-python - 1.9.9 | Applying Python AutoTrace to process: pid=2248569, name=/daprd, exe=/daprd, args=[--mode, kubernetes, --dapr-http-port, 3500, --dapr-grpc-port, 50001, --dapr-internal-grpc-port, 50002, --dapr-listen-addresses, [::1],127.0.0.1, --dapr-public-port, 3501, --app-port, 5001, --app-id, python-subscriber, --control-plane-address, dapr-api.dapr-system.svc.cluster.local:80, --app-protocol, http, --placement-host-address, dapr-placement-server.dapr-system.svc.cluster.local:50005, --config, --log-level, info, --app-max-concurrency, -1, --sentry-address, dapr-sentry.dapr-system.svc.cluster.local:80, --enable-metrics=true, --metrics-port, 9090, --dapr-http-max-request-size, -1, --dapr-http-read-buffer-size, -1, --dapr-graceful-shutdown-seconds, -1, --enable-api-logging=false, --enable-mtls]
2022-12-30T01:00:55.626+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ace-discovery/1.0.49
2022-12-30T01:00:55.627+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-action-discovery/1.0.2
2022-12-30T01:00:55.628+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-activemq-artemis-discovery/1.0.7
2022-12-30T01:00:55.628+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-activemq-discovery/1.1.13
2022-12-30T01:00:55.628+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-alicloud-discovery/1.0.11
2022-12-30T01:00:55.629+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-apmproxy-discovery/1.0.7
2022-12-30T01:00:55.629+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-azure-discovery/1.2.29
2022-12-30T01:00:55.630+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-aws-discovery/1.0.116
2022-12-30T01:00:55.631+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-cassandra-discovery/1.1.37
2022-12-30T01:00:55.631+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-beeinstana-discovery/1.0.4
2022-12-30T01:00:55.632+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-biztalk-discovery/1.0.5
2022-12-30T01:00:55.632+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ceph-discovery/1.1.3
2022-12-30T01:00:55.633+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-cpp-discovery/1.1.6
2022-12-30T01:00:55.633+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-cockroachdb-discovery/1.0.8
2022-12-30T01:00:55.634+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-consul-discovery/1.0.16
2022-12-30T01:00:55.634+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-containerd-discovery/1.0.25
2022-12-30T01:00:55.635+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-clickhouse-discovery/1.0.23
2022-12-30T01:00:55.635+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-cloudfoundry-discovery/1.0.5
2022-12-30T01:00:55.636+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-clr-discovery/1.1.72
2022-12-30T01:00:55.636+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-couchbase-discovery/1.0.3
2022-12-30T01:00:55.637+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-crowdstrike-falcon-discovery/1.0.3
2022-12-30T01:00:55.637+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-crystal-discovery/1.1.1
2022-12-30T01:00:55.638+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-crio-discovery/1.0.10
2022-12-30T01:00:55.638+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-db2-discovery/1.0.33
2022-12-30T01:00:55.639+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-db2z-discovery/1.0.0
2022-12-30T01:00:55.639+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-docker-discovery/1.1.67
2022-12-30T01:00:55.640+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-domino-discovery/1.0.3
2022-12-30T01:00:55.640+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-dropwizard-discovery/1.1.26
2022-12-30T01:00:55.641+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ebpf-discovery/1.0.23
2022-12-30T01:00:55.641+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ec2-discovery/1.1.10
2022-12-30T01:00:55.642+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-elasticsearch-discovery/1.1.59
2022-12-30T01:00:55.642+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-envoy-discovery/1.0.2
2022-12-30T01:00:55.643+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-etcd-discovery/1.0.27
2022-12-30T01:00:55.643+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-f5-discovery/1.0.1
2022-12-30T01:00:55.644+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-finagle-discovery/1.1.0
2022-12-30T01:00:55.644+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-garden-discovery/1.0.5
2022-12-30T01:00:55.645+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-gce-discovery/1.1.25
2022-12-30T01:00:55.645+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-gcp-compute-engine-discovery/1.2.5
2022-12-30T01:00:55.646+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-gcp-discovery/1.2.5
2022-12-30T01:00:55.646+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-generic-hardware-discovery/1.1.6
2022-12-30T01:00:55.646+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-glassfish-discovery/1.1.10
2022-12-30T01:00:55.647+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-golang-discovery/1.3.4
2022-12-30T01:00:55.647+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-hadoop-yarn-discovery/1.1.4
2022-12-30T01:00:55.647+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-hadoop-yarn-node-discovery/1.1.1
2022-12-30T01:00:55.648+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-haproxy-discovery/1.1.32
2022-12-30T01:00:55.648+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-haskell-discovery/1.0.3
2022-12-30T01:00:55.648+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-hazelcast-discovery/1.3.2
2022-12-30T01:00:55.649+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-hbase-discovery/1.1.0
2022-12-30T01:00:55.649+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-host-discovery/1.1.147
2022-12-30T01:00:55.649+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-httpd-discovery/1.2.7
2022-12-30T01:00:55.650+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ibmiseries-discovery/1.0.21
2022-12-30T01:00:55.710+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ibmmq-discovery/1.0.40
2022-12-30T01:00:55.711+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ibmmq-mft-discovery/1.0.5
2022-12-30T01:00:55.712+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ibmdatapower-discovery/1.0.12
2022-12-30T01:00:55.714+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ibmopenstack-discovery/1.0.3
2022-12-30T01:00:55.715+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-instana-backend-discovery/1.1.1
2022-12-30T01:00:55.719+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ibmcloud-discovery/1.0.11
2022-12-30T01:00:55.721+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-java-discovery/1.2.121
2022-12-30T01:00:55.721+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-jboss-as-discovery/1.1.47
2022-12-30T01:00:55.722+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-jboss-as4-discovery/1.0.8
2022-12-30T01:00:55.722+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-jboss-datagrid-discovery/1.1.12
2022-12-30T01:00:55.723+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-jenkins-discovery/1.0.18
2022-12-30T01:00:55.723+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-jetty-discovery/1.1.13
2022-12-30T01:00:55.724+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-jira-discovery/1.1.12
2022-12-30T01:00:55.724+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-kafka-connect-discovery/1.0.3
2022-12-30T01:00:55.725+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-kafka-discovery/1.1.48
2022-12-30T01:00:55.725+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-kubernetes-discovery/1.2.139
2022-12-30T01:00:55.726+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-liferay-discovery/1.0.1
2022-12-30T01:00:55.726+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-lxc-discovery/1.0.1
2022-12-30T01:00:55.727+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-mariadb-discovery/1.1.23
2022-12-30T01:00:55.727+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-memcached-discovery/1.0.19
2022-12-30T01:00:55.728+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-mongodb-discovery/1.1.51
2022-12-30T01:00:55.728+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-mysql-discovery/1.1.72
2022-12-30T01:00:55.729+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-msiis-discovery/1.1.9
2022-12-30T01:00:55.730+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-mule-discovery/1.0.5
2022-12-30T01:00:55.730+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-mssql-discovery/1.1.33
2022-12-30T01:00:55.731+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-neo4j-discovery/1.0.0
2022-12-30T01:00:55.731+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-netcore-discovery/1.0.37
2022-12-30T01:00:55.732+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-nginx-discovery/1.1.79
2022-12-30T01:00:55.732+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-nodejs-discovery/1.2.18
2022-12-30T01:00:55.733+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-nomad-discovery/1.0.8
2022-12-30T01:00:55.733+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-opc-discovery/1.1.1
2022-12-30T01:00:55.734+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-openstack-nova-discovery/1.1.6
2022-12-30T01:00:55.734+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-openldap-discovery/1.0.9
2022-12-30T01:00:55.735+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-oracledb-discovery/1.1.24
2022-12-30T01:00:55.735+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-perfcounters-discovery/1.0.0
2022-12-30T01:00:55.736+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-packet-discovery/1.1.1
2022-12-30T01:00:55.736+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-php-cgi-discovery/1.0.0
2022-12-30T01:00:55.737+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-php-cli-discovery/1.0.1
2022-12-30T01:00:55.737+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-php-fpm-discovery/1.1.67
2022-12-30T01:00:55.738+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-php-discovery/1.1.76
2022-12-30T01:00:55.738+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ping-discovery/1.0.10
2022-12-30T01:00:55.739+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ping-directory-discovery/1.0.6
2022-12-30T01:00:55.739+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-postgresql-discovery/1.0.44
2022-12-30T01:00:55.740+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-power-hmc-discovery/1.0.8
2022-12-30T01:00:55.740+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-process-discovery/1.1.46
2022-12-30T01:00:55.741+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-prometheus-discovery/1.0.24
2022-12-30T01:00:55.741+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-python-discovery/1.9.9
2022-12-30T01:00:55.742+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-redis-discovery/1.1.42
2022-12-30T01:00:55.742+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-redis-enterprise-discovery/1.0.1
2022-12-30T01:00:55.743+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-rabbitmq-discovery/1.1.47
2022-12-30T01:00:55.743+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-rocketmq-discovery/1.0.8
2022-12-30T01:00:55.744+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-ruby-discovery/1.6.4
2022-12-30T01:00:55.744+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-sap-hana-discovery/1.1.3
2022-12-30T01:00:55.745+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-solr-discovery/1.0.14
2022-12-30T01:00:55.745+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-statsd-discovery/1.1.9
2022-12-30T01:00:55.746+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-spark-application-discovery/1.1.6
2022-12-30T01:00:55.746+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-spark-standalone-discovery/1.1.2
2022-12-30T01:00:55.747+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-springboot-discovery/1.1.42
2022-12-30T01:00:55.747+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-sybase-discovery/1.0.0
2022-12-30T01:00:55.748+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-tanzu-foundation-member-discovery/1.0.0
2022-12-30T01:00:55.748+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-tomcat-discovery/1.1.32
2022-12-30T01:00:55.748+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-tibcoems-discovery/1.0.8
2022-12-30T01:00:55.749+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-traefik-discovery/1.0.1
2022-12-30T01:00:55.750+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-varnish-discovery/1.0.23
2022-12-30T01:00:55.810+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-vault-discovery/1.0.8
2022-12-30T01:00:55.812+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-vsphere-discovery/1.0.24
2022-12-30T01:00:55.814+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-websphere-discovery/1.1.31
2022-12-30T01:00:55.816+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-websphere-liberty-discovery/1.1.2
2022-12-30T01:00:55.817+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-zhmc-discovery/1.0.10
2022-12-30T01:00:55.818+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-zookeeper-discovery/1.0.7
2022-12-30T01:00:55.819+00:00 | INFO  | agent-starter                    | turesManagerImpl | com.instana.agent-bootstrap - 1.2.25 | Installed instana-weblogic-discovery/1.1.18
2022-12-30T01:00:55.825+00:00 | INFO  | agent-starter                    | AgentBootstrap   | com.instana.agent-bootstrap - 1.2.25 | Starting agent now
2022-12-30T01:00:55.826+00:00 | INFO  | agent-starter                    | AgentImpl        | com.instana.agent - 1.1.669 | Starting Instana agent Discovery.
2022-12-30T01:00:55.827+00:00 | INFO  | agent-starter                    | AgentImpl        | com.instana.agent - 1.1.669 | Instana agent Discovery started.
2022-12-30T01:01:13.922+00:00 | INFO  | tana-sensor-scheduler-thread-2-4 | Ebpf             | com.instana.discovery-ebpf - 1.0.23 | Successfully started eBPF sensor.

and now I see results in Instana

/content/images/2023/01/instana-12.png

Initially I see no data

/content/images/2023/01/instana-13.png

Even 6 hours later I didn’t see services show anything (even having banged on a calculator app and Azure vote app)

/content/images/2023/01/instana-14.png

Let’s check into straight Kubernetes monitoring

Kubernetes

If we go to the Kubernetes area, we’ll see any clusters we added

/content/images/2023/01/instana-15.png

I can actually see quite a few things, from Nodes to Deployments to resource consumption from the main page

/content/images/2023/01/instana-16.png

Nodes

Nodes shows me an overview of my cluster nodes

/content/images/2023/01/instana-17.png

I can dig in to any given node

/content/images/2023/01/instana-18.png

This includes labels

/content/images/2023/01/instana-19.png

as well as annotations

/content/images/2023/01/instana-20.png

If I go to the stack menu, I can quickly see what workloads are associated to this node. I find this quite helpful if I suspect one of my cluster nodes is acting up and needs a reboot.

/content/images/2023/01/instana-21.png

Another way we can look at our cluster is by the workload. If we go to Deployments, we can see all the deployments on this cluster

Deployments

/content/images/2023/01/instana-22.png

I can look at things at the deployment level. For instance, I can check the Calculator app

/content/images/2023/01/instana-23.png

I can get specific details like lables, annotations or the full spec

/content/images/2023/01/instana-24.png

as well as events and conditions which is handy to see if the deployment has had scaling events lately

/content/images/2023/01/instana-25.png

Services

Say I wish to examine the service, I can click on the ‘calculator-front-end’ service this time and see the IP and other details

/content/images/2023/01/instana-26.png

It seems silly, but most of the time I screw something up, it’s not properly setting the service to find the pod behind it. We can easily see the pod(s) associated to this service under pods

/content/images/2023/01/instana-27.png

And again, details on the pod

/content/images/2023/01/instana-28.png

I know I’ve been hitting my services in the last hour. So I was suprised to see no data collected on calls in the last hour

/content/images/2023/01/instana-29.png

In fact, if I clear all filters and say to show all call types, it still shows no data

/content/images/2023/01/instana-30.png

Logs

I’m just going to take a quick moment to say that Instana quite deliberately does not do logging. Instead, they will integrate with 5 (presently) external logging services: Coralogix, ELK, Humio (Crowstrike Falcon LogScale), LogDNA (Mezmo), and Splunk.

/content/images/2023/01/instana-31.png

While 4 are commercial offerings, one can always stand up an internal ELK stack

/content/images/2023/01/instana-32.png

Website and Mobile Apps

In Dynatrace, for instance, we call this “Agentless Monitors”

We can add a website to monitor

/content/images/2023/01/instana-33.png

It then wants us to add some scripting to the HEAD block of our site

/content/images/2023/01/instana-34.png

While I couldn’t actually modify the guts of the harbor front end service on the fly, I was able to update my ‘test’ website back end and poke that a few times

/content/images/2023/01/instana-35.png

I thought, let’s try tracking the primary site for a few hours

/content/images/2023/01/instana-36.png

I quickly added in the block by hand

$ aws s3 cp s3://mys3freshbrewed.science/index.html ./index-prod.html
download: s3://mys3freshbrewed.science/index.html to ./index-prod.html
$ vi index-prod.html
$ aws s3 cp ./index-prod.html s3://mys3freshbrewed.science/index.html --acl public-read
upload: ./index-prod.html to s3://mys3freshbrewed.science/index.html

I didn’t immediately see results because I front my site with a CDN.

I just need to create a quick invalidation to make index live

$ aws cloudfront create-invalidation --distribution-id E3U2HCN2ZRTBZN --paths "/index.html"
{
    "Location": "https://cloudfront.amazonaws.com/2019-03-26/distribution/E3U2HCN2ZRTBZN/invalidation/I1XOXOX3R0K4B8",
    "Invalidation": {
        "Id": "I1XOXOX3R0K4B8",
        "Status": "InProgress",
        "CreateTime": "2022-12-30T13:23:55.759Z",
        "InvalidationBatch": {
            "Paths": {
                "Quantity": 1,
                "Items": [
                    "/index.html"
                ]
            },
            "CallerReference": "cli-1672406633-934184"
        }
    }
}

I can see details on page load times

/content/images/2023/01/instana-37.png

Resources gathered

/content/images/2023/01/instana-38.png

and in settings, I can adjust the privacy settings. By default it masks IPv4 and IPv6

/content/images/2023/01/instana-39.png

This would clearly sort out my ‘front door’, but not tracking the various sub pages. Most of my traffic comes through searches to posts I’ve written in the past.

A more scalable way would be to add to the head.html template in jekyll _includes

/content/images/2023/01/instana-40.png

AWS Lambda

We will simply change our last deployment cloud formation to switch from Dynatrace

Resources:
  LambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      FunctionName: myseconddynafunction
      Handler: index.handler
      Runtime: nodejs14.x
      Role: !GetAtt LambdaFunctionRole.Arn
      Layers:
        - arn:aws:lambda:us-east-1:725887861453:layer:Dynatrace_OneAgent_1_255_2_20221205-113810_nodejs:1
      MemorySize: 1024
      Environment:
        Variables:
          AWS_LAMBDA_EXEC_WRAPPER: /opt/dynatrace # Use the wrapper from the layer
          DT_TENANT: kaz10218
          DT_CLUSTER_ID: 21111114
          DT_CONNECTION_BASE_URL: https://kaz10218.live.dynatrace.com
          DT_CONNECTION_AUTH_TOKEN: dtasdfasdfadfasdfasdfasdfsadfasdfasdf
          DT_OPEN_TELEMETRY_ENABLE_INTEGRATION: true
      Code:
        ZipFile: |
          exports.handler = async (event) => {
            return "Hello World!";
          }
  LambdaFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: AppendToLogsPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Resource: "*"

To Instana

Resources:
  LambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      FunctionName: myseconddynafunction
      Handler: instana-aws-lambda-auto-wrap.handler
      Runtime: nodejs14.x
      Role: !GetAtt LambdaFunctionRole.Arn
      Layers:
        - arn:aws:lambda:us-east-1:410797082306:layer:instana-nodejs:126
      MemorySize: 1024
      Environment:
        Variables:
          INSTANA_ENDPOINT_URL: https://serverless-coral-saas.instana.io
          INSTANA_AGENT_KEY: asdfadfasdfasdfasdf
      Code:
        ZipFile: |
          exports.handler = async (event) => {
            return "Hello World!";
          }
  LambdaFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: AppendToLogsPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Resource: "*"

After updating in the AWS CFN designer, I’ll deploy

/content/images/2023/01/instana-41.png

and when it updates

/content/images/2023/01/instana-42.png

Now when I go to the Lambda function to test

/content/images/2023/01/instana-43.png

I can see execution results that show it updated Instana

/content/images/2023/01/instana-44.png

What is rather fun is we can turn on Services in Instana to “Live” and watch our invokations show up

With the Lambda calls we can dig into any give call to see specifics

/content/images/2023/01/instana-46.png

And each call has trace details

/content/images/2023/01/instana-47.png

IsoMetric view

I love a good Isometric view. I’ve used Datadog Cloudcraft.co for a long time.

We can view our Infrastructure in an isometric view as well

/content/images/2023/01/instana-48.png

Including looking at layer details

/content/images/2023/01/instana-49.png

Open Telemetry (OTel)

For the OpenTelemetry collector example - which is provided by the local agent, not a SaaS endpoint.

There are two easy ways to handle this. First, one could set it as a variable in the helm deployment to k8s

helm install --create-namespace instana-agent --namespace instana-agent \
    --repo https://agents.instana.io/helm \
    --set agent.key='<your agent key>' \
    --set agent.endpointHost='<your host agent endpoint>' \
    --set cluster.name='<your-cluster-name>' \
    --set zone.name='<your-zone-name>' \
    --set opentelemetry.enabled=true \ # this is the setting
    instana-agent

The other is we could update the configuration YAML that is actually stored in the Instana configmap

From our configuration.yaml we used to install. Here I added it to the top of the YAML file:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: instana-agent
  namespace: instana-agent
  labels:
    app.kubernetes.io/name: instana-agent
    app.kubernetes.io/version: 1.2.48
data:
  cluster_name: "mac81k3s"
  configuration.yaml: |
    com.instana.plugin.opentelemetry:
      enabled: true
    # Manual a-priori configuration. Configuration will be only used when the sensor
    # is actually installed by the agent.

I’ll then update

$ vi /mnt/c/Users/isaac/Downloads/configuration.yaml
$ kubectl apply -f /mnt/c/Users/isaac/Downloads/configuration.yaml
namespace/instana-agent unchanged
serviceaccount/instana-agent unchanged
secret/instana-agent unchanged
configmap/instana-agent configured
daemonset.apps/instana-agent configured
clusterrole.rbac.authorization.k8s.io/instana-agent unchanged
clusterrolebinding.rbac.authorization.k8s.io/instana-agent unchanged

Just updating a CM won’t rotate pods

$ kubectl get pods -l app.kubernetes.io/name=instana-agent -n instana-agent
NAME                  READY   STATUS    RESTARTS   AGE
instana-agent-bhxss   2/2     Running   0          13h
instana-agent-dwgvb   2/2     Running   0          13h
instana-agent-hvxd4   2/2     Running   0          13h

We can do that now

$ kubectl delete pods -l app.kubernetes.io/name=instana-agent -n instana-agent
pod "instana-agent-bhxss" deleted
pod "instana-agent-dwgvb" deleted
pod "instana-agent-hvxd4" deleted
$ kubectl get pods -l app.kubernetes.io/name=instana-agent -n instana-agent
NAME                  READY   STATUS    RESTARTS   AGE
instana-agent-22zws   2/2     Running   0          15s
instana-agent-xbjzb   2/2     Running   0          15s
instana-agent-ph28w   2/2     Running   0          14s

I checked the pods.. They don’t seem to be setup to listen on port 4317 for GRPC OTLP calls

The logs, however, look like they are listening

$ kubectl logs instana-agent-22zws -n instana-agent | grep 4317
Defaulted container "instana-agent" out of: instana-agent, leader-elector
2022-12-30T14:33:02.444+00:00 | INFO  | instana-grpc-service             | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on 127.0.0.1, ports 55680,4317
2022-12-30T14:33:02.450+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:470:a6ff:fe58:dd88%veth3d6263bf:4317
2022-12-30T14:33:02.455+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:78cd:3aff:fe92:189c%vethc1d893b6:4317
2022-12-30T14:33:02.459+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:18f4:1eff:fead:2ea8%veth35e711c2:4317
2022-12-30T14:33:02.462+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:8cad:3dff:fe80:144b%veth64f217ee:4317
2022-12-30T14:33:02.469+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:801f:12ff:fed4:701c%veth0a26485d:4317
2022-12-30T14:33:02.474+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on 10.42.0.0:4317
2022-12-30T14:33:02.477+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on 10.42.0.1:4317
2022-12-30T14:33:02.510+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:cb3:14ff:fef0:4dd0%veth5a1d57b0:4317
2022-12-30T14:33:02.513+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:c78:3dff:fe5c:62e5%veth814cc878:4317
2022-12-30T14:33:02.516+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:ac53:bff:fe49:ea2c%veth77e5ad6e:4317
2022-12-30T14:33:02.518+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:4c1f:87ff:fe10:c919%veth670efb43:4317
2022-12-30T14:33:02.521+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:84c5:8eff:fed1:a62e%vethcf809f69:4317
2022-12-30T14:33:02.523+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on 192.168.1.81:4317
2022-12-30T14:33:02.525+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:d447:aeff:febb:973c%vethad442497:4317
2022-12-30T14:33:02.527+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:fca7:9aff:fe16:afe2%vethcdd4cfa9:4317
2022-12-30T14:33:02.529+00:00 | INFO  | features-3-thread-1              | tGrpcServiceImpl | com.instana.agent-grpc - 1.0.11 | Listening on fe80:0:0:0:a43f:ebff:fec9:8e19%veth4b7dd3a4:4317

My first path will be to route the appconfig used via Dapr to send traces DIRECTLY to one of the Instana agent pods

$ diff appconfig.yaml appconfig.yaml.bak
18,21c18,19
<     otel:
<       endpointAddress: "192.168.1.81:4317"
<       isSecure: false
<       protocol: grpc
---
>     zipkin:
>       endpointAddress: http://my-opentelemetry-collector.default.svc.cluster.local:9411
$ kubectl apply -f appconfig.yaml
configuration.dapr.io/appconfig configured

I rotated all the pods and tried the calculator app a few times

It was not instant, mind you, but in a minute or so I started to see all the trace data come flooding in to Instana

/content/images/2023/01/instana-50.png

Picking a trace, I can see details including the fact it was ingested via OTel

/content/images/2023/01/instana-51.png

If I pivot over to services, I can now see more details gathered on the service invokation

/content/images/2023/01/instana-52.png

All the services pulled in via OpenTelemetry were tagged as such. We can use the filter to view all OTel based services

/content/images/2023/01/instana-53.png

However, I continue to not see the Calculator frontend and without it, I can’t see a proper service map between the various services.

I tried updating the config

/content/images/2023/01/instana-54.png

Prod

I pushed the update to head and then created a CloudFront invalidation for all the index files

invalidations.json

{
    "Paths": {
        "Quantity": 258,
        "Items": [
            "/page2/index.html",
            "/page34/index.html",
            "/writings/ds-humio-wlb-draft1/index.html",
            "/writings/ds-humio-draft1/index.html",
            "/page24/index.html",
            "/page12/index.html",
            ...

Which I’ll then use

$ aws cloudfront create-invalidation --distribution-id E3U2HCN2ZRTBZN --invalidation-batch file://invalidations.json
{
    "Location": "https://cloudfront.amazonaws.com/2019-03-26/distribution/E3U2HCN2ZRTBZN/invalidation/I34ZGDWPFWSLU9",
    "Invalidation": {
        "Id": "I34ZGDWPFWSLU9",
        "Status": "InProgress",
        "CreateTime": "2022-12-31T16:07:47.881Z",
        "InvalidationBatch": {
            "Paths": {
                "Quantity": 258,
                "Items": [
                    "/2022/03/08/harbor-backup.html",
                    "/2020/01/26/self-hosted-container-registry-in-kubernetes.html",
                    "/page8/index.html",
                    "/2022/07/12/awx-in-k8s.html",
                    "/2021/04/09/dapr-part-3-digging-into-secrets.html",
                    "/2019/04/07/automating-aks-deployments-like-a-boss-part-4.html",
                    ...

And I quickly saw some results

/content/images/2023/01/instana-55.png

We can look at an Origin and see stats related to the loading of that site

/content/images/2023/01/instana-56.png

Tracking for a week, it was interesting to see anytime the page loads were too high right from the dashboard:

/content/images/2023/01/instana-191.png

While the dashboard doesn’t show why, we can view a result to get an idea of what pages are being loaded

/content/images/2023/01/instana-192.png

I can use Analytics to sort by “Beacon Count” to see what is getting the most views in a given week

/content/images/2023/01/instana-193.png

What is really interesting with the above (and I feel okay sharing since it’s their tool), is clearly the draft article was passed around IBM since I see the 3rd highest viewed in a week was the draft article with whom I only shared with the Instana folks.

I can use the filters to see that it was a mix of Mac OS, Windows, iOS (That can’t be me) and over US, Hong Kong, UK and Spain.

/content/images/2023/01/instana-194.png

I hope this resonates with the power of the “Website” monitoring. It really can nail down who is visiting what and when. I can get a lot of data from AWS CloudFront - but it’s hard to filter bots from users.

I know people pull from the Ghost-based original articles (which I never removed), so all I can aggregate on is the last two years of Jekyll. But I see from AWS, that about 19% of my traffic in a week comes from bots

/content/images/2023/01/instana-195.png

That said, we can see Instana shows a week of regular user viewers of the current site to be around 511. Whereas AWS would suggest a bit higher (around 6k requests a day)

/content/images/2023/01/instana-196.png

I’m sure it’s in the middle - users without javascript enabled, or blockers as well as those going through translate sites. I certainly know what I pay for by way of data usage.

/content/images/2023/01/instana-197.png

By the time you read this, I’ll have disabled the Instana collector on the website, but it was interested to see a time window of details.

Summary

/content/images/2023/01/instana-198.png

This week we covered the history and setup of Instana. We looked at Kubernetes monitoring and spoke to logs. We followed up with Website and Mobile apps including monitoring a production site fronted through a CDN (This one actually). We looked at hosts and wrapped by digging into Trace data using OpenTelemetry.

Next week we’ll cover GCP Services including Cloud Run, Pub/Sub, Storage and Cloud SQL. We’ll look at onprem Linux, Windows and PostgreSQL (On-Prem) observability. We cover monitoring including integration with MS Teams and PagerDuty. We’ll really dig into alerting and escalation showing a demo of cloud-based services and outages.

instana ibm

Have something to add? Feedback? You can use the feedback form

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes