isItObservable/tracing_eda

Is it Observable

Episode : Building Traces for Event Driven Architecture

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to Traces for Event Driven Architecture.

using Solace PubSub:

What you will learn :

• How to use Span Links

This repository showcase the usage of OpenTelemetry with :

• The modified version of Solace trace tutorial
• The OpenTelemetry Operator
• Dynatrace
• Solace

We will send all Telemetry data produced from python application to Dynatrace. We will have 1 producer sending messages to Solace Pub/Sub and 2 consumers.

Let's see 2 way of tracing this type of application.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

Deployment Steps in GCP

You will first need a Kubernetes cluster with 2 Nodes. You can either deploy on Minikube or K3s or follow the instructions to create GKE cluster:

1.Create a Google Cloud Platform Project

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
    cloudtrace.googleapis.com \
    clouddebugger.googleapis.com \
    cloudprofiler.googleapis.com \
    --project ${PROJECT_ID}

2.Create a GKE cluster

ZONE=europe-west3-a
NAME=isitobservable-eda
gcloud container clusters create "${NAME}" --zone ${ZONE} --machine-type=e2-standard-2 --num-nodes=2

Getting started

Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dyntrace tenant , then i suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

Create a Dynatrace token with the following scope ( left menu Acces Token):

• ingest metrics
• ingest OpenTelemetry traces
• ingest logs

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

Solace Service

1. Solace Service - start a trial

If you don't have any Solace account , then i suggest to create a trial using the following link : Solace Account Once you have your Solace Accout, login and go to Cluster Manager

Click on Create a new Service, and specify :

• the name
• the Cloud location ( in my case i'm using the same reagion as my GKE cluster)

And click on Create Service.

Once the service is created click , on the connect tab and expand Solace Messaging to get the Connection details required for this tutorial :

SOLACE_HOST=<YOUR Solace Host>
SOLACE_VPN=<YOUR Solace VPN>
SOLACE_USERNAME=<YOUR solace Username>
SOLACE_PASSWORD=<Your Solace Password>

2. Download the trusted store

From the Connect tab, download the TrustStore ( .pem) and name it: DigiCertGlobalRootCA.crt.pem

Make sure to copy/paste the certificate in the following folders:

• src/consumer_rest
• src/consumer_database
• src/publisher

3. Create your docker containers

Publisher

Create a account on dockerhub

DOCKER_HUB_ACCOUNTNAME= <YOUR ACCOUNTNAME>

run the command :

cd src/publisher
docker build . -t $DOCKER_HUB_ACCOUNTNAME/soloacedemo-publisher:0.1

once the container created, publish it in docker hub

docker push $DOCKER_HUB_ACCOUNTNAME/soloacedemo-publisher:0.1

Consumer Rest

in /src/consumer_rest, modify the followin line : COPY ./solace_telemetry_consumer_REST.py ./solace_telemetry_consumer_REST.py by : COPY ./solace_telemetry_consumer_REST_without_span_links.py ./solace_telemetry_consumer_REST.py

then build the container :

docker build . -t $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-rest:0.1

once the container created, publish it in docker hub

docker push $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-rest:0.1

Consumer Database

in /src/consumer_rest, modify the followin line : COPY ./solace_telemetry_consumer_Database.py ./solace_telemetry_consumer_Database.py by : COPY ./solace_telemetry_consumer_Database_without_spanlinks.py ./solace_telemetry_consumer_Database.py

then build the container :

docker build . -t $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-database:0.1

once the container created, publish it in docker hub

docker push $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-database:0.1

Modify the deployment file of the demo application to use your containers

Modify Manifest/deployment_publisher.yaml Replace the various Deployment and CronJob using the image pointing to my docker hub account: hrexed replace it with your own docker hub.

3.Clone the Github Repository

https://github.com/isItObservable/tracing_eda
cd tracing_eda

Copy your certificate DigiCertGlobalRootCA.crt.pem in:

• src/consumer_database
• src/consumer_rest
• src/publisher

5.Configure the Telemetry Profile in Solace

Create the Telemtry Profile

Within the default message VPN, navigate to Telemetry and select Create Telemetry Profile

Name the Telemetry profile and click Apply. We'll come back to this page later to update additional settings.

Next, we need to enable the receiver. From the trace Telemetry Profile page, select Receiver Connect ACLs and update the Client Connect Default Action to AllowTip: Double-click the input to enable edit mode

After applying the ACL, edit the trace Telemetry Profile page to enable the Reciever and Trace settings.

Finally, let's create a Trace Filter and add a subscription that will attract all topic messages (using the > subscription)

Create the filter with name default. Be sure to enable before clicking Apply.

Add the > subscription

Create OpenTelmetry Collector Client UserName

Within the default message VPN. Navigate to Access Control -> Client Usernames and add a new Client Username.

Create the new client username with a name of trace. Apply the following settings to the trace client username:

• Enable the client username
• Change the password to trace
• Assign #telemetry-trace for both the Client Profile and ACL Profile

Note your telemetry user & password :

SOLACE_TELEMETRY_USER=<YOUR SOLACE telemetry user>
SOLACE_TELEMETRY_PASSWORD=<Your Solace telemetry password>

Get the telemetry queue name

In the Solace, click on Queues. You should see a queue create from our Telemetry Profile.

Note the name of your telemetry queue

SOLACE_TELMETRY_QUEUE=<YOUR SOLACE TElemetry queue name>

Get AMQP url of our telemetry queue

In cluster management UI of Solace, click on "Connect", expend the AMQP section. It will give us the connection details using the amqp protocol:

Note the Url without amqp://

SOLACE_TELEMETRY_AMQP_URL=<YOUR Solace AMQP URL>

4.Deploy most of the components

The application will deploy the openTelemtry Solace Tutorial :

chmod 777 deployment.sh
./deployment.sh  --solaceusername "${SOLACE_USERNAME}" --dthost "${DT_TENANT_URL}" --dttoken "${DATA_INGEST_TOKEN}" --solacepassword "${SOLACE_PASSWORD}" --solacehost "${SOLACE_HOST}"  --solacevpn "${SOLACE_VPN}" --solaceamqpurl "${SOLACE_TELEMETRY_AMQP_URL}" --solacetelemetryqueue "${SOLACE_TELMETRY_QUEUE}" --solacetelemetryuser "${SOLACE_TELEMETRY_USER}" --solacetelemetrypwd "${SOLACE_TELEMETRY_PASSWORD}"

5. Let's modify the application to use Span Links

Consumer Rest

in /src/consumer_rest, modify the followin line : COPY ./solace_telemetry_consumer_REST_without_span_links.py ./solace_telemetry_consumer_REST.py by : COPY ./solace_telemetry_consumer_REST.py ./solace_telemetry_consumer_REST.py

then build the container :

docker build . -t $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-rest:0.2

once the container created, publish it in docker hub

docker push $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-rest:0.2

Consumer Database

in /src/consumer_rest, modify the followin line : COPY ./solace_telemetry_consumer_Database_without_spanlinks.py ./solace_telemetry_consumer_Database.py by : COPY ./solace_telemetry_consumer_Database.py ./solace_telemetry_consumer_Database.py

then build the container :

docker build . -t $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-database:0.2

once the container created, publish it in docker hub

docker push $DOCKER_HUB_ACCOUNTNAME/soloacedemo-consumer-database:0.2

Update the deployment files

Modify Manifest/deployment_publisher.yaml Make sure to update the 2 deployments of the consumer to use the image with the tag 0.2 ( the new version of the containers using span links)

Now we can deploy the modified deployment:

kubectl apply -f Manifest/deployment_publisher.yaml -n eda