Distributed tracing using Redis as message queue

An example of distributed tracing with Redis as message queue for propagate the context.

Description

[TODO]

Project requirements

Docker compose 1.25.4
GNU/Linux

Project setup

Clone the repo and open the root folder

Start all microservices with the command:

 docker-compose -f docker-compose.yml up -d --build

Visit http://localhost:16686 to view traces

Project architecture

The system is composed by the follow modules:

Apps (that contains two NodeJS apps and a Phyton app)
Redis
Jaeger Agent
Jaeger Collector
Jaeger Query/UI
Elasticsearch

Each module is deployed by a docker container through a docker-compose.

Tracing infrastructure

End to end transaction cycle

Tracing details

Redis as message bus

As described above, the 3 apps: main-service, worker-one and worker-two use Redis to pass the span context.
The context is saved in Redis using a job-id as key.

Main Service code (NodeJS)

At startup the main-service creates a new span and saves the context it in Redis.

    /* apps/nodejs-src/app.js */
    const jobId = Math.floor(new Date() / 1000)
    const mainSpan = await tracer.createSpan(`job-${jobId}`)
    ...
    await tracer.saveContext(jobId, mainSpan.context())

    /* apps/nodejs-src/tracer.js */
    saveContext(id, context) {
        return new Promise((resolve, reject) => {
            var map = {}
            this.tracer.inject(context, opentracing.FORMAT_TEXT_MAP, map)
            redis.set(id, JSON.stringify(map), (err) => {
                if (err) { reject(err) }
                resolve()
            })
        })
    }

Then it executes the 2 workers launching them through two separated shell commands.

    /* apps/nodejs-src/app.js */ 
    async function taskOne(jobId) {
        await runCommand(`node /usr/src/nodejs-app/worker.js ${jobId}`)
    }

    async function taskTwo(jobId) {
        await runCommand(`python3 /usr/src/python-app/worker.py ${jobId}`)
    }

Worker apps code

The two workers receive in input just a job-id, but not the context span, that is retrieved through Redis instead.

Below the code used from the two apps to read the span context from Redis.

Worker-one (NodeJS code)

    /* apps/nodejs-src/tracer.js */
    async createContinuationSpan(spanName, contextId) {
        return this.createSpan(
            spanName, 
            this.extractSpan(await this.loadContext(contextId))
        )
    }
    
    extractSpan(context) {
        if (context) {
            return this.tracer.extract(opentracing.FORMAT_TEXT_MAP, context)
        }
        return undefined
    }
    
    loadContext(id) {
        return new Promise((resolve, reject) => {
            if (!id) resolve(undefined)
            redis.get(id, (err, reply) => {
                if (err) {
                    reject(err)
                }
                if (reply) { 
                    resolve(JSON.parse(reply.toString()))
                } else {
                    resolve(undefined)
                }
            })
        })
    }

Worker-two (Python code)

    def create_continuation_span(self, span_name, context_id):
        return self.create_span(
            span_name, 
            self.extract_span(self.load_context(context_id))
        )
        
    def load_context(self, id):
        if id is None:
            return None
        print('id: {}'.format(id) )
        byte = self.redis_cli.get(id)
        if byte is None:
            return None
        try:
            context = json.loads(byte.decode("utf-8"))
        except Exception as e:
            return e
        return context

    def extract_span(self, context):
        if (context is None):
            return None
        return self.tracer.extract(
            format=Format.TEXT_MAP,
            carrier=context
        )