Need to handle Multi Threading for Manual Instrumentation in Faust Application and How to enable opentracing in Faust

[Chiraguthra]

Hi Team,

Need help on below problem/implementation for Manual Instrumentation in python for faust streaming application.

Currently on the streaming process we use async to manage concurrency on streaming part which manages the event properly with following code.

@app.agent(topic, concurrency=3)
async def topic_func(topic):
async for event in topic.events():

Now we have started using opentelemetry to span traces for the application. Since Faust doesnt support auto instrumentation yet, we progressed with Manual instrumentation.

For this we are managing and creating context for span in separate library which is outside this async scope and using the same across to create child span on different methods using decorator.

Due to streaming service has concurrency in place which creates multi thread, hence using the context variable which is acting as global variable getting conflicted in multiple threads.

Need to know if there is any existing library which can manage this multi threading and I can use the same context accross to create the child spans.

Also, if possible, could you please also help on , how can I enable opentracing library in faust to enable traces ?

Faust Ver - faust-streaming==0.8.4

Looking forward for the response.

Thanks.
Chirag

[wbarnha]

Now we have started using opentelemetry to span traces for the application. Since Faust doesnt support auto instrumentation yet, we progressed with Manual instrumentation.

I've actually been meaning to migrate Faust from Opentracing to Opentelemetry in #382. I'm very interested to see what you've done so far.

Need to know if there is any existing library which can manage this multi threading and I can use the same context accross to create the child spans.

Sadly I don't know of any, yet.

I know @patkivikram added use cases for opentracing w/Faust in https://github.com/faust-streaming/faust/blob/master/faust/sensors/distributed_tracing.py, perhaps he could provide some insight?

[Chiraguthra]

Hi @patkivikram .. Could you help on this? how can we enable tracing in faust using opentracing library ?

[patkivikram]

We use Datadaog as the implementation for opentracing... but at a high level we do the following in our app

    tracing_sensor = TracingSensor()
    app.sensors.add(tracing_sensor)