About

This repository contains source code with examples for this article about Distributed tracing. The article breaks down concepts of distributed tracings and shows how to instrument your python/golang microservices.

Example microservice architecture

The repo contains an example of a system of two microservices, imagine you have a blog where the main service is responsible for showing articles to users (articles-microservice), and you also want to recommend users articles that are similar to the ones they already have in their feed (suggestions-microservice)

The system diagram:

+-------------------+
|                   |
|                   |      enrich articles response
|  articles         |       with suggested articles
|  microservice     <----------------------+
|                   |                      |
|                   |                      |
+---------^---------+                      |
          |load data                       |
          |                                |
   +------v--------+                       |
   |               |              +--------v-----------+
   |               |              |                    |
   | articles.json |              |    suggestions     |
   |               |              |   microservice     |
   |               |              |                    |
   +---------------+              |                    |
                                  +--------------^-----+
                                                 |
                                                 |load data
                                                 |
                                                 |
                                       +---------v------+
                                       |                |
                                       |suggestions.json|
                                       |                |
                                       |                |
                                       +----------------+

As a data source, the systems use files generated by this command

export number_of_articles=10
export number_of_suggestions_per_article=5

scripts/generate_mocks ${number_of_articles} ${number_of_suggestions_per_article}

How to run the system

run the main docker-compose file with microservices

docker compose up

run Jaeger docker-compose

docker compose -f jaeger/docker-compose.yaml up

now you can interact with the system, you can try to make a request, get a response and trace samples

curl http://localhost:8001/articles/v1/?with_suggested=1

open JaegerUI to see tracing data

How to run it with different from Jaeger tracing backend

For this purpose we will use Tempo - open source distributed tracing backend from Grafana Labs, here you can find preconfigured Grafana stack (Tempo, Prometheus, Grafana)

There are just a few commands you have to run to make it work

here and here replace TRACING_BACKEND_URL=jaeger:4317 with TRACING_BACKEND_URL=tempo:4317
run the main docker-compose file

docker compose up

run Grafana stack docker compose

docker compose -f grafana/docker-compose.yaml up

make a request

curl http://localhost:8001/articles/v1/?with_suggested=1

enjoy your traces in Grafana

This shows how easily you can replace the tracing backend by instrumenting your code with opentelemetry.

How data flows in the system

When you make a request to articles microservice without with_suggested query parameter, articles microservice retrieves requested data from the memory (data was loaded on the startup from the mocks/articles.json), and gives it to you, when you add with_suggested parameter, for every article in the list articles microservice query suggestions microservice, which retrieves a list of suggested articles from its memory (data was loaded on the startup from the mocks/suggestions.json), then articles microservice enrich these id's with articles data and reply with this data combination to the user

2 cents about the @Trace() decorator in the suggestions microservice

It's well documented right in the code, but I want to add several words about what it can do

How to use it?

We can decorate classes or functions. when we decorate class all its methods automatically decorate with @Trace() and provided argument, it also works for classmethods and staticmethods.

Example:

@Trace()
class SuggestionsService:
    def get_suggestions(self) -> Dict[str, List[str]]:
        ...

    def get_suggestions_for_article(self, article_id: str) -> List[str]:
        ...

the code above means that all methods of SuggestionsService will be instrumented, this is equivalent of

class SuggestionsService:
    @Trace()
    def get_suggestions(self) -> Dict[str, List[str]]:
        ...

    @Trace()
    def get_suggestions_for_article(self, article_id: str) -> List[str]:
        ...

but what if we want to disable tracing for some methods of SuggestionsService?

Example:

@Trace()
class SuggestionsService:
    def get_suggestions(self) -> Dict[str, List[str]]:
        ...

    @Trace(disable_tracing=True) 
    def get_suggestions_for_article(self, article_id: str) -> List[str]:
        ...

What If I want to disable tracing on selected endpoints?

Everything you have to do is decorate its view with @Trace(disable_tracing=True)

@Trace(disable_tracing=True)
def get_by_article_id(article_id):
  ...

this will automatically disable tracing for the whole request, even on methods used inside get_by_article_id view.

And lastly, it is worth mentioning that this decorator doesn't add any overhead in runtime, as all heavy introspection happens during the module's initialization step, during the application startup.