Multiprocessing Proposal - Redis backend

[Llandy3d]

Multiprocessing Proposal - Redis

When moving away from a single process, into multiple processes you start to have the problem of having x amount of services trying to update the same metric that will need to be coordinated.

The ideal solution should be a pluggable system for easily swapping the prefered backend to solve the problem, while the interface used by the client library users should remain the same, the ideal scenario is that it will only be a configuration change. Everything should work as it is in either single process or multiprocess mode.

For this prototype I’m considering a solution with the Redis server as a backend, it provides two useful datastructures for this problem:

• https://redis.io/docs/data-types/hashes/
• https://redis.io/docs/data-types/lists/

We can directly consider to use hashes as they are optimized in memory when with a small amount of fields (although optimization is not the topic of interest right now)

It will work well for Counters, Gauges & also Histograms as this last one is a series of buckets, so basically a series of values to keep track of.

Another important piece to make this solution possible and work is the INCRBYFLOAT & HINCRBYFLOAT operations:

• https://redis.io/commands/hincrbyfloat/
• https://redis.io/commands/incrbyfloat/

Those are fast operations to increase a “float” value, most importantly those are ATOMIC operations meaning that if multiple clients attempt to modify the value, you are assured that the operations will be done correctly and will end up with the correct one.

This would allow the backend to be the place of truth, where each metrics collector will update or modify values, and when you expose the values for scraping, you will be exposing them from this one single truth.
With this approach the idea is that the complexity of syncing metrics between multiple process services will be transparent and easy for the user.

Regarding the how, one option seems to be that the metrics value could be a pluggable class, depending on the environment you want to operate in, and will hide the logic between systems, whether it will be a locked single threaded value or a redis backed backend.

That’s all folks!