3scale/apisonator

Incompatible with Redis Cluster

Opened this issue · 3 comments

davidor commented

Apisonator is not compatible with the Redis cluster mode.

I did not try all the Apisonator functionality, I just tried to make a basic test work. The problems can be summarized in two:

  • Apisonator does not guarantee that all the keys inside a pipeline belong to the same shard, which is a requirement to use Redis Cluster. There are many examples of pipelines with keys that are not guaranteed to belong to the same shard (do not include the "{}" hashtag): Alerts.update_utilization, Application.save, EventStorage.pending_ping?, Metric.save, etc.
  • Apisonator does not guarantee that all the keys passed as params for a single command belong to the same shard, which is also a requirement to use Redis cluster. This affects all commands that accept multiple keys, for example, mget, blpop, hmget, etc. Examples: Application.load, Service.get_service, some methods in UsageLimit, etc. Also, the blpop used to fetch jobs from the queue.

I don't see a straightforward solution for those problems.

Modifying all the keys to ensure that in every operation they belong to the same shard might be complicated. Migrating a running system might be very challenging.

The solution of replacing all the mgets with multiple gets, remove the pipelines, etc. is not feasible. The performance hit would be very high.

andrewdavidmackenzie commented

Maybe some combination:

  • remove pipelining for non-performance sensitive methods (e.g. is Application.save used "a lot"?)
  • review implementation of the remaining ones that are considered performance sensitive?

Out of curiosity, is twemproxy handling all of these cases when we shard manually (non-cluster)?

unleashed commented

I think the real fix to be able to support Redis Cluster is to organise the keys used by apisonator in data structures with namespacing containing associated data (ie. any non plain KV structure, such as hashes, sets, lists, etc), which would also help in being able to perform transactions in clusters (due to dealing with a single shard).

The short term option I see is rewriting pipelines into normal 1-by-1 commands when using Redis Cluster, but that would severely hinder performance.

Edit: another option is decomposing pipelines into per-shard operations and submit those operations as per-shard pipelines to be executed in parallel.

Edit2: of course that would mean hooking up into the Redis client and knowing about how the sharding is done...

davidor commented

We should probably give a try to the proxies that support Redis cluster:

  • 👍1