Slow message delivery

[ramank775]

There is significant delay observed with sending messages.

Observation:

• It seems like message delivery taking significant time to send message to gateway. Possible area to investigate
  • time taken to save message is db
  • fetching user state
  • Formation of groups

Possibility

• Delay at message broker

[ramank775]

Issue seems to be with the kafka cluster we are using i.e. basic tier of confluent cloud.
It seems like they are rate limiting which leads to some delay. As per the documentation rate limit is set to 80 connection per second and 1500 requests per second. Which is high enough for us at the present level of load.
In order to verify, monitor performance with self hosted version. With Self hosting version I am expecting to gain some performance as part of network also local network access vs over the internet.