Weird performance issue inside "RabbitMQ/Exchange/Named/hutch" on consumers' side.
Azdaroth opened this issue · 1 comments
Hi,
We've been using Hutch for a while without any issues and recently, we introduced some RPC-style calls (turned out that it's pretty straight-forward with Hutch) and it's been working great besides some outliers that happen rarely but fairly consistently.
We use New Relic and maybe once every 50 calls I can see a very long trace that starts with RabbitMQ/Exchange/Named/hutch lasting 936 000 ms (yeah, over 15 minutes), which is followed by a super quick execution of the application's code. Since this is the beginning of the trace, I believe it is about getting a message from RabbitMQ by Hutch, and not sending back the response.
For RabbitMQ hosting, we use CloudAMQP with 3 nodes, the application is on Rails 5.1.7 and Ruby 2.6.5, Hutch is up-to-date. I tried correlating it with anything "interesting" in the metrics on CloudAMQP but it doesn't correspond to anything out of ordinary, everything looks just fine there. And we have definitely more Hutch workers available to handle the load than the traffic we generate.
Have you ever seen anything like this?
I haven't and there isn't enough details to suggest anything. Hutch is a fairly small library and you expect what it does on top of Bunny without a significant investment of time. Exchanges specifically are just names for routing tables, they don't really do much work. In Bunny they are represented as objects so maybe that's what the trace suggests.