Performance regression Producer 3.5.0 vs 3.4.0
Closed this issue · 9 comments
We have upgraded fs2-kafka to 3.5.0 from 3.4.0, and it seems the performance and CPU usage got a serious hit.
In the screenshots above, you can see when we switch back to 3.4.0 at 14:18
Looking at the list of changes between 3.4.0 and 3.5.0, we tried downgrading only fs-core to 3.9.4 as we thought could be the root cause, but still got terrible performance.
It looks to me, but I don't know the internals of cats-effect or fs2 that well, that the problem might be in the change from Sync[F].blocking to Sync[F].interruptible, as from my limited experience, I don't see how any of the other changes could cause this.
I am happy to provide more information if I can. I will try to test my hypothesis above in the meantime. Will report results If I find anything.
I reached the exact same conclusion this afternoon, this PR #1126 seems to be the culprit (tested by reverting it and publishing a local version).
Wow... I'm astonished...
I'm also not well versed on the CE internals, and by looking at the docs for interruptible it didn't seem to hurt, that's why I merged that PR 😢
OFC, I'm going to create a fix for this now, but now I'm really curious of why such a performance drop 🤔 cc @armanbilge was that expected?
Maybe interruptible doesn't shift to the blocking pool?
I can see some complex logic there, including semaphores, AtomicRefs and busy waits not running in the Blocking TP (can explain the CPU increased usage)?
But it does seem that the action is run in the Blocking TP.
We can possibly open an issue for clarification in the cats-effect repo.
Btw thanks a million for the quick reaction and the release 🙇
Just dropping confirmation, the new release 3.5.1 fixes the issue:
I am good to close the issue. Unless you want to keep it open for tracking the interruptible issue.
It's fine to close it. Thank you!