ArtRand/kafka-actix-example

Consider `add_stream()` alternative

jerry73204 opened this issue · 1 comments

I ran into the need to integrate actix and kafka as well. I looked into your kafka.rs and there might be a elegant implementation there. The ctx.run_interval() in original code makes it poll periodically. That would hurt the performance in case of high throughput. Let me paste another example using add_stream(). I'm glad if you have comments.

Hello @jerry73204, thanks for pasting this! You're correct that add_stream might be more idiomatic. Do you know a way to add back pressure with that approach? For example imagine that sending to the redis actor kicked off an expensive computation and you didn't want a "thundering herd" to throttle the compute (or your wallet), how could you do it with add_stream? I appreciate that getting the messages at an interval seems unconventional, however, the kafka consumer is already eagerly fetching the messages up to fetch.message.max.bytes anyways, so the interval polling is just a controller on that. Another approach would be to have the final sink (redis in this case) apply the back pressure (only consume so fast).