fd4s/fs2-kafka

Create GracefulShutdown utility

Closed this issue · 1 comments

https://github.com/fd4s/fs2-kafka/blob/series/1.x/docs/src/main/mdoc/consumers.md#graceful-shutdown has an example of graceful shutdown code, but it's quite a bit of code. I imagine a lot of users might not be aware of it and consider a KafkaConsumer a Resource that will be shutdown gracefully.

The docs state:

stopConsuming is just a building block for making your own graceful shutdown, not a ready-made solution for all needs. This design is intentional, because different applications may need different graceful shutdown logic.
However we could extract the code in the example as a base for simple-shutdown logic, and develop it for more complex situations later.

I imagine the most difficult part will be the tests, and proving that code has been shutdown in the correct order.

LMnet commented

Hi @agustafson. I understand your point and agree with it. Yes, the current solution is pretty complex and low level. And I explicitly mentioned this in the pull request:

About design decisions. This method is pretty low-level and not opinionated. It's a building block for making your own graceful shutdown, not a ready-made solution. At first, I tried to add some high-level graceful shutdown functionality. But I found that with the current library design it would be problematic in terms of API. All my experiments end with pretty complex and at the same time not very extensible implementations. So, I decided to provide only a low-level API with detailed documentation.

The main problem for a high-level graceful shutdown is that you need to mark the user stream as uncancelable (point 7 in the docs). You just can't do this from the library API.

I imagine a lot of users might not be aware of it and consider a KafkaConsumer a Resource that will be shut down gracefully.

Not sure what you mean here. KafkaConsumer is a Resource and it follows its rules and guarantees. But consumer.stream not. And graceful shutdown through stopConsuming gives you a way to shut down both consumer and stream gracefully.

You can think about this functionality like about Resource#allocated method. In most situations you don't need it, you could just call use on your resource. But for some complex or custom scenarios, you will need some low-level handling of your resource. So, stopConsuming for KafkaConsumer is like allocated for Resource.

Anyway, If you will end up with some simple and extensible high-level API for graceful shutdown it would be very cool.