Producer only configuration
Opened this issue · 5 comments
Hey!
We're using this library in a mode where multiple Orleans clusters (different services) publish events to a single topic. However, they themselves aren't subscribing on streams. Meaning, those clusters are publishers only.
We've noticed that consumers from those services are connecting to the Kafka topics that we are publishing to, even though we don't have stream consumers in those services.
I'm thinking maybe its possible to add a mode to this library to allow null/empty topic list so that no consumers are raised, while publishing logic is still possible.
thoughts?
@jonathansant ping
Hey Eugene. This is implemented according to the Orleans StreamAdapter pattern, which requires a receiver to be created on startup. Having said that, it should not be a problem to implement. offhand I'm thinking of two ways to do it. Either create a NoopAdapterReceiver or simply not subscribe to the topic from the KafkaAdapterReceiver
I'm not sure I see a way to do it with Orleans.Streams.Kafka without creating a custom nuget, am I mistaken?
Not sure what you mean, but we can do it in Orleans.Streams.Kafka and obviously release a new version
What I meant is the current version of the Orleans.Streams.Kafka code doesn't allow much in terms of replacing the moving parts inside. hence, either a custom nuget is needed to solve the specific issue, or an PR into this repo.
off topic: the batch container is not JSON serializable, and is not really replaceable. Should open another issue and maybe PR?