Redesign KafkaSinkCluster to handle multiple shotover instances
Closed this issue · 0 comments
rukai commented
There are two possible approaches:
- Send a metadata message before routing a fetch/produce if we dont know about that topic yet.
- Communicate kafka metadata between shotover nodes i.e. use raft
However I can already see some problems with a raft solution.
- race conditions - the client might query a shotover node before it receives the metadata it requires from another shotover node
- it doesnt handle deletion of metadata - currently topic metadata will grow unboundedly, we arent sure if this is a problem yet, but a raft solution wont fix this while sending our own metadata messages will fix it by allowing us to refetch cleared data if we need it again.