shotover/shotover-proxy

Redesign KafkaSinkCluster to handle multiple shotover instances

Closed this issue · 0 comments

rukai commented

There are two possible approaches:

  • Send a metadata message before routing a fetch/produce if we dont know about that topic yet.
  • Communicate kafka metadata between shotover nodes i.e. use raft

However I can already see some problems with a raft solution.

  • race conditions - the client might query a shotover node before it receives the metadata it requires from another shotover node
  • it doesnt handle deletion of metadata - currently topic metadata will grow unboundedly, we arent sure if this is a problem yet, but a raft solution wont fix this while sending our own metadata messages will fix it by allowing us to refetch cleared data if we need it again.