psarna/seastar

Improve metadata

Opened this issue · 0 comments

  • Refactor refresh_metadata() from kafka_producer into separate class (maybe metadata_updater, metadata_manager?). Warning: you may need to change field connection_manager in kafka_producer from connection_manager type to lw_shared_ptr<connection_manager> type in order to pass it to metadata_updater.
  • Currently each produce request in kafka_producer refreshes the matadata information. Modify it to cache the information (inside metadata_updater) - refresh_metadata() in metadata_updater should refresh and store the information and subsequent get_metadata() in metadata_updater should read the stored metadata. produce in kafka_producer should use get_metadata() and init in kafka_producer should run refresh_metadata().

I recommend making a PR after ↑ previous 2 points ↑, so that we can use this code in other tasks (next 2 points won't change the interface of the class)

  • Implement functionality to update the cache every X seconds (run refresh_metadata() automatically every X seconds inside metadata_updater).
  • Add error handling/retries to metadata_updater. Currently we ask only 1 broker about the metadata, however that broker could become nonresponsive (metadata._error_code != NONE). In such case, we should ask another broker for metadata (and if it fails, another broker, and another...). Keep in mind that at the start, we have only 1 user-provided bootstrap broker (also modify it to be a list of many bootstrap brokers). Once we get our first metadata response, we should rely on its broker list (for next refresh_metadata() and its retries).