mailgun/kafka-pixy

Consumption of partition stops if segment expires

Closed this issue · 1 comments

If a segment of a topic partition is deleted by Kafka due to retention period exceeded while Kafka-Pixy reads from it, then consumption from the partition stops. There is no even log record with ERROR severity reported.

Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> started
Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> partition claimed: via=/_[0]/cons[0]/G:analytics[1]/member[0], retries=0, took=280ms
Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0]> started
Mar 23 07:01:38  </_[0]/offset_mgr_f[0]/mapper[0]/reassign[164]> assign /_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0] -> /_[0]/offset_mgr_f[0]/broker_2_aggr[0] (ref=9)
Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0]> assigned /_[0]/offset_mgr_f[0]/broker_2_aggr[0]
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> initialized: offset=485662619, sparseAcks=
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> started
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/mapper[0]/reassign[5]> assign /_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0] -> /_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/broker_0_aggr[0] (ref=2)
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> assigned /_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/broker_0_aggr[0]
Mar 23 07:02:39  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> fetch failed: err=kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
Mar 23 07:02:39  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> stopped

Apparently this issue was not fixed, we still experience it on v0.13.0