Consumption of partition stops if segment expires
Closed this issue · 1 comments
horkhe commented
If a segment of a topic partition is deleted by Kafka due to retention period exceeded while Kafka-Pixy reads from it, then consumption from the partition stops. There is no even log record with ERROR severity reported.
Mar 23 07:01:38 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> started
Mar 23 07:01:38 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> partition claimed: via=/_[0]/cons[0]/G:analytics[1]/member[0], retries=0, took=280ms
Mar 23 07:01:38 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0]> started
Mar 23 07:01:38 </_[0]/offset_mgr_f[0]/mapper[0]/reassign[164]> assign /_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0] -> /_[0]/offset_mgr_f[0]/broker_2_aggr[0] (ref=9)
Mar 23 07:01:38 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0]> assigned /_[0]/offset_mgr_f[0]/broker_2_aggr[0]
Mar 23 07:01:40 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> initialized: offset=485662619, sparseAcks=
Mar 23 07:01:40 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> started
Mar 23 07:01:40 </_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/mapper[0]/reassign[5]> assign /_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0] -> /_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/broker_0_aggr[0] (ref=2)
Mar 23 07:01:40 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> assigned /_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/broker_0_aggr[0]
Mar 23 07:02:39 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> fetch failed: err=kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
Mar 23 07:02:39 </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> stopped
horkhe commented
Apparently this issue was not fixed, we still experience it on v0.13.0