How to set consumer option "auto.offset.reset" ?

Question

How to set consumer option "auto.offset.reset" ?

nobusue opened this issue 8 years ago · 8 comments

I'd like to consume from latest message when I don't have committed offset (ex: new consumer group).
At v0.3.2, HighLevelConsumer consumes from oldest message, but it isn't expected.

Answer 1 · 2016-03-31T16:12:19.000Z

You can set fromOffset true and give the offset you wish to consume in payload, example:

consumer = new HighLevelConsumer(
        client,
        [
            { topic: 't', offset: 1000 }
        ],
        { fromOffset: true }
    );

Answer 2 · 2016-04-01T03:17:07.000Z

Thanks haio, but it doesn't match my needs.
I just intend to start consume latest message for "fresh consumer" (= there is no committed offset entry in zookeeper). In other language like Java, Kafka client API supports to set option map, like "auto.offset.reset=latest".
Is there any way to set Kafka client API option map directly?

Answer 3 · 2016-04-01T09:31:35.000Z

This feature is not supported at the moment.

Answer 4 · 2016-05-13T14:30:54.000Z

I'll have a PR coming for picking up from the latest offset.

Answer 5 · 2017-02-09T19:35:16.000Z

@CharlesWall any option like auto.offset.reset=latest , sometime soon for 0.8.* version of kafka for node client.

Answer 6 · 2017-02-09T19:42:56.000Z

@shubSK I wouldn't hold my breath waiting for that.

Answer 7 · 2017-02-10T14:11:03.000Z

I haven't looked at any of this stuff since May. I learned how this module worked to write the fix I needed. @shubSK You should consider doing the same.

Answer 8 · 2017-04-20T13:49:22.000Z

The solution is to use ConsumerGroup which supports both earliest and latest. I don't think it's worth updating the HLC since it has been deprecated by Kafka project.