RocketMQ-Flink

RocketMQ integration for Apache Flink. This module includes the RocketMQ source and sink that allows a flink job to either write messages into a topic or read from topics in a flink job.

相较于rocketmq-externals/rocketmq-flink 版本对比：

优化：消息传输byte[]以nio方式读取数据
新增功能：实现指定offset定点位置启动
修复bug：自带checkpoint灵活性差，且会导致重新配置消费旧数据失效

指定位置

代码

consumerProps.setProperty(RocketMQConfig.CONSUMER_OFFSET_RESET_TO, RocketMQConfig.CONSUMER_OFFSET_SITE);
consumerProps.setProperty(RocketMQConfig.CONSUMER_OFFSET_SITE_STARTING_OFFSETS, "{\"2\":{\"iZbp1f9edjszup3fshxxheZ\":{\"0\":2,\"1\":3,\"2\":4,\"3\":3,\"4\":4,\"5\":3,\"6\":4,\"7\":5}}}");

配置

{
    "2": {
        "iZbp1f9edjszup3fshxxheZ": {
            "0": 2,
            "1": 3,
            "2": 4,
            "3": 3,
            "4": 4,
            "5": 3,
            "6": 4,
            "7": 5
        }
    }
}

问题

测试发现，部分分区可能会出现丢失

cmmsky/rocketmq-flink

RocketMQ-Flink

指定位置

问题

赞助