Flume version : 1.7.0
Canal version : 1.0.24
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
https://github.com/alibaba/canal
阿里巴巴mysql数据库binlog的增量订阅&消费组件
flume-canal-source 是对flume的source扩展。从canal获取数据到flume channel。 进而可以实现binlog数据到kafka/hdfs/hive/elasticsearch等等。 canal 和 flume 都有高可用的解决方案，这种方式同步 binlog 可用性非常高。
部署 canal、flume 这里忽略。
- 配置 source 类型*
agent.sources = canalSource
agent.sources.canalSource.type = com.weiboyi.etl.flume.source.canal.CanalSource- 配置连接 canal 的三种方式*
- canal zookeeper servers
agent.sources.canalSource.zkServers = zookeeper-host:2181- canal server urls
agent.sources.canalSource.serverUrls = canal-server1:111111,canal-server2:111111- canal server urls
agent.sources.canalSource.serverUrl = canal-server1:111111- 配置 canal destination*
agent.sources.canalSource.destination = example- 配置用户名密码
agent.sources.canalSource.username = user
agent.sources.canalSource.password = passwd- binlog batch size, default 1024
agent.sources.canalSource.batchSize = 1024- 是否需要 MySQL 修改前的数据, default true
agent.sources.canalSource.oldDataRequired = true