Flume Hive Batch Sink is a flume plugin to sink event to hive in a batch mode.
Compile the project:
mvn package
Example Configuration:
a1.sinks.k1.type = org.apache.flume.sink.hive.batch.HiveBatchSink
a1.sinks.k1.hive.database = default
a1.sinks.k1.hive.table = mytable
a1.sinks.k1.hive.path=/user/hive/warehouse/default.db/mytable/%Y%m/%Y%m%d
a1.sinks.k1.hive.partition = logdate=%Y%m%d%H%M
a1.sinks.k1.hive.filePrefix = mytable-%Y%m%d%H%M
a1.sinks.k1.hive.serde = com.example.serde.MySerde
a1.sinks.k1.hive.round = true
a1.sinks.k1.hive.roundValue = 5
a1.sinks.k1.hive.roundUnit = minute
- Sink flume event to hive table in a batch mode.
- Support add hive parition.
- With orc storage.
- Support custom hive serde.
- Better sink counter and support persist the counter data to database.
- Leader election when distributed deployed. The leader can determine if all the sink have finished their batch work.