/flink-iceberg-demo

flink iceberg integration tests, jobs running on yarn.

Primary LanguageJavaApache License 2.0Apache-2.0

Flink iceberg integration tests

Testing flink jobs on yarn, jobs integrated with iceberg, hive, hbase, kafka, elasticsearch, etc.

Used components

  • Apache Hadoop 3.2
  • Kafka 2.0.0
  • Hbase 2.0.0
  • Hive 3.1.2
  • ELK 7.9.1
  • Iceberg 0.10.0
  • Flink 1.11.3

How to Run

  • Deploy a testing big data platform using docker-compose. Download here

  • Build each jobs and run it on yarn with yarn session or pre-job.

TODO

  1. testing ozone and alluxio.
  2. deep integration with iceberg and flink sql.
  3. running jobs on k8s.