Welcome to arctic, arctic is a streaming lake warehouse system open sourced by NetEase. Arctic adds more real-time capabilities on top of iceberg and hive, and provides stream-batch unified, out-of-the-box metadata services for dataops, allowing Data lakes much more usable and practical.
Arctic is a streaming lakehouse service built on top of apache iceberg table format.
Through arctic, users could benefit optimized CDC、streaming update、fresh olap etc. on engines like flink, spark, and trino.
Combined with efficient offline processing capabilities of data lakes, arctic can serve more scenarios where streaming and batch are fused.
At the same time, the function of self-optimization、concurrent conflict resolution and standard management tools could effectively reduce the burden on users in data lake management and optimization.
Arctic services are presented by deploying AMS, which can be considered as a replacement for HMS (Hive Metastore), or HMS for iceberg. Arctic uses iceberg as the base table format, but instead of hacking the iceberg implementation, it uses iceberg as a lib. Arctic's open overlay architecture can help large-scale offline data lakes quickly upgraded to real-time data lakes, without worrying about compatibility issues with the original data lakes, allowing data lakes to meet more real-time analysis, real-time risk control, Real-time training, feature engineering and other scenarios.
- Efficient streaming updates based on primary keys
- Data auto bucketing and self-optimized for performance and efficiency
- Encapsulating data lake and message queue into a unified table to achieve lower-latency computing
- Provide standardized metrics, dashboard and related management tools for streaming lakehouse
- Support spark and flink to read and write data, support trino to query data
- 100% compatible with iceberg / hive table format and syntax
- Provide transactional guarantee for streaming and batch concurrent writing
Arctic contains modules as below:
arctic-corecontains core abstractions and common implementions for other modulesarctic-flinkis the module for integrating with Apache Flink (use arctic-flink-runtime for a shaded version)arctic-sparkis the module for integrating with Apache Spark (use arctic-spark-runtime for a shaded version)arctic-trinonow provides query integrating with apache trino, built on JDK11arctic-optimizingexposes optimizing container/group api and provides default implemetionarctic-amsis arctic meta service moduleams-apicontains ams thrift apiams-dashboardis the dashboard frontend for amsams-serveris the backend server for ams
Arctic is built using Maven with Java 1.8 and Java 11(only for trino module).
- To build Trino module need config
toolchains.xmlin${user.home}/.m2/dir, the content is
<?xml version="1.0" encoding="UTF-8"?>
<toolchains>
<toolchain>
<type>jdk</type>
<provides>
<version>11</version>
<vendor>sun</vendor>
</provides>
<configuration>
<jdkHome>${yourJdk11Home}</jdkHome>
</configuration>
</toolchain>
</toolchains>
- To invoke a build and run tests:
mvn package -P toolchain - To skip tests:
mvn -DskipTests package -P toolchain
Arctic support multiple processing engines as below:
| Processing Engine | Version |
|---|---|
| Flink | 1.12.x, 1.14.x and 1.15.x |
| Spark | 3.1+ |
| Trino | 380 |
Visit https://arctic.netease.com/ch/docker-quickstart/ to quickly explore what arctic can do.
If you are interested in Lakehouse, Data Lake Format, welcome to join our community, we welcome any organizations, teams and individuals to grow together, and sincerely hope to help users better use Data Lake Format through open source.
Join the Arctic WeChat Group: Add " kllnn999 " as a friend on WeChat and specify "Arctic lover".