LakeSoul is a unified streaming and batch table storage for fast data processing built on top of the Apache Spark engine by the DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and streaming & batch unification.
LakeSoul implements incremental upserts for both row and column and allows concurrent updates on the same partition. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieve very high write throughput (30MB/s/core) on cloud object store like S3 while providing optimized merge on read performance. LakeSoul scales meta data management by using distributed NoSQL DB Cassandra.
More detailed features please refer to our wiki page: Wiki Home
Please find usage documentations in project's wiki: Usage Doc
Follow the Quick Start to quickly set up a test env.
Checkout the CDC Ingestion with Debezium and Kafka example on how to sync LakeSoul table with OLTP dbs like MySQL in a realtime manner.
- Meta Management
- Multiple Level Partitioning: Multiple range partition and at most one hash partition
- Concurrent write with auto conflict resolution
- MVCC with read isolation
- Write transaction through Postgres Transaction
- Table operations
- LSM-Tree style upsert for hash partitioned table
- Merge on read for hash partition with upsert delta file
- Copy on write update for non hash partitioned table
- Compaction
- Spark Integration
- Table/Dataframe API
- SQL support with catalog except upsert
- Query optimization
- Shuffle/Join elimination for operations on primary key
- Merge UDF (Merge operator)
- Merge Into SQL support
- Merge Into SQL with match on Primary Key (Merge on read)
- Merge Into SQL with match on non-pk
- Merge Into SQL with match condition and complex expression (Merge on read when match on PK)
- Flink Integration
- Table API
- Flink CDC
- Hive Integration
- Export to Hive partition after compaction
- Realtime Data Warehousing
- CDC ingestion
- Time Travel (Snapshot read)
- Snapshot rollback
- MPP Engine Integration
- Presto
- Apache Doris
- Cloud Native
- Object storage IO optimization
- Multi-layer storage classes support with data tiering
Please feel free to open an issue or dicussion if you have any questions.
Join our slack user group
Email us at opensource@dmetasoul.com.
LakeSoul is opensourced under Apache License v2.0.