bitsail: A Java repository from xuedipiaofei

Introduction

BitSail is ByteDance's open source data integration engine which is based on distributed architecture and provides high performance. It supports data synchronization between multiple heterogeneous data sources, and provides global data integration solutions in batch, streaming, and incremental scenarios. At present, it serves almost all business lines in ByteDance, such as Douyin, Toutiao, etc., and synchronizes hundreds of trillions of data every day.

Official website of BitSail: https://bytedance.github.io/bitsail/

Why Do We Use BitSail

BitSail has been widely used and supports hundreds of trillions of large traffic. At the same time, it has been verified in various scenarios such as the cloud native environment of the volcano engine and the on-premises private cloud environment.

We have accumulated a lot of experience and made a number of optimizations to improve the function of data integration

Global Data Integration, covering batch, streaming and incremental scenarios
Distributed and cloud-native architecture, supporting horizontal scaling
High maturity in terms of accuracy, stability and performance
Rich basic functions, such as type conversion, dirty data processing, flow control, data lake integration, automatic parallelism calculation , etc.
Task running status monitoring, such as traffic, QPS, dirty data, latency, etc.

BitSail Use Scenarios

Mass data synchronization in heterogeneous data sources
Streaming and batch integration data processing capability
Data lake and warehouse integration data processing capability
High performance, high reliability data synchronization
Distributed, cloud-native architecture data integration engine

Features of BitSail

Low start-up cost and high flexibility
Stream-batch integration and Data lake-warehouse integration architecture, one framework covers almost all data synchronization scenarios
High-performance, massive data processing capabilities
DDL automatic synchronization
Type system, conversion between different data source types
Engine independent reading and writing interface, low development cost
Real-time display of task progress, under development
Real-time monitoring of task status

Architecture of BitSail

Source[Input Sources] -> Framework[Data Transmission] -> Sink[Output Sinks]

The data processing pipeline is as follows. First, pull the source data through Input Sources, then process it through the intermediate framework layer, and finally write the data to the target through Output Sinks

At the framework layer, we provide rich functions and take effect for all synchronization scenarios, such as dirty data collection, auto parallelism calculation, task monitoring, etc.

In data synchronization scenarios, it covers batch, streaming, and incremental data synchronization

In the Runtime layer, it supports multiple execution modes, such as yarn, local, and k8s is under development

Supported Connectors

DataSource	Sub Modules	Reader	Writer
Assert	-		✅
ClickHouse	-	✅	-
Doris	-		✅
Druid	-		✅
Elasticsearch	-		✅
Fake	-	✅
FTP/SFTP	-	✅
Hadoop	-	✅	✅
HBase	-	✅	✅
Hive	-	✅	✅
Hudi	-	✅	✅
LocalFileSystem	-	✅
JDBC	MySQL	✅	✅
	Oracle
	PostgreSQL
	SqlServer
Kafka	-	✅	✅
Kudu	-	✅	✅
LarkSheet	-	✅
MongoDB	-	✅	✅
Print	-		✅
Redis	-		✅
RocketMQ	-		✅
SelectDB	-		✅