rewrite-bigdata-in-rust/RBIR

A collection of RBIR projects and posts for anyone interested in joining this journey.

RBIR

RBIR stands for Rewrite Bigdata in Rust. RBIR aims to create a big data ecosystem using Rust.

This project declares our manifesto and serves as a collection of RBIR projects and posts for anyone interested in joining this journey.

Projects

• Apache DataFusion Comet

  A high-performance accelerator for Apache Spark, built on top of the powerful Apache DataFusion query engine.

• Apache HoraeDB (incubating)

  A high-performance, distributed, cloud native time-series database.

• Arroyo

  A distributed stream processing engine written in Rust, designed to efficiently perform stateful computations on streams of data.

• BLAZE

  The Blaze accelerator for Apache Spark leverages native vectorized execution to accelerate query processing.

• Daft

  A distributed query engine for large-scale data processing in Python and is implemented in Rust.

• Databend

  An open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake

• Fluvio

  Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.

• GlareDB

  An analytics DBMS for distributed data.

• GreptimeDB

  An open-source, cloud-native, unified time series database for metrics, logs and events with SQL/PromQL supported.

• LanceDB

  An open-source database for vector-search built with persistent storage, which greatly simplifies retrieval, filtering and management of embeddings.

• ParadeDB

  An Elasticsearch alternative built on Postgres.

• Quickwit

  Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo

• RisingWave

  A Postgres-compatible SQL database engineered to provide the simplest and most cost-efficient approach for processing, analyzing, and managing real-time event streaming data

• SlateDB

  A cloud native embedded storage engine built on object storage.

• TiKV

  Distributed transactional key-value database, originally created to complement TiDB

• influxdb

  The leading open source time series database for metrics, events, and real-time analytics.

Libraries

• Apache Arrow Rust

  Native Rust implementation of Apache Arrow

• Apache Avro Rust

  Rust implementation of Apache Avro

• Apache DataFusion

  An extensible query engine written in Rust that uses Apache Arrow as its in-memory format.

• Apache Hudi Rust

  Rust implementation of Apache Hudi

• Apache Iceberg Rust

  Rust implementation of Apache Iceberg

• Apache OpenDAL

  A unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services.

• Apache Orc Rust

  Rust implementation of Apache ORC

• Apache Paimon Rust

  Rust implementation of Apache Paimon

• Apache Parquet Rust

  Rust implementation of Apache Parquet

Posts

• Rewrite Bigdata in Rust by @Xuanwo