Apache OpenDAL™

OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.

Major components of the project include:

Libraries

Rust Core
C Binding not released
Cpp Binding not released
Haskell Binding not released
Java Binding
Lua Binding not released
Node.js Binding
Python Binding
Ruby Binding not released
Swift Binding not released
Zig Binding not released

Applications

oli: OpenDAL Command Line Interface
oay: OpenDAL Gateway

Services

Standard Storage Protocols (like ftp, webdav)

ftp: FTP and FTPS
http: HTTP read-only services
sftp: SFTP services being worked on
webdav: WebDAV Service

Object Storage Services (like s3, gcs, azblob)

azblob: Azure Storage Blob services
cos: Tencent Cloud Object Storage services
gcs: Google Cloud Storage Service
obs: Huawei Cloud Object Storage Service (OBS)
oss: Aliyun Object Storage Service (OSS)
s3: AWS S3 alike services
supabase: Supabase Storage Service being worked on
wasabi: Wasabi Cloud Storage

File Storage Services (like fs, azdls, hdfs)

fs: POSIX alike file system
azdls: Azure Data Lake Storage Gen2 services (As known as ABFS)
hdfs: Hadoop Distributed File System(HDFS)
ipfs: InterPlanetary File System HTTP Gateway
ipmfs: InterPlanetary File System MFS API being worked on
webhdfs: WebHDFS Service

Consumer Cloud Storage Service (like gdrive, onedrive)

gdrive: Google Drive being worked on
onedrive: OneDrive being worked on

Key-Value Storage Service (like rocksdb, sled)

cacache: cacache backend
dashmap: dashmap backend
memory: In memory backend
persy: persy backend
redis: Redis services
rocksdb: RocksDB services
sled: sled backend
redb: redb backend
tikv: tikv backend
atomicserver: Atomicserver services

Cache Storage Service (like memcached, moka)

ghac: GitHub Action Cache Service
memcached: Memcached service
mini_moka: Mini Moka backend
moka: Moka backend
vercel_artifacts: Vercel Remote Caching Service being worked on

Welcome to add any services that are not currently supported here.

Examples

The examples are available at here.

Documentation

The documentation is available at https://opendal.apache.org.

Contribute

OpenDAL is an active open-source project. We are always open to people who want to use it or contribute to it. Here are some ways to go.

Start with Contributing Guide.
Submit Issues for bug report or feature requests.
Discuss at dev mailing list (subscribe / unsubscribe / archives)
Asking questions in the Discussions.
Talk to community directly at Discord.

Who is using OpenDAL?

Rust Core

Databend: A modern Elasticity and Performance cloud data warehouse.
GreptimeDB: An open-source, cloud-native, distributed time-series database.
deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.
mozilla/sccache: sccache is ccache with cloud storage
RisingWave: A Distributed SQL Database for Stream Processing
Vector: A high-performance observability data pipeline.
OctoBase: the open-source database behind AFFiNE, local-first, yet collaborative.
Pants: A fast, scalable, user-friendly build system for codebases of all sizes.
QuestDB: An open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

C Binding

Milvus: A cloud-native vector database, storage for next generation AI applications

Java Binding

QuestDB: An open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

Branding

The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.

For more details, see the Apache Product Name Usage Guide.

License

Licensed under the Apache License, Version 2.0

BeautyyuYanli/incubator-opendal