OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.
Major components of the project include:
Libraries
- Rust Core
- Node.js Binding
- Python Binding
- C Binding being worked on
- Java Binding being worked on
- Ruby Binding being worked on
Applications
Services
Standard Storage Protocols (like fs, ftp)
Object Storage Services (like s3, gcs, azblob)
- azblob: Azure Storage Blob services
- gcs: Google Cloud Storage Service
- obs: Huawei Cloud Object Storage Service (OBS)
- oss: Aliyun Object Storage Service (OSS)
- s3: AWS S3 alike services
- supabase: Supabase Storage Service being worked on
- wasabi: Wasabi Cloud Storage
File Storage Services (like azdfs, hdfs)
- azdfs: Azure Data Lake Storage Gen2 services (As known as abfs)
- hdfs: Hadoop Distributed File System(HDFS)
- ipfs: InterPlanetary File System HTTP Gateway
- ipmfs: InterPlanetary File System MFS API being worked on
- webhdfs: WebHDFS Service
Consumer Cloud Storage Service (like gdrive, onedrive)
- gdrive: Google Drive being worked on
- onedrive: OneDrive being worked on
Key-Value Storage Service (like rocksdb, sled)
Cache Storage Service (like memcached, moka)
- ghac: Github Action Cache Service
- memcached: Memcached service
- moka: moka backend
- vercel_artifacts: Vercel Remote Caching Service being worked on
Welcome to add any services that are not currently supported here.
Examples
The examples are available at here.
Documentation
The documentation is available at https://opendal.apache.org.
Contribute
OpenDAL is an active open-source project. We are always open to people who want to use it or contribute to it. Here are some ways to go.
- Start with Contributing Guide.
- Submit Issues for bug report or feature requests.
- Discuss at dev mailing list (subscribe / unsubscribe / archives)
- Asking questions in the Discussions.
- Talk to community directly at Discord.
Who is using OpenDAL?
- Databend: A modern Elasticity and Performance cloud data warehouse.
- GreptimeDB: An open-source, cloud-native, distributed time-series database.
- deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.
- mozilla/sccache:
sccache
isccache
with cloud storage - RisingWave: A Distributed SQL Database for Stream Processing
- Vector: A high-performance observability data pipeline.
License
Licensed under the Apache License, Version 2.0