jackwener/opendal

OpenDAL: Access data freely, painlessly, and efficiently

Apache OpenDAL

OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.

Major components of the project include:

Libraries

• Rust Core
• C Binding not released
• Cpp Binding not released
• Haskell Binding not released
• Java Binding
• Lua Binding not released
• Node.js Binding
• Python Binding
• Ruby Binding not released
• Swift Binding not released
• Zig Binding not released

Applications

• oli: OpenDAL Command Line Interface
• oay: OpenDAL Gateway

Services

Standard Storage Protocols (like ftp, webdav)

• ftp: FTP and FTPS
• http: HTTP read-only services
• sftp: SFTP services being worked on
• webdav: WebDAV Service

Object Storage Services (like s3, gcs, azblob)

• azblob: Azure Storage Blob services
• cos: Tencent Cloud Object Storage services
• gcs: Google Cloud Storage Service
• obs: Huawei Cloud Object Storage Service (OBS)
• oss: Aliyun Object Storage Service (OSS)
• s3: AWS S3 alike services
• supabase: Supabase Storage Service being worked on
• wasabi: Wasabi Cloud Storage

File Storage Services (like fs, azdls, hdfs)

• fs: POSIX alike file system
• azdls: Azure Data Lake Storage Gen2 services (As known as ABFS)
• hdfs: Hadoop Distributed File System(HDFS)
• ipfs: InterPlanetary File System HTTP Gateway
• ipmfs: InterPlanetary File System MFS API being worked on
• webhdfs: WebHDFS Service

Consumer Cloud Storage Service (like gdrive, onedrive)

• gdrive: Google Drive being worked on
• onedrive: OneDrive being worked on

Key-Value Storage Service (like rocksdb, sled)

• cacache: cacache backend
• dashmap: dashmap backend
• memory: In memory backend
• persy: persy backend
• redis: Redis services
• rocksdb: RocksDB services
• sled: sled backend
• redb: redb backend
• tikv: tikv backend
• atomicserver: Atomicserver services

Cache Storage Service (like memcached, moka)

• ghac: GitHub Action Cache Service
• memcached: Memcached service
• mini_moka: Mini Moka backend
• moka: Moka backend
• vercel_artifacts: Vercel Remote Caching Service being worked on

Welcome to add any services that are not currently supported here.

Examples

The examples are available at here.

Documentation

The documentation is available at https://opendal.apache.org.

Contribute

OpenDAL is an active open-source project. We are always open to people who want to use it or contribute to it. Here are some ways to go.

• Start with Contributing Guide.
• Submit Issues for bug report or feature requests.
• Discuss at dev mailing list (subscribe / unsubscribe / archives)
• Asking questions in the Discussions.
• Talk to community directly at Discord.

Who is using OpenDAL?

Rust Core

• Databend: A modern Elasticity and Performance cloud data warehouse.
• GreptimeDB: An open-source, cloud-native, distributed time-series database.
• deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.
• mozilla/sccache: sccache is ccache with cloud storage
• RisingWave: A Distributed SQL Database for Stream Processing
• Vector: A high-performance observability data pipeline.
• OctoBase: the open-source database behind AFFiNE, local-first, yet collaborative.
• Pants: A fast, scalable, user-friendly build system for codebases of all sizes.
• QuestDB: An open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

C Binding

• Milvus: A cloud-native vector database, storage for next generation AI applications

Java Binding

• QuestDB: An open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

License

Licensed under the Apache License, Version 2.0