/wasi-keyvalue

WASI Key-Value Store

A proposed WebAssembly System Interface API.

Current Phase

WASI Key-Value is currently in Phase 2.

Champions

  • Dan Chiarlone
  • David Justice
  • Jiaxiao Zhou
  • Taylor Thomas

Portability Criteria

wasi:keyvalue must have at least two implementations in terms of relevant mainstream key-value stores in each of the following categories:

  1. Open-source key-value stores like Redis, Memcached, Etcd etc.
  2. Proprietary key-value stores like AWS DynamoDB, Azure CosmosDB, Google Cloud Firestore etc.

The implementations must be able to run on Linux, MacOS, and Windows.

A testsuite that passes on the platforms and implementations mentioned above.

Table of Contents

Introduction

This wasi:keyvalue proposal defines a collection of interfaces for interacting with key-value stores. It additionally defines a world, wasi:keyvalue/imports, that groups together common interfaces including

  1. Store operations on single key-value pairs
  2. Atomic operations on single key-value pairs
  3. Batch operations on multiple key-value pairs

There is also an additional world called wasi:keyvalue/handle-watch, which includes the full imports world as well as a set of interfaces for watching for changes to a key-value store.

The API is designed to hide data-plane differences between different key-value stores. The control-plane behavior/functionality (e.g., including cluster management, data consistency, replication, and sharding) are not specified, and are provider specific.

Goals

The primary goal of this API is to allow WASI programs to access a wide variety of key-value stores. This means the wasi:keyvalue interfaces should be implementable by a wide variety of key-value stores, including but not limited to: in-memory key-value stores, on-disk key-value stores, document databases, relational databases, and either single-node or distributed key-value stores.

The second goal of wasi:keyvalue interfaces is to abstract away the network stack, allowing applications that need to access key-value stores to not worry about what network protocol is used to access the key-value store.

A third design goal of wasi:keyvalue interfaces is to allow performance optimizations without compromising the portability of the API. This includes minimizing the number of copies of data, minimizing the number of requests and round-trips, and support for asynchronous operations.

Non-goals

  • The wasi:keyvalue interfaces only focus on the most common use cases for key-value stores. It does not attempt to cover all possible use cases.
  • The wasi:keyvalue interfaces do not guarantee data consistency except as defined in the interface documentation. Data consistency models (eventual, strong, casual etc.) are diverse and are specific to each key-value store's architecture. The implication is that components importing wasi:keyvalue interfaces would need to be aware of the data consistency model of the key-value store they are using.
  • The wasi:keyvalue interfaces do not handle data replication and sharding. These are control-plane behaviors that are specific to each key-value store's architecture. Replication needs and sharding management are left to the outside of the wasi:keyvalue interfaces.
  • No cluster management is provided. Operational aspects like scaling, node management, and cluster health are to be managed separately from the wasi:keyvalue interface wasi:keyvalue interfaces.
  • No built-in monitoring. Monitoring of key-value store performance and health should be done using external tools and not be expected as part of the wasi:keyvalue interfaces.

API walk-through

The proposal can be understood by first reading the comments of world.wit, store.wit, atomic.wit, and batch.wit.

Working with the WIT

Bindings can be generated from the wit directory via:

wit-bindgen c wit/ --world imports

and can be validated and otherwise manipulated via:

wasm-tools component wit wit/ ...

References & acknowledgements

This proposal was inspired by Dapr's [State API]

State API

Change log

  • 2024-03-29:
    • Simplified interfaces to 3 key pieces of functionality: store, atomics, and batch. Also removed the use of streams to simplify the starting point of the API
  • 2024-01-16:
    • Changed the readwrite and batch interface names to eventual and eventual-batch. See more details here.
  • 2023-12-19:
    • Changed the size to incoming-value-size and it's signature
  • 2023-11-30:
    • Changed the get and get-many and keys signatures
    • Updated comments in all the interfaces.
    • Renamed wasi-cloud-error to wasi-keyvalue-error
  • 2023-05-17: Updated batch example to use one interface instead of 2
  • 2023-05-25: Change the WITs to the newest syntax.
  • 2023-02-13: The following changes were made to the API:
    • Added bucket type to the types interface.
    • Uses pseudo-stream and pseudo-resource types inspired from wasi-io
    • Added *.wit files for the interfaces and worlds, which are verified by the wasm-tools
    • Added a inbound-keyvalue interface that handles the watch operation.
  • 2023-01-28: The following changes were made to the API:
    • Changed stream<T> type to list<T> type because stream<T> is not supported by the current version of *.wit.
    • Removed the worlds section because the star import syntax is not supported by the current version of *.wit
  • 2022-11-29: Renamed bulk-get to get-multi and bulk-set to set-multi to be consistent with the naming convention of the other interfaces.
  • 2022-10-31: Initial draft