riteraft-py

⚠️ This repository is porting of riteraft and no longer maintained. If you're looking for a high-level Raft implementation, you can check out raftify, which is a fork of riteraft-py.

riteraft-py is an attempt to provide a higher-level Raft implementation in Python based on rraft-py.

Please note that this repository is still experimental and has not been tested thoroughly.

Also, if you want to build featureful Raft implementation, rraft-py could be a good starting point instead of this lib.

Note: Although this library originated from ritelabs/riteraft, it does not guarantee to provide the same API with ritelabs/riteraft, API could change.

Why?

Since raft-rs only offers an implementation for the consensus module, it appears that users may encounter difficulties figuring out how to utilize this library when they first encounter the problem. (Refer: tikv/raft-rs#402)

Attempts to provide a higher-level Raft implementation, such as riteraft, have been made to address this issue.

This repository utilizes riteraft as a starting point to resolve the issue in Python.

Getting started

Installation

TBU

Example

To use Raft storage, we need to implement Storage for it. Below is an example with HashStore, which is a thread-safe wrapper around a HashMap.

class HashStore:
    def __init__(self):
        self._store = dict()
        self._lock = Lock()

    def get(self, key: int) -> Optional[str]:
        with self._lock:
            return self._store.get(key)

    async def apply(self, msg: bytes) -> bytes:
        with self._lock:
            message = InsertMessage.decode(msg)
            self._store[message.key] = message.value
            logging.info(f'Inserted: ({message.key}, "{message.value}")')
            return pickle.dumps(message.value)

    async def snapshot(self) -> bytes:
        with self._lock:
            return pickle.dumps(self._store)

    async def restore(self, snapshot: bytes) -> None:
        with self._lock:
            self._store = pickle.loads(snapshot)

Only 3 methods need to be implemented for the Store:

apply: applies a committed entry to the store.
snapshot: returns snapshot data for the store.
restore: applies the snapshot passed as argument.

Running the raft

async def main() -> None:
    setup_logger()
    parser = argparse.ArgumentParser()
    parser.add_argument("--raft-addr", required=True)
    parser.add_argument("--peer-addr", default=None)
    parser.add_argument("--web-server", default=None)

    args = parser.parse_args()

    options = Options(
        raft_addr=args.raft_addr,
        peer_addr=args.peer_addr,
        web_server=args.web_server,
    )

    store = HashStore()
    raft = Raft(options.raft_addr, store, logger)
    mailbox = raft.mailbox()

    tasks = []
    if options.peer_addr:
        logger.info("Running in follower mode")
        tasks.append(raft.join(options.peer_addr))
    else:
        logger.info("Running in leader mode")
        tasks.append(raft.lead())

    runner = None
    if options.web_server:
        app = Application()
        app.add_routes(routes)
        app["state"] = {
            "store": store,
            "mailbox": mailbox,
        }

        host, port = options.web_server.split(":")
        runner = web.AppRunner(app)
        await runner.setup()
        site = web.TCPSite(runner, host, port)
        tasks.append(site.start())

    try:
        await asyncio.gather(*tasks)
    finally:
        if runner:
            await runner.cleanup()
            await runner.shutdown()

The mailbox provides a method to interact with the raft, such as sending a message or leaving the cluster, for example.

For the complete example code, consult this link.

Reference

ritelabs/riteraft - A raft framework, for regular people. Written in Rust.
lablup/rraft-py - Unofficial Python Binding of the tikv/raft-rs. API using in this lib under the hood.
lablup/aioraft-ng - Unofficial implementation of RAFT consensus algorithm written in asyncio-based Python.
tikv/raft-rs - Raft distributed consensus algorithm implemented in Rust.