Rain

Rain is an open-source distributed computational framework for processing of large-scale task-based pipelines.

Rain aims to lower the entry barrier to the world of distributed computing. Our intention is to provide a light yet robust distributed framework that features an intuitive Python API, straightforward installation and deployment with insightful monitoring on top.

Despite that this is an early release of Rain, it is a fully functional project that can be used out-of-the box. Being aware that there is still a lot that can be improved and added, we are looking for external users and collaborators to help to move this work forward. Talk to us online at Gitter or via email and let us know what your projects and use-cases need, submit bugs or feature requests at GitHub or even contribute with pull requests.

Features

Dataflow programming. Computation in Rain is defined as a flow graph of tasks. Tasks may be built-in functions, Python/C++/Rust code, or an external applications, short and light or long-running and heavy. The system is designed to integrate any code into a pipeline, respecting its resource requirements, and to handle very large task graphs (hundreds thousands tasks).
Easy to use. Rain was designed to be easy to deployed anywhere, ranging from a single node deployments to large-scale distributed systems and clouds ranging thousands of cores.
Rust core, Python/C++/Rust API. Rain is written in Rust for safety and efficiency and has a high-level Python API to Rain core infrastructure, and even supports Python tasks out-of-the-box. Rain also provides libraries for writing own tasks in C++ and Rust.
Monitoring. Rain is designed to support both online and postmortem monitoring.

Documentation

Overview • Quickstart • User guide • Python API • Examples

Quick start

Download binary

$ wget https://github.com/substantic/rain/releases/download/v0.4.0/rain-v0.4.0-linux-x64.tar.xz
$ tar xvf rain-v0.4.0-linux-x64.tar.xz

Install Python API

$ pip3 install rain-python

Start server & a single local governor

$ ./rain-v0.4.0-linux-x64/rain start --simple

Rain "Hello world" in Python

from rain.client import Client, tasks, blob

client = Client("localhost", 7210)

with client.new_session() as session:
    task = tasks.Concat((blob("Hello "), blob("world!")))
    task.output.keep()
    session.submit()
    result = task.output.fetch().get_bytes()
    print(result)

Installation via cargo

If you have installed Rust, you can install and start Rain as follows:

$ cargo install rain_server

$ pip3 install rain-python

$ rain start --simple

Read the docs for more examples.