sd37/hydroflow

Hydro's low-level dataflow runtime

Hydroflow

Hydroflow is a low-latency dataflow runtime written in Rust. The Hydro Project will empower developers to harness the full potential of the cloud by making distributed programs easy to specify and automatic to scale. Hydroflow serves as the lowest level in the Hydro stack, serving as a single-node low-latency runtime with explicit networking. This allows us to support not just data processing pipelines, but distributed protocols (e.g. Paxos) and real-world long-running applications as well.

Take a look at the Hydroflow Book.

The Hydroflow Surface Syntax

Hydroflow comes with a custom "surface syntax" domain-specific language which serves as a very simple, readable IR for specifying single-node Hydroflow programs, intended to be stitched together by the Hydro stack to create larger autoscaling distributed systems.'

Here's a simple example of the surface syntax. Check out the Hydroflow Playground for an interactive demo.

source_iter(0..10)
  -> map(|n| n * n)
  -> filter(|&n| n > 10)
  -> foo;

foo = map(|n| (n..=n+1))
  -> flatten()
  -> for_each(|n| println!("Howdy {}", n));

For more, check out the surface syntax section of the Hydroflow book.

Start with a Template Program

We provide a cargo-generate template for you to get started from a simple working example.

To install cargo-generate, run the following:

cargo install cargo-generate

Then run

cargo generate gh:hydro-project/hydroflow-template

and you will get a well-formed Hydroflow/Rust project to use as a starting point. It provides a simple Echo Server and Client, and advice for adapting it to other uses.

Dev Setup

See the setup section of the book.

The Examples Container

The hydroflow/examples subdirectory of this repository includes a number of examples. To make running these examples in the cloud easier, we've created a Docker image that contains compiled versions of those examples. The image is defined in the Dockerfile in the same directory as this README.

If you want to build the examples container locally, you can run

docker build -t hydroflow-examples .

This will build an image suitable for your architecture.

The scripts/multiplatform-docker-build.sh <image name> script will build both arm64 and amd64 versions of the image and push them to the image name specified. By default, this will push the image to DockerHub; if you want to push the image to another repository, you can pass an image URL as the argument to multiplatform-docker-build.sh instead.

Example binaries are located in /usr/src/myapp.