Ballista is a proof-of-concept distributed compute platform based on Kubernetes and the Rust implementation of Apache Arrow.
This is not my first attempt at building something like this. I originally wanted DataFusion to be a distributed compute platform but this was overly ambitious at the time, and it ended up becoming an in-memory single-threaded query execution engine for the Rust implementation of Apache Arrow. However, DataFusion now provides a good foundation to have another attempt at building a modern distributed compute platform in Rust.
My goal is to use this repo to move fast and try out ideas that help drive requirements for Apache Arrow and DataFusion.
This demo shows a Ballista cluster being created in Minikube and then shows the nyctaxi example being executed, causing a distributed query to run in the cluster, with each executor pod performing an aggregate query on one partition of the data, and then the driver merges the results and runs a secondary aggregate query to get the final result.
Here are the commands being run, with some explanation:
# create a cluster with 12 executors
cargo run --bin ballista -- create-cluster --name nyctaxi --num-executors 12 --template examples/nyctaxi/templates/executor.yaml
# check status
kubectl get pods
# run the nyctaxi example application, that executes queries using the executors
cargo run --bin ballista -- run --name nyctaxi --template examples/nyctaxi/templates/application.yaml
# check status again to find the name of the application pod
kubectl get pods
# watch progress of the application
kubectl logs -f ballista-nyctaxi-app-n5kxl- README describing project
- Define service and minimal query plan in protobuf file
- Generate code from protobuf file
- Implement skeleton gRPC server
- Implement skeleton gRPC client
- Client can send query plan
- Server can receive query plan
- Server can translate protobuf query plan to DataFusion query plan
- Server can execute query plan using DataFusion
- Create Dockerfile for server
- Ballista CLI - create cluster
- Ballista CLI - delete cluster
- Ballista CLI - run application
- Simple example application works end to end
- Add support for aggregate queries
- Server can return Arrow data back to the application (in CSV format for now)
- Example application can aggregate the aggregate results from each partition/node
- Write blog post and announce Ballista
- Distributed query planner
- Implement support for all DataFusion logical plan and expressions
- Server can write results to CSV files
- Server can write results to Parquet files
- Implement Flight protocol
- Support user code as part of distributed query execution
- Interactive SQL support
- Java bindings (supporting Java, Kotlin, Scala)
See CONTRIBUTING.md for information on contributing to this project.