BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.
Key Features (all provided out-of-the-box):
- Automatic resource creation / management (Infrastructure as Code) powered by Pulumi (Infrastructure from Code)
- Automatic parallelism & concurrency powered by Ray
- Dynamic autoscaling: scale up during high traffic / reduce costs during low traffic
pip install buildflowBuildFlow uses Pulumi to manage resources used by your BuildFlow Nodes and Processors. To install Pulumi visit: https://www.pulumi.com/docs/install/
Installing Pulumi unlocks:
- allows BuildFlow to manage resource creation and destruction
- full access to Pulumi API / CLI
- fine-grained control over Pulumi Stacks & Resources
- Docs: https://www.buildflow.dev/docs
- Walkthroughs: https://www.buildflow.dev/docs/walkthroughs/realtime-image-classification
- Discord: https://discordapp.com/invite/wz7fjHyrCA
- Contribute: https://www.buildflow.dev/docs/developers/contribute
We use black and ruff with pre-commit hooks to perform health checks. To setup these locally:
- Clone the repo
- Install the
devdependencies like `python -m pip install .[dev] - Check if pre-commit is installed correctly by running
pre-commit --version - Setup pre-commit to run before every commit on staged changes by running
pre-commit install - Pre-commit can also be ran manually as
pre-commit run --all-files