Simpleflow is a Python library that provides abstractions to write programs in the distributed dataflow paradigm. It coordinates the execution of distributed tasks with Amazon SWF.
It relies on futures to describe the dependencies between tasks. A Future object
models the asynchronous execution of a computation that may end. It tries to mimic
the interface of the Python concurrent.futures library.
- Provides a
Futureabstraction to define dependencies between tasks. - Define asynchronous tasks from callables.
- Handle workflows with Amazon SWF.
- Implement replay behavior like the Amazon Flow framework.
- Handle retry of tasks that failed.
- Automatically register decorated tasks.
- Encodes/decodes large fields to S3 objects transparently (aka "jumbo fields").
- Handle the completion of a decision with more than 100 tasks.
- Provides a local executor to check a workflow without Amazon SWF (see
simpleflow --localcommand). - Provides decider and activity worker process for execution with Amazon SWF.
- Ships with the
simpleflowcommand.simpleflow --helpfor more information about the commands it supports.
You can read more in the Features section of the documentation.
Please read and even run the demo script to have a quick glance of
simpleflow commands. To run the demo you will need to start decider
and activity worker processes.
Start a decider with:
$ PYTHONPATH=$PWD simpleflow decider.start --domain TestDomain --task-list test examples.basic.BasicWorkflow
Start an activity worker with:
$ simpleflow worker.start --domain TestDomain --task-list quickstart
Then execute ./extras/demo.
Read the main documentation at https://botify-labs.github.io/simpleflow/.