Paraffin, derived from the Latin phrase parum affinis
meaning
little related
, is a Python package designed to run DVC
stages in parallel. While DVC does not currently support this directly, Paraffin
provides an effective workaround. For more details, refer to the DVC
documentation on
parallel stage execution.
[!WARNING] Although DVC supports running multiple
dvc repro
commands simultaneously, spawning many workers that finish at the same time may lead to DVC lock issues and unexpected failures.
Install Paraffin via pip:
pip install paraffin
To use Paraffin, you can run the following to run up to 4 DVC stages in parallel:
paraffin -n 4 <stage names>
If you have pip install dash
you can also access the dashboard by running
paraffin --dashboard <stage names>
For more information, run:
paraffin --help
[!TIP] If you are building Python-based workflows with DVC, consider trying our other project ZnTrack for a more Pythonic way to define workflows.