bodywork-ml/bodywork-core

Enable the use of `requirements.txt` files to specify stage dependencies

AlexIoannides opened this issue · 0 comments

"As a ML Engineer, I would like to be able to specify the requirement for a pipeline stage by referencing a requirements.txt file, so that I do not have to specify requirements in YAML that can be very clunky and lead to bloated bodywork.yaml files."

Tasks

  • Extend the YAML schema defined in bodywork.config so that a path to a requirements.txt files can be specified for each stage's dependencies, that overrides any other dependency.
  • Extend the validation of bodywork.yaml so that it can check for the existence of any requirements.txt files.
  • Extend bodywork.stage_execution so that it can install requirements from the requirements.txt files provided in the config.
  • Add unit tests to cover this new functionality.