Add pipeline and run configuration
Closed this issue · 0 comments
MaximLippeveld commented
There is an important distinction between pipeline configuration and run configuration. The pipeline configuration concerns things like what channels to load, which segmentation algorithm to use on what channel, which features to compute... The run configuration concerns things like how many workers to use, how much memory each worker receives, what logging level to use...
Pipeline configuration should be implemented with a YAML config file that can be passed when launching the command line interface (CLI) on the command line. Example:
data_loading:
file_format: multiframe_tiff
channels:
- 0
- 1
- 2
- 4
- 5
segmentation:
noisy_channels:
- 0
- 5
The run configuration will be passed to the pogram through arguments of the CLI. Example
sip --n-workers 4 --worker-mem 20G
Argument handling is done with the click
Python library.