Pipeline is a task execution engine similar to Make.
Tasks are defined in task generator scripts in Python (see section below).
Running pipeline executes the task generator task.py in the current dir, and then executes all generated tasks. Use pipeline -g <generator-file> to specify a different generator file.
Additional arguments to pipeline are passed on to the task generator, and become available as pipeline.args (see details below).
Current directory is working directory for all tasks, regardless of what generator file they come from.
When a task generator script is executed, pipeline is defined as an instance of class Pipeline, which facilitates generating a task list file.
pipeline.args is a list of additional arguments passed to the pipeline program. For example executing pipeline a b c will set pipeline.args to ['a', 'b', 'c'].
Example tasks.py:
# Access additional arguments:
print("Args: "+ str(pipeline.args))
# Get pathlib.Path objects for relative paths
# currentDir is location of current task generator file
input_a = pipeline.current_dir() / 'input.txt'
# equivalent to 'input_a'
input_b = pipeline.relativePath('input.txt')
# rootDir is location of first task generator file
input_c = pipeline.root_dir() / 'input.txt'
# Add a task.
pipeline.add({
'name': 'my-task1',
'command': 'cp {} output.txt'.format(input_a),
'input_files': [ str(input_a) ],
'output_files': [ 'output.txt' ]
})
# Add another task that depends on the previous task.
# In case dependencies between tasks can not be expressed through files,
# they can be explicitly added like this.
pipeline.add({
'name': 'my-task2',
'command': 'cat output.txt',
'dependencies': [ 'my-task1' ]
})
A task list is a JSON file generated by the class Pipeline using task generator scripts, and stored as .pipeline.json in the working directory.
The following task list is produced by the example generator in the previous section:
[
{
"name": "my-task1",
"command": "cp /root/dir/input.txt output.txt",
"input_files": [
"/root/dir/input.txt"
],
"output_files": [
"output.txt"
]
},
{
"name": "my-task2",
"command": "cat output.txt",
"dependencies": [
"my-task1"
]
},
{
"name": "..."
}
]