Parallelism!

[aleozlx]

• DAG workflow with a fork-join pattern
• Parameter grid (yaml tag annotation would actually help here)

[aleozlx]

Here's an idea for the parameter grid

steps:
- action: sys_fork
  states:
    from: named.yml # mutex this if it is some local file
  # stdout for each child process will be redirected to a separate file `./playbook-{name},{tid}.log`
  resource: # can only acquire one type of resource for now... deadlock danger...
    cuda_devices: ["0", "1", "2", "3"] # this also determines nproc
  grid: # cartesian product + a stable way of assigning tid
  - param1: [10, 20, 40, 80, 160]
  - param2: [0.03, 0.01, 0.003, 0.001]