aleozlx/playbook-rs

Parallelism!

Closed this issue · 1 comments

  • DAG workflow with a fork-join pattern
  • Parameter grid (yaml tag annotation would actually help here)

Here's an idea for the parameter grid

steps:
- action: sys_fork
  states:
    from: named.yml # mutex this if it is some local file
  # stdout for each child process will be redirected to a separate file `./playbook-{name},{tid}.log`
  resource: # can only acquire one type of resource for now... deadlock danger...
    cuda_devices: ["0", "1", "2", "3"] # this also determines nproc
  grid: # cartesian product + a stable way of assigning tid
  - param1: [10, 20, 40, 80, 160]
  - param2: [0.03, 0.01, 0.003, 0.001]