Parallelization on a single node level

[TobiasDuswald]

Dear Maestro-developers,

I wanted to use Maestro in a research project that involves the calibration of a stochastic model. The model is rather cheap (~ 15 seconds, single core, does not scale) and I wanted to see if I can speed up the sensitivity analysis with Maestro. I have access to a 72 core machine and my hope was to run 72 models with different parameters in parallel to accelerate the process. I'm sure Maestro does introduce some overhead but I was hoping that the factor 72 outweighs it. I've managed to get everything running and assumed that blocks with no dependences should be executed in parallel but it turned out out that the models are executed sequentially. I reckon that was a little naiv. Anyways, my basic setup is the following at the moment (running locally on my 8 core notebook):

batch:
    shell: /bin/zsh

study:
    - name: compile-forward-model
          description: Does not matter here
          ...
      
    - name: run-simulation
          description: Run simulation for given parameter sets
          run:
              cmd: |
                $(compile-forward-model.workspace)/build/my_model <my_param_1> <my_param_2>
              depends: [compile-forward-model]

I've checked the repository and the documentation but was not able to find an answer to my question. So I wondered if Maestro supports the kind of parallelism that I'm looking for, and if so, what kind of keywords do I need to include to end up where I want to be? Basically locally with n_workers = n_processors.

Thank you very much for your help & kind regards,
Tobias

[FrankD412]

Hi @TobiasDuswald --

At the moment, Maestro does not support local parallel processes concurrently. I had explored allowing Maestro to do so previously, but due to constraints in time and the refactoring it would require I had to shelve it. I was planning to use part of another package I had started writing called pyaestro. In pyaestro, there's a parallelized executor based on the concurrent futures ProcessPool that allows local execution to behave more like a scheduled task. You can find that here -- you may be able to come up with something that works within a step that uses the Executor class.

I recently changed positions and try to maintain Maestro in my free time, so I may revisit the refactor soon; however, I am still focused on learning my new role so that might be a little ways out. Please let me know if you have any further questions or need clarification on anything else. :-)

[FrankD412]

Additionally, @jwhite242 is looking at making an in-allocation local execution. See PR #337 if you'd like to post some requirements for it.

[TobiasDuswald]

Hi @FrankD412,

Thank you very much for getting back to me so quickly; I appreciate it. Also, thank you for the references - I'll take a look and evaluate how I can approach the problem with it. Maestro is a very cool project, great job bringing it and keeping it alive! 👏 Hope you have a good start in your new job position.

[FrankD412]

@TobiasDuswald -- No problem! I'm happy that folks in the community are finding use for it! It's why I made it, in hopes that it would make life easier for folks :-) And thanks you for the well wishes!