MPI or multiprocessing

Question

MPI or multiprocessing

Closed this issue 4 months ago · 5 comments

gwm17 commented 4 months ago

To be cluster or not to be cluster that is the question

Answer 1 · 2024-08-26T14:17:35.000Z

Auxiliary: psutil >= 5.9.8 and tqdm >=4.65.0 if same code on Polaris

Answer 2 · 2024-08-26T20:52:13.000Z

After some discussion, for now we carry on. Talking to some polaris folks to see what the deal is

Answer 3 · 2024-08-30T17:21:00.000Z

Better yet, after talking to the folks at polaris, we can drop in replace using dragon which basically layers over multiprocessing with node-awareness

Answer 4 · 2024-08-30T17:21:52.000Z

Just need to find out if/where it is installed on Polaris. Also, will need to remove the SharedMemoryManager in favor of explicit memory management (but still SharedMemory)

Answer 5 · 2024-09-11T20:02:26.000Z

Calling this closed. Talked to HPE-Dragon team and things look g2g there.