JohannesBuchner/BXA

How to use more than one core for computation?

Closed this issue · 5 comments

alexkolo commented

At the moment BXA (actually pymultinest) only uses one core. How can I make it use more than one?

JohannesBuchner commented

Install MPI, recompile Multinest to make sure you have libmultinest_mpi.so. Install the python package mpi4py. Then run with "mpiexec -np 4 python3 myprogram.py".

alexkolo commented

I was afraid you will say that. The problem is that "myprogram.py" was not designed to be running in parallel and adjusting it is quite the effort.
I hoped that one could just use Pool from multiprocessing for just executing a BXA function but I guess that's not an option :/
PS: Thx for the super quick feedback!

alexkolo commented

I managed to rewrite my code in order to make use of the multiprocessing function of MulitNest with the help of mpiexec. One important step was that I needed to (re)build MulitNest with the exactly same MPI package, which is used by the python packaged mpi4py. (in my case that would be cmake -DCMAKE_PREFIX_PATH=~/Software/Science/anaconda3/envs/science .. && make )
I also optimized the BXA code (editing standard_analysis and nested_run functions), since only when executing pymultinest.run I need more than one process. I placed a MPI.COMM_WORLD.Barrier() just before executing pymultinest.run to make sure that all processes are at the same step but I'm not 100% sure if that's really necessary.

I'm still rather new to python, made my transition from IDL less than a year ago. Hence, I'm yet not familiar with forking and updating code on GitHUB. Otherwise, I would have uploaded my version of BXA here. In any case, I would like to use the opportunity here to thank you for building such an amazing tool. It's super convenient and handy.

However, I notice that the computation time doesn't reduce linear with the number of process used. For instance, when using 7 cores/processes the time is only reduced by a factor of 3 in comparison to using one core/process.
Is this to be expected?

JohannesBuchner commented

Glad you got it working.

The steps to contribute changes are:

  • Make a fork (top right button on github repo page)
  • clone it locally (git clone with your repo URL, which you can find with the green clone button on your fork)
  • make changes & do git commit.
  • "git push" to upload it to your github fork
  • open a pull request on github so I can merge it to my repository.

If this is too complicated, you can also email me (or attach here) the output of git diff.

  • 👍1
JohannesBuchner commented

And thank you for the compliments. I still feel BXA should be 1) easier to install, 2) have more functionality (for visualization, model checking) and 3) be easier to understand for new-comers. If you have any suggestions, contributions and changes are very welcome.

  • 👍1