Chianti is taking more threads than specified
Closed this issue · 2 comments
Hello, I am currently running chianti 14.1 on a server to calculate spectra with the mspectrum routine. The problem is that 5 to 10 minutes after I run my python script on the server, python starts to use all of the 48 threads available in the server, instead of 5 threads as I am specifying in my python script. This is causing really long computing times. Is this a problem with the mspectrum routine? currently, I am calculating just one spectrum.
To the mspectrum function I am passing an emission measure array with 100 bins in temperature and I am running it for a wavelength range of 100 points. I am calculating it for ~ 20 elements and I'm including both the continuum and emission lines.
To have a reference, on my MacBook air with an M1 3.2 GHz processor this same routine takes approximately 40 mins to run, without using the mspectrum function, which apparently does not work on mac.
Does any one have encountered a similar problem?
I do not have any good ideas of how to solve your problem. mspectrum uses the multiprocessing class and the Queue method to create a queue and the Process method to send a job, one by one, to the available cores. It does not use threads. multiprocessing is aware of how many cores are available and should limit itself to those (as I understand it).
I do my computing on 2 linux computers running on intel cpus with 4 and 6 cores. I use Python3.6 and 3.10. So, I do not have any experience with M1 processors. However, it sounds like you are trying to run this on a server with a large number of cores. Have you tried running a shorter problem on you MacBook? I assume that it has more than a single core.
To follow up, what is the configuration of your server?
Also, you might try ipymspectrum where you start a certain number of cups given by n
> ipcluster start --n =4
where you can specify n to fit your case