Processor limit?

[dustyny]

I've been testing Deco on a AWS instance with 36 cores and but Deco seems to max out at 16-18. Is there a limit to how many processes Deco can run?

Thanks

[dustyny]

Sorry my mistake.. The additional processes came up when my Database cluster scaled up

[alex-sherman]

Do you mean this is still an issue, or were you able to resolve it?

muliprocessing.Pool is supposed to default to the number of physical cores the machine has, but you can explicitly pass in a number of processes to deco:

@concurrent(processes=36)
def do_work():
   ...

You may also be hitting some other bottleneck in either deco, or multiprocessing.Pool. I've never attempted to run deco on anything with 36 cores myself, so seeing someone trying it is awesome!

[dustyny]

Thank you for getting back to me so quickly. I've had to manually set the number of CPUs. It didn't seem to auto-detect. I can try that again and see if is true or if I made a premature judgement there.

I didn't realize I was pointed to the wrong end point (write vs read) and my Database wasn't scaling up. Since the database server was maxed out at 100%, the other Deco processes were sitting idle. Once I correct that, the cluster scaled up and I was able to use all 36 processors..

Great work btw.. I've never had such an easy time scaling up python.. 👍