amplab/snap

Performance tweaking wrt parallel runs

matthdsm opened this issue · 9 comments

matthdsm commented

Hi,

I'm trying to run multiple instances of snap (in docker) through a workflow manager. However, I'm noticing only one of the instances is using the full CPU capacity at the time.

Are there any settings to tweak so we can fully use the cpu's of the entire node?

htop screengrab

Currently, we're using default settings with sorting enabled on 18 threads per instance. I tried using -map- but that didn't help.

Thanks for the info
Matthias

matthdsm commented

I also noticed when monitoring the processes that each thread uses on average 30% with peaks to 50% of the allocated core capacity.

slw287r commented

Multiple snap-aligner instances will use up to the number of threads specified by -t. Try -t 72 for each instance to see the difference.

matthdsm commented

So to be clear:
when I run 3 instances with each 18 cores allocated (-t 18), all three instances will be limited to the same 18 cores?
I would assume they each take up the full CPU of the 18 allocated cores, so 18 cores/instance , thus filling the entire node.

bolosky commented
matthdsm commented

Oh wow, adding -b- has a tremendous effect on efficiency! I can now see all fully cores used.
Is there a reason this isn't default behaviour?

bolosky commented
matthdsm commented

I'm working on a heterogenous cluster through a workflow manager, so the number is threads is set before scheduling.
This means I can't use the full node, since I've no idea where it will be scheduled.
I'm thinking we only see about 30% usage per thread since there were 3 instances running on the same 18 cores, which would mean all 3 instances are competing for the same CPU.

Background info (for this node):

  • Ubuntu 18.04.5 LTS
  • Intel(R) Xeon(R) CPU E5-2698
  • CephFS backend
bolosky commented
matthdsm commented

Allright, thanks for all the help!