Node-DC/Node-DC-EIS

Client is not nicely parallelizable

Opened this issue · 0 comments

runspec.py uses eventlet for concurrency, which uses green threads under a single Python process (under the GIL). Thus only one core will be used by a single runspec.py instance. To saturate the server I need to use multiple cores.

There appears to be a multi-instance mode for the client (runspec.py) but it is undocumented. It does seem to be used in run_multiple_instance.sh though.

I prefer a "one-stop shop" client like ab that handles multiple threads itself.

I created a patch that uses the multiprocess module for the requestBased mode here. Each process runs the existing client logic (eventlet pool, etc.), and runspec.py then merges the resulting log files for a comprehensive summary.

Thoughts? Interested?