Figure out if batch work for the cluster workers is even faster
balupton opened this issue · 0 comments
balupton commented
Right now we send each task for the worker individually, this may incur a communication cost, perhaps we can batch the tasks or queue the tasks so that instead of sending each task to the worker individually, we send a batch of 10 at a time, or however many we have queued up while the worker was running, this may reduce communication cost and increase performance at the cost of additional complexity.