[benchmark] benchmark for various thread pools, but I am not sure if it is rigorous enough

[KonghaYao]

The result is below

Running "Worker Pool in Nodejs" suite...
Progress: 100%

  WorkerPool:
    31 ops/s, ±0.52%   | 3.13% slower

  TinyPool:
    32 ops/s, ±0.75%   | fastest

  Piscina:
    17 ops/s, ±5.95%   | 46.88% slower

  Threads:
    14 ops/s, ±1.17%   | slowest, 56.25% slower

Finished 4 cases!
  Fastest: TinyPool
  Slowest: Threads

My source code is here. https://github.com/KonghaYao/workerpool/tree/benchmark/benchmark
Every library need to run 40 tasks. The task is to have a function in the worker that takes a string as input and returns it reversed to main thread.

But I am not sure if it is rigorous enough, can anyone help me? 😊

[josdejong]

I think in a good benchmark, you can distinguish:

• Real world task: executing a real world case (i.e. a CPU heavy task, that's what you use a thread for normally) either in the main thread vs a worker thread.
• Worker overhead: get insight in the communication overhead introduced by using a tread: compare executing a negligible task in the main thread vs in a worker thread.
• Starting up a worker thread is heavy. Give insight in how long it takes to start a thread, and separate this clearly from executing a task.

In your current benchmark, you have a negligible task, and you mix starting up new threads with actually executing the task. Since reversing a short string takes 0ms and starting a worker takes something like 50ms, I expect you're mostly measuring starting up 4 or 8 workers right now (depending on how many CPU's you have). I don't think that is your intention?