Better benchmarks
Closed this issue · 5 comments
I found this jsperf benchmark, which I think is way too simple of a use case: http://jsperf.com/bluebird-vs-rsvp/89
So I added your library to the bluebird benchmark suite, which tests a set of small 1ms I/O operations instead of the synchronous .resolve(1).
To run them, just clone the repo, run npm install the main repo and the benchmark directory and run ./bench doxbee and ./bench parallel.
These are the result I am getting back - let me know what you think of these numbers:
Doxbee:
results for 10000 parallel executions, 1 ms per I/O op
file time(ms) memory(MB)
callbacks-baseline.js 227 43.60
promises-bluebird-generator.js 266 33.36
promises-bluebird.js 425 47.75
promises-cujojs-when.js 446 71.76
promises-tildeio-rsvp.js 520 66.28
promises-bluejava-zousan.js 551 71.25
callbacks-caolan-async-waterfall.js 692 76.12
promises-lvivski-davy.js 706 108.98
promises-dfilatov-vow.js 838 155.99
promises-calvinmetcalf-lie.js 858 165.59
generators-tj-co.js 1153 147.88
promises-ecmascript6-native.js 1288 176.08
promises-obvious-kew.js 1397 235.82
promises-then-promise.js 1650 217.07
promises-medikoo-deferred.js 1973 172.57
observables-Reactive-Extensions-RxJS.js 2895 287.86
promises-kode4food-welsh.js 3535 80.52
observables-pozadi-kefir.js 5909 189.24
observables-caolan-highland.js 6860 548.75
promises-kriskowal-q.js 8323 717.75
observables-baconjs-bacon.js.js 20751 846.07
Parallel:
results for 10000 parallel executions, 1 ms per I/O op
file time(ms) memory(MB)
promises-bluebird.js 491 106.86
promises-bluebird-generator.js 626 110.79
callbacks-baseline.js 670 42.71
promises-cujojs-when.js 950 164.30
promises-tildeio-rsvp.js 964 205.40
promises-bluejava-zousan.js 1050 212.48
callbacks-caolan-async-parallel.js 1248 214.15
promises-lvivski-davy.js 1702 267.55
promises-calvinmetcalf-lie.js 1844 354.80
promises-dfilatov-vow.js 2790 512.38
promises-ecmascript6-native.js 3227 519.22
promises-then-promise.js 3299 670.76
promises-kode4food-welsh.js 3725 210.52
promises-medikoo-deferred.js 3931 507.14
promises-obvious-kew.js 9455 960.55
Thanks Andreas - interesting. Looks like Zousan did fare, but not fantastic here. (Thanks for doing the work to test it!). I'm a bit surprised there is such a divergence here. jsperf is acting very buggy lately, but that test is also legit IMHO. resolve(1) is not synchronous - but the test is serialized promises, not parallel. So it is just testing a different use case. And of course, it is in browsers, not node. [but those funky results jsperf is showing is jsperf acting up - the tests used to work fine and it was a valid data point.
I think Zousan is pretty optimized. The one place I could see a further optimization is for the single listener case - which is fairly common. In that case, rather than define an array to hold the listeners, I could simply directly hold a reference to the single listener. And then if additional listeners are added, THEN create the array with the two (or more) listeners. But that would add some size to the module, and I was trying to keep it tiny. Also it would actually slow it slightly in the multi-listener case - so its not entirely a win.
Thanks again for your work on this Andreas!
No problem.
I think your arguments against speeding up the single listener case are valid. What I quite like about the doxbee and parallel testcases is that they are a lot more involved than the jsperf tests and the perf diff is probably due to more than the single listener case. I really like the idea of a small & fast promise library, so I was hoping that you'd find new inspirations to go even faster if you ever find the time to look at the results in more detail - no pressure, it's already plenty fast at it is :).
It seems I forgot to paste the link to these benchmarks, so here we go: https://github.com/alubbe/bluebird
To run them, just clone the repo, run npm install the main repo and the benchmark directory and run ./bench doxbee and ./bench parallel.
Thanks Andreas - and thanks for the link - (though I was able to find it before by poking around). And yah, I'd certainly be happy to make it faster. I could examine those tests closely and then probably find some tweaks to Zousan to perform better in those tests. But of course, that is only useful if those tests are representative of your real world application. (Which of course, it may well be).
Aso, any speed up in the Promise implementation will be diluted by the overall execution time of the full routine. And really, we are down to an array push, a for loop, and a couple function calls - so it would have to be some incredibly tight code surrounding the promise in order for even a large speed improvement to have any effect.
Of course, I'm sure know this! ;-) In fact, you sound a bit like I did - just wanting a super small and super fast promise so I never had to consider it when tuning for performance!
Haha, exactly - just think of it as something to for your intellectual curiosity if you're ever bored. Front-end promises don't get called nearly as often as the ones in the backend of some highly frequented service, so a 2-3x speedup is by no means essential to most applications.
Thanks Andreas - yah, I've done some NodeJS apps that heavily relied on async - would have been SO much better written with Promises. On the browser, usage is generally a bit lighter - but I now find myself looking for excuses to use them. ;-)