The benchmarks are utterly broken and always have been
Closed this issue · 2 comments
I wanted to test by own memoizing library against other ones to compare my speed and I wanted to try out your benchmarks. All the functions which you are testing are incorrect. You are using fibonacci functions which call themselves and are later memoized, but the non-memoized versions are called. You didn't notice because all your benchmarks are with input 35 which is low enough for the slowdown not to show.
const fib = n => n < 2 ? n : fib(n -1) + fib(n - 2)
const memo = mMemo(fib)
As you can clearly see - fib is calling fib, not mMemo. You don't have memoization at all.
const fib = mMemo(n => n < 2 ? n : fib(n -1) + fib(n - 2))
That is the correct way.
Preface
Before we dive in, please understand that this reads very aggressively. I welcome and appreciate anyone motivated to create their own library, but attacking someone else in the community is a very negative way to approach things.
Accuracy of claim
Where it is true
- When the cache size is greater than 1 (sometimes)
There is the potential to optimize future calls (not guaranteed) based on the chance of a previous recursive call leveraging the same parameter as the top-level entry. The chances of this become less and less likely as we extrapolate this to real-world implementations, which may use multiple parameters that are also complex objects.
Where it is not true
- When the cache size is 1
The first call would use uncached versions of each executionary step, and then cache the final result, which is the same result as using the unmemoized versions internally. It would actually be slower to do what you describe because if the internal caching mechanics storing values and immediately throwing them away. A large number of memoization libraries use a cache size of 1 (memoize-one for example), and micro-memoize is another case (as default).
- Third-party code
What you describe works perfectly fine if you want to memoize nothing but your own functions (because you control the internals), but as soon as you no longer control the internals your test is invalid as a comparison from library to library
General benchmark guidelines
Not an absolute truth
Benchmarks are relative and to be taken with a grain of salt. These values differ per box, per system load, per operating system ... the number of variables are huge. I provided the benchmarks less as a point of which is fastest in which scenario, and more to showcase how speed is fairly consistent between common use-cases.
Benchmarking your setup
It should be whatever you want to achieve. If you want to follow the example of micro-memoize (highlight common use-cases, and the speed difference between them) or go for naive-but-blazing-fast (optimize for the single number parameter use-case), that is up to you. The goals for your library are your own.
Conclusion
I wish you luck on your journey in creating your own library and benchmarking scenario, but please in the future try to keep your discussion with other developers civil. It will behoove you even in scenarios where you aren't glaringly wrong.
Ok, first of all - I was a little mad when I saw that these benchmarks are being used for multiple libraries (namely your 2 libraries and nano-memoize), so I am sorry if I offended you. It was not my intention at all.
About "Where it is true":
I may be mistaken but I think it will always be true, because you want for the other parameters to be cached as well, thus levereging the cache more.
About "Where it is not true":
I don't see the point in testing specifically the described fibonacci function on libraries which support only 1 cache size. It won't benefit from the caching at all and it is obviously not the use case for those libraries - the function called with or without the memo wrapper has the same performance.
I agree on the third-party code, but if that is the case, why not just write any generic function which takes a while, but use fibonacci which is obviously meant to be used with caching internally (to avoid memory leaks and boost performance) when using recursion?
About benchmarks:
I understand that benchmarks are not the ultimate truth, but they mean something when the library is about caching, which is, after all, about performance. Comparison between different caching libraries is something to consider when choosing one.
When you read this, please don't think that I am being rude, because it is not my intention at all and I am sorry if I come off like that.