r-lyeh-archived/scriptorium

Add more benchmarks

Opened this issue · 1 comments

cgbystrom commented

Right now the benchmarks look like they are all based on a simple fibonacci test, which I guess is a good start considering the number of runtimes being tested.

But just relying on this single type of benchmark is a bit one-sided and expanding the number of benchmarks is needed for a more accurate picture.

Perhaps we can list a few good candidates here. Two good sources I found are
https://github.com/kragen/shootout/tree/master/bench and https://github.com/attractivechaos/plb

Would be nice to pick something easy to implement but at the same time showing a different execution pattern than the fibonacci test.

r-lyeh-archived commented

Great links thanks!

Yeah absolutely. The benchmark/project is more like a stub rather than a complete benchmark to be honest.

At least for me, I would like to bench/add:

  • string, list, vector, hash benchmarks { insert, delete, iteration }
  • script->native round trip
  • native->script round trip
  • memory consumption
  • memory de/allocation // GC times
  • warm up/spinning time
  • binary size overhead

Despite the good intentions, only the most common languages are production ready. Many of the languages listed in this repository lack objects, strings, memory functions, timers, collections/containers and so on. It is almost impossible to replicate the same tests for all of the languages without proper source modifications (heavy modifications in many cases).

Fibonacci on the other hand is almost portable and universal, and it is a good indicator to discard a few languages that are too slow (compared to the others) or those that do not provide tail/call optimization at all (for example, I wouldnt use them if I am coding a procedural game or city generator :). If there are other good universal tests that you might be aware of, I am open for them (simple hashing/encryption comes to mind).

Also:

  1. I plan to evaluate a fib(0) test and subtract the time from the total already, so we retrieve the warm up time for those languages that require some initialization.

  2. If there is some kind of idea on your side to do a fair scoring system (for those incomplete languages) I'm willing to collaborate and update the bechmarks/tests with new formulae.

  3. I am also fond of merging pull requests from anyone, so feel free to start experimenting asap :)

Thanks for the interest too :)