issues in ets_test benchmark
Closed this issue · 2 comments
See bencherl/bench/ets_test/src/ets_test.erl for the referenced code.
main bug in ets_test
The ets_test microbenchmark has a severe bug that is not catched by bencherl:
When running concurrently, the w-workers will inserts numbers from N down to 1, while the r-workers will lookup these same values from N down to 1.
If one r-worker does this faster, the match in line 75 will fail, and the worker process wil crash (badmatch). This will then bring the main Process down, as they are linked, and subsequently bring the entire benchmark to a hold.
bencherl does not catch that the benchmark application crashed, which probably could be changed to find such errors more easily on future benchmark applications.
Thus this benchmark currently measures how long it takes for a r-worker to overtake all w-workers.
The easiest way to solve this problem would be to just remove the match in line 75, but reversing lines 65 and 66 would also be an option. However, this would change the semantics of the benchmark even more than the easier change.
other problems
Another problem with the ets_test benchmark is that it uses
T = ets:new(x, [public]),
in line 43, which creates an ETS table of type set with read_concurrency and write_concurrency both set to false.
This somewhat counteracts any kind of scaling behaviour expected from this benchmark.
As there are more issues to the usefulness of this benchmark, like all workers having the exactly same access pattern to the table, the long term solution maybe is to replace it with some more expressive ETS benchmarks. I will create a pull request for mine once it is stable.
main problem fixed in 8d35ece
concurrency problems fixed in b11637c