Whats next?
rolandtritsch opened this issue · 5 comments
Hi @rahulmutt,
with the latest merge we are at a point where we can reliably build and run the benchmarks.
The loose ends are ...
- investigate and fix that last/remaining tests that fail
- investigate and fix the problem(s) with the (failing) output checks
- investigate and fix all the new problems (e.g. NPEs) that come with 0.8.6b5
... but I am not sure, if this is time well spend, because right now for me the biggest gap is that we are running the benchmarks and we are getting a lot of data, but no insight.
To fix this, I am proposing to add an ability to write the benchmark results to a database so that we can query the database to compare two runs.
This would allow us to compare versions of eta. Lateron we could add tests to compare eta with scala and haskell.
WDYT?
@rolandtritsch That's been a feature I've been wanting for a while! Please go ahead and do it.
A start would be to explore the JMH options that export the results as csv - machine-readable I think they call it.
You don't even need to add a database. Simply collect all the results into a single file and write a tool that can compare two benchmark results files. You can configure CircleCI to save the artifacts of a build - declare the single file as an artifact to be saved.
Ok. I will investigate and get back to you with a design doc.
Hi @rahulmutt. Quick update. Another project got in the way. Will not be able to spend significant time on this in the next 2 month, but will chip away on it as good/much as I can. - RT
@rolandtritsch Sounds good, thanks for the update.