Integrate with ann-benchmarks as primary benchmarking tool
alexklibisz opened this issue · 0 comments
alexklibisz commented
Background
The current custom benchmark suite is difficult to use, maintain, and share results. A custom solution made sense originally, as it was intended for large-scale distributed benchmarks, e.g., in Kubernetes. However, it turns out that local, single-node benchmarks are really all we need to tell if performance is improving or degrading, and should be simpler to maintain.
Ann-benchmarks is the defacto standard benchmark. It provides useful feedback and is reasonably simple to run on a single node, e.g., on a developer laptop. So, we'll switch to using ann-benchmarks as the primary benchmarking tool for Elastiknn.
Deliverables
- Add ann-benchmarks as a git submodule.
- Taskfile tasks to setup the submodule and environment.
- Taskfile tasks to run containerized fashion MNIST benchmark.
- Taskfile tasks to execute MINIST benchmark against an elasticsearch instance on localhost:9200.
- Taskfile tasks to generate markdown file with latest benchmark figures and tables.
- Tests in CI to validate the benchmarks are working on a small dataset.
- Remove the existing Scala benchmarks.
- Remove the existing ann-benchmarks example.
Related Issues
Several previous benchmarking iterations: