etcd-io/bbolt

Introduce benchmarking for PRs

ivanvc opened this issue · 6 comments

ivanvc commented

Formalizing the comment from PR #691 (comment). Adding a GitHub action or Prow Job to compare benchmarks vs. main would help catch issues sooner, like the one tracked in #720.

@ahrtr, a couple of questions to finalize scoping this task:

  • What would be a good set of benchmark parameters to run?
  • Should this run on GitHub Actions or as a Prow Job?
  • What would be the priority for this task?
ahrtr commented

Thanks @ivanvc for raising this issue. Please see my response below,

  • What would be a good set of benchmark parameters to run?

For the writing test, I think we should commit the TXN at least multiple times, i.e 10 times. So options.Iterations/options.BatchSize > 10. For the options.BatchSize, it should be big enough, i.e. at least 10K.

  • Should this run on GitHub Actions or as a Prow Job?

We can add it on github action firstlyh. Eventually we may want to migrate all workflow checks to Prow.

What would be the priority for this task?

It may not be a urgent task.

fuweid commented

Eventually we may want to migrate all workflow checks to Prow.

Is it any reason to move workflow to Prow?

ahrtr commented

It's part of the effort of kubernetes/k8s.io#6102.

At least the Prow has more powerful machine, we can have shorter running time for the performance test. Please see #691

fuweid commented

It's part of the effort of kubernetes/k8s.io#6102.

At least the Prow has more powerful machine, we can have shorter running time for the performance test. Please see #691

I was thinking that github action can provide VM which supports nested virtualization and we can simulate real power-failure. And it's more easy to debug CI changes in GitHub action. Just my two cents. Anyway, thanks for sharing the background.

ivanvc commented

The initial implementation in #750 is benchmarking only the default write and read modes (write=rnd, read=seq).

@ahrtr, should we expand the benchmarks to other combinations?

ahrtr commented

@ahrtr, should we expand the benchmarks to other combinations?

Yes, it's a good point. etcd is actually sequentially writing the keys. We need to consider different write modes and read modes, also different key & value sizes. Usually K8s's value size is bigger than 1K+? Could you draft a simple doc /doc/benchmark.md firstly? thx