libp2p/test-plans

Automate performance benchmarking

mxinden opened this issue · 3 comments

mxinden commented

Overarching tracking issue: #63

#163 allows running the various perf protocol implementations (libp2p/specs#478) on cloud instances.

Ideally we have a CI setup, that allows users to trigger the above. This removes the need for users to run it on their local machine.

Workflow

Starting point: libp2p implementation maintainer wants to test a new release candidate.

  1. They open a pull request to libp2p/test-plans, adding their release candidate to perf/impl.
  2. They trigger the automation via a GitHub comment.
  3. CI runs:
    1. Generate an ssh keypair and set the public key at perf/terraform/user.pub.
    2. Run cd perf/terraform && terraform apply.
    3. Run cd perf/runner && npm run start -- --client-public-ip $(terraform output -raw -state ../terraform/terraform.tfstate client_public_ip) --server-public-ip $(terraform output -raw -state ../terraform/terraform.tfstate server_public_ip). This will write the benchmark results to perf/runner/benchmark-results.json.
    4. Run cd perf/terraform && terraform destroy.
    5. Push perf/runner/benchmark-results.json to the pull request.
  4. Depending on the benchmark results:
    • In case the automation catches a regression, the maintainer can cut another patch release and update the libp2p/test-plans pull request and retrigger the automation.
    • In case the results are fine, the maintainer can cut the release. Once released, they update the libp2p/test-plans pull request, retrigger the automation and merge. Thus libp2p/testplans master branch always contains the latest benchmarking results.
mxinden commented

//CC @galargh I extracted the automation section from #163 to this new issue.

  • 👍1
galargh commented

Thanks :) Here's the first pass at the automation - mxinden#2

mxinden commented

First successful automation run on libp2p/test-plans happened today. See #63 (comment). Thus merging here.

Thank you @galargh!

  • 🚀1