Consider implementing a way to postpone running benchmarks

[aochagavia]

There are a few scenarios where it makes sense to postpone running benchmarks, to avoid competing for resources and prevent noise in the results:

1. When manually running other performance benchmarks (like in this case).
2. When building and deploying a new version of the application (since the app is built on the server).

Currently, the only way to avoid running benchmarks in reaction to GitHub events is to shut down the application entirely. If we do so, however, GitHub events get lost, which is unfortunate. The current way to recover them is that someone goes to the rustls-benchmarking GitHub app's page and manually redelivers any relevant webhooks.

I'm tempted to leave this as wontfix, because I don't expect it will cause problems in practice. That way we avoid the extra complexity.

[aochagavia]

I've been thinking some more about how to approach this. The main challenge is IMO to come up with a suitable admin interface, exposing the "postpone benchmarks" feature in a secure way, while at the same time avoiding an explosion in complexity (e.g. a login system, HTML generation and form handling, etc). My current idea is to associate a systemd socket with the service, so maintainers with SSH access can send commands to the service's stdin (here is a SE question/answer showing a configuration in which echo "hello" > /run/user/1000/foo.stdin is sent to a service's stdin).

@cpu do you have thoughts on this?

[cpu]

My current idea is to associate a systemd socket with the service, so maintainers with SSH access can send commands to the service's stdin

That sounds reasonable if we were thinking there might be additional commands in the future. I think with this design we'd need ~2 commands at minimum: pause [on/off] and status to show what the pause toggle state is.

Alternatively, if this is likely to be the only administrative action we anticipate needing maybe we can get away with just using a canary file on disk in a well known location. Something like /home/ci-bench-runner/pause. If we touch that file, the system will be paused. If we rm it, the pause is removed.

[aochagavia]

I like the canary file alternative!

Regarding other possible commands, there is one that could be useful in the future: running the benchmarks for the current main branch, one or more times, and storing the results so they get picked up when deriving the significance thresholds. The only way to do this right now is by updating the main branch or by spoofing a GitHub webhook for that event (that's what I did last time, POSTing 30 times to the server, to initialize the database).

On the other hand, the command I mention would almost never be necessary (and there is a reasonable workaround for it), so we should probably go with the canary file. Agree?

[cpu]

so we should probably go with the canary file. Agree

Yeah, I think starting simple sounds good :-) We can always keep the stdin approach in our back pocket for the future.