Add a benchmark action

[vaind]

I've done away with the lambdas as well (these are now inline functions).

The tests seem to pass now. 🎈

So indeed it seems like it has been the performance issue. Maybe we should check check that we have some benchmarks for these critical paths and introduce https://github.com/marketplace/actions/continuous-benchmark to the repo. Although it looks like it doesn't support PRs from outside contributors at the moment.

Originally posted by @vaind in #3321 (comment)