Improve benchmarking

[ericcornelissen]

Engineering initiative

Description

Currently, the benchmarking suite simply measures the performance of escaping a hardcoded argument for each of the supported shells (pending #271).

This is useful, it allows for detecting performance issues with these functions - and these functions are the most interesting parts of the code. However, it leaves out the rest of the code base, which may also has an impact on performance.

Considerations

Consideration	Package API	Escape functions
It can be run on any system1	No	Yes
It makes no assumptions about the internals of the system2	Yes	No

Proposal

To be decided.

Footnotes

1. This is considered important because someone is ideally not excluded from running benchmarks altogether just because they don't have access to a specific OS. ↩

2. This is useful because it means the results are easier to compare over longer periods of time. If the internal architecture changes, the benchmark change and it may be hard to correlate the old and new results. ↩