Devise a test harness for testing performance

Question

Devise a test harness for testing performance

godlygeek opened this issue 2 years ago · 5 comments

We know there's low hanging fruit available for optimizing PyStack, but we currently have no good way to benchmark our performance and quantify any improvements. Design some sort of a test harness that can be used for measuring the performance impact of our changes, possibly using https://asv.readthedocs.io/en/stable/

Answer 1 · 2023-04-25T20:48:52.000Z

I'll look into this today and tomorrow.

Answer 2 · 2023-08-30T05:58:32.000Z

Wasn't able to look into this. Feel free to unsassign

Answer 3 · 2024-01-13T01:22:35.000Z

How is the impact of the performance impact of changes defined? Is it meant to be a time and space benchmark against a predefined use?
Also is it meant to be integrated into a github workflow or is it meant to be a separate module of its own?

Answer 4 · 2024-01-25T17:53:53.000Z

How is the impact of the performance impact of changes defined? Is it meant to be a time and space benchmark against a predefined use? Also is it meant to be integrated into a github workflow or is it meant to be a separate module of its own?

My understanding is that we want to have something similar to the CodeCov integration, where we can easily tell, for any given PR, if a number of predefined measurements differ when measured on main vs on the PR branch. This way, we can both have a better understanding on the current time and space performance of PyStack, and be aware if we are significantly changing them with a PR.

Any PR whose goal is to change performance would be close to meaningless without a way of measuring the difference.

Answer 5 · 2024-01-25T17:55:53.000Z

How is the impact of the performance impact of changes defined? Is it meant to be a time and space benchmark against a predefined use? Also is it meant to be integrated into a github workflow or is it meant to be a separate module of its own?

My understanding is that we want to have something similar to the CodeCov integration, where we can easily tell, for any given PR, if a number of predefined measurements differ when measured on main vs on the PR branch. This way, we can both have a better understanding on the current time and space performance of PyStack, and be aware if we are significantly changing them with a PR.

Any PR whose goal is to change performance would be close to meaningless without a way of measuring the difference.

Hi Gus,

I implemented it in the PR #165. Please have a look