Investigate possible performance degradation following PR #706
lindsay-stevens opened this issue · 0 comments
lindsay-stevens commented
Software and hardware versions
pyxform v2.0.3 at 1599d1f, Python 3.10, 3.11, 3.12
Problem description
Changes in #706 seem to have resulted in rather slower test runs than before.
It's not clear whether this is due to slower execution of source or tests (or both).
Pre 706, actions, Post 706 actions runs were roughly:
- macos: pre 2min 30s, post 2min 55s (+17%)
- ubuntu pre 55s, post 1min 25s (+55%)
- windows pre 2min 15s, post 3min 30s (+78%)
Also in the performance test in test_translations.py, test times increased since last run in this commit (Jan 2022):
- 500 questions: pre 1s, post 3s (3x)
- 1000 questions: pre 2s, post 10s (5x)
- 2000 questions: pre 4s, post 30s (7.5x)
Possible reasons (scope of #706):
- updated the Python versions from 3.7, 3.8, 3.9 to 3.10, 3.11, 3.12
- updated lxml from 5.1.0 to 5.2.2 (also ruff but that's formatting only)
- changes from
%
string formats to f-strings, or other format diffs
The performance test stats have additional factors that could explain the larger diffs:
- Prior stats updated over 2 years ago (many code changes)
- Laptop now 2 years crustier
- Ran on different VM setup:
- Half the vCPU and RAM allocated
- Dynamic RAM activated
- M.2 NVME -> SATA SSD
- CentOS 7 -> Rocky 9
Steps to reproduce the problem
Run the test suite, note longer execution time as described above.
Expected behavior
Little to no performance change (or improvement??).