Significant problem with results
Closed this issue · 5 comments
Hi Ram,
There is a significant (and I'm sure unintentional) problem with the results that you have posted.
The slower contenders have 1000 users right from the get-go. But the faster contenders increase users only gradually, up to about 100 users and never reaches 1000. You must have told locust to add 1 user/second. This is clearly visible from the "Number of users" graph in each report.
For example, your report lists Lua as having a 50%tile response time of 37, whereas if you run the test with 1000 users, the same result is 280 -- quite a large difference.
The incorrect reports affected by this are: lua, php, ruby, C++, rust, and M (yottaweb only).
What I suspect you'll find is that this change will mean that Go will come in as faster than Lua.
I also had trouble getting repeatable results when running on gitpod because gitpod loading varied widely, and I have no control over it.
Because of this, I re-ran all the tests sequentially on my laptop -- at least I know it's not loaded. To do this, I created a script to automatically run the test for each language in succession. You can see this script at my fork of this project.
Here are my corrected results for all languages. These fix the error above so that all language tests start with 1000 users up-front, and the tests are also run sequentially on my unloaded laptop.
Hi @berwynhoyt , many thanks for taking the time to assess this repo and suggest/make improvements. These are much appreciated.
Is there any chance that you could accept my Pull Request that solve this issue? It's been sitting there for a long time now.
Changes have now been merged and so am now closing issue.
Much appreciated!