500 errors are not logged as request failures
jgrevich opened this issue · 0 comments
jgrevich commented
Summary
I noticed a number of 500 errors while running a load test however I did not see them reported as request failures in the results summary.
Detail
I started the SP based sign in test using the following configuration:
NUM_USERS=100000 SP_HOST=https://sp-oidc-sinatra.pt.identitysandbox.gov locust --locustfile load_testing/sp_sign_in.locustfile.py --host https://idp.pt.identitysandbox.gov --users 50 --hatch-rate 10 --run-time 30m --headless
While running I noticed repeating 500 errors:
[2020-10-05 14:36:43,524] j12/ERROR/locust.user.task: 500 Server Error: Internal Server Error for url: https://sp-oidc-sinatra.pt.identitysandbox.gov/auth/result?code=c409a63a-614a-48f9-874f-c488cfea0bea&state=b6426c42524518350dff2ea8a9d368cf
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 284, in run
self.execute_next_task()
File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 309, in execute_next_task
self.execute_task(self._task_queue.pop(0))
File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 321, in execute_task
task(self)
File "/Users/jjg/Documents/sources/login/identity-loadtest/load_testing/sp_sign_in.locustfile.py", line 9, in sp_sign_in_load_test
flow_sp_sign_in.do_sp_sign_in(self)
File "/Users/jjg/Documents/sources/login/identity-loadtest/load_testing/common_flows/flow_sp_sign_in.py", line 69, in do_sp_sign_in
resp = do_request(
File "/Users/jjg/Documents/sources/login/identity-loadtest/load_testing/common_flows/flow_helper.py", line 26, in do_request
resp.raise_for_status()
File "/usr/local/lib/python3.8/site-packages/locust/clients.py", line 22, in raise_for_status
Response.raise_for_status(self)
File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://sp-oidc-sinatra.pt.identitysandbox.gov/auth/result?code=c409a63a-614a-48f9-874f-c488cfea0bea&state=b6426c42524518350dff2ea8a9d368cf
However, no corresponding failures are reported in the summary as expected.
Name # reqs # fails Avg Min Max | Median req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
POST / 592 0(0.00%) 1572 1281 2322 | 1500 1.40 0.00
POST /login/two_factor/sms 568 0(0.00%) 18148 622 28144 | 20000 0.10 0.00
POST /sign_up/completed 26 0(0.00%) 20044 10213 26638 | 20000 0.00 0.00
GET https://idp.pt.identitysandbox.gov/openid_connect/logout 564 0(0.00%) 11728 5115 24145 | 9400 0.20 0.00
GET https://sp-oidc-sinatra.pt.identitysandbox.gov 608 0(0.00%) 4744 612 13295 | 4200 2.00 0.00
GET https://sp-oidc-sinatra.pt.identitysandbox.gov/auth/request?aal=&ial=1 596 0(0.00%) 7061 735 13325 | 7400 1.20 0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Aggregated 2954 0(0.00%) 8621 612 28144 | 6900 4.90 0.00
Steps To Reproduce
See above.
Expected Behavior
500 errors should result in a request failure and logged to the results.
Actual Behavior
500 errors are ignored and I assume the request is retried.
Suggested Actions
We should make sure the 500 errors (and any others) are logged and counted as request failures. Perhaps we should have some way of confirming the request counts by type using a separate system.
Acceptance Criteria
- 500 errors are logged as request failures