unexpected failures with prod_simulator flow
jgrevich opened this issue · 1 comments
jgrevich commented
Summary
The prod_simulator flow has a significant number of unexpected failures.
Detail
Name # reqs # fails | Avg Min Max Median | req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET / 357 0(0.00%) | 181 117 829 140 | 1.19 0.00
POST / 323 2(0.62%) | 809 618 1886 720 | 1.08 0.01
GET /account 223 36(16.14%) | 205 136 789 170 | 0.74 0.12
POST /login/two_factor/sms 37 1(2.70%) | 354 295 618 340 | 0.12 0.00
GET /logout 187 0(0.00%) | 151 112 779 130 | 0.62 0.00
GET /phone_setup 32 0(0.00%) | 188 138 452 160 | 0.11 0.00
POST /phone_setup 32 0(0.00%) | 540 454 1120 490 | 0.11 0.00
POST /sign_up/create_password 32 0(0.00%) | 753 660 1025 710 | 0.11 0.00
GET /sign_up/email/confirm?confirmation_token= 32 0(0.00%) | 337 256 823 290 | 0.11 0.00
GET /sign_up/enter_email 32 0(0.00%) | 177 121 695 140 | 0.11 0.00
POST /sign_up/enter_email 32 0(0.00%) | 506 396 925 460 | 0.11 0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Aggregated 1319 39(2.96%) | 374 112 1886 180 | 4.40 0.13
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|------------------------------------------------------------|---------|------|------|------|------|------|------|------|------|------|------|------|
GET / 140 150 160 170 400 420 430 460 830 830 830 357
POST / 720 760 840 940 1100 1300 1500 1600 1900 1900 1900 323
GET /account 170 180 190 210 290 420 570 580 790 790 790 223
POST /login/two_factor/sms 340 340 350 360 380 600 620 620 620 620 620 37
GET /logout 130 140 140 150 160 220 450 460 780 780 780 187
GET /phone_setup 160 170 180 180 190 430 450 450 450 450 450 32
POST /phone_setup 490 500 530 550 770 830 1100 1100 1100 1100 1100 32
POST /sign_up/create_password 710 730 740 750 960 1000 1000 1000 1000 1000 1000 32
GET /sign_up/email/confirm?confirmation_token= 290 300 310 330 540 580 820 820 820 820 820 32
GET /sign_up/enter_email 140 160 160 170 180 390 700 700 700 700 700 32
POST /sign_up/enter_email 460 480 500 520 700 890 930 930 930 930 930 32
--------|------------------------------------------------------------|---------|------|------|------|------|------|------|------|------|------|------|------|
None Aggregated 180 430 660 690 760 940 1100 1300 1700 1900 1900 1319
Error report
# occurrences Error
--------------------------------------------------------------------------------------------------------------------------------------------
36 GET /account: CatchResponseError('You wanted /account, but got https://idp.pt.identitysandbox.gov/ for a url',)
1 POST /login/two_factor/sms: CatchResponseError('You wanted /account, but got https://idp.pt.identitysandbox.gov/ for a url',)
2 POST /: CatchResponseError('You wanted /login/two_factor/sms, but got https://idp.pt.identitysandbox.gov/account for a url',)
--------------------------------------------------------------------------------------------------------------------------------------------
Steps To Reproduce
- Copy over the images needed for IAL2 flows
- Run the prod_simulator flow
id-locust prod_simulator pt -u 10 -t 5m
Expected Behavior
No errors aside from those seen at high load.
Actual Behavior
Errors during low load sanity check runs.
Suggested Actions
Debug above errors.
Acceptance Criteria
prod_simulator flow runs without failures at normal load levels.
jgrevich commented