RESTler ran forever: "Exhausted collection..." loop prevented RESTler from honoring --time_budget
wilbaker opened this issue · 0 comments
Description
In a recent run of RESTler, the remaining_time_budget
check started getting skipped because ExhaustSeqCollectionException
kept getting thrown in a loop.
Because RESTler never made it far enough to check remaining_time_budget
here:
timeout_reached
was never being set to true in driver.py here:
restler-fuzzer/restler/engine/core/driver.py
Line 840 in 7503a34
And RESTler never set should_stop
to true here:
restler-fuzzer/restler/engine/core/driver.py
Line 878 in 7503a34
The following was logged over and over until RESTler was killed:
Testing completed -- below are the final stats:
Setting fuzzing schemes: random-walk
2024-04-09 13:22:24.000: Going to fuzz a set with 47 requests
2024-04-09 13:22:24.000: Request-0: Value Combinations: 4
2024-04-09 13:22:24.000: Request-1: Value Combinations: 4
2024-04-09 13:22:24.000: Request-2: Value Combinations: 4
2024-04-09 13:22:24.000: Request-3: Value Combinations: 4
2024-04-09 13:22:24.000: Request-4: Value Combinations: 4
2024-04-09 13:22:24.000: Request-5: Value Combinations: 4
2024-04-09 13:22:24.000: Request-6: Value Combinations: 8
2024-04-09 13:22:24.000: Request-7: Value Combinations: 4
2024-04-09 13:22:24.000: Request-8: Value Combinations: 4
2024-04-09 13:22:24.000: Request-9: Value Combinations: 4
2024-04-09 13:22:24.000: Request-10: Value Combinations: 4
2024-04-09 13:22:24.000: Request-11: Value Combinations: 4
2024-04-09 13:22:24.000: Request-12: Value Combinations: 4
2024-04-09 13:22:24.000: Request-13: Value Combinations: 4
2024-04-09 13:22:24.000: Request-14: Value Combinations: 2
2024-04-09 13:22:24.000: Request-15: Value Combinations: 4
2024-04-09 13:22:24.000: Request-16: Value Combinations: 4
2024-04-09 13:22:24.000: Request-17: Value Combinations: 4
2024-04-09 13:22:24.000: Request-18: Value Combinations: 4
2024-04-09 13:22:24.000: Request-19: Value Combinations: 4
2024-04-09 13:22:24.000: Request-20: Value Combinations: 4
2024-04-09 13:22:24.000: Request-21: Value Combinations: 4
2024-04-09 13:22:24.000: Request-22: Value Combinations: 4
2024-04-09 13:22:24.000: Request-23: Value Combinations: 4
2024-04-09 13:22:24.000: Request-24: Value Combinations: 4
2024-04-09 13:22:24.000: Request-25: Value Combinations: 4
2024-04-09 13:22:24.000: Request-26: Value Combinations: 4
2024-04-09 13:22:24.000: Request-27: Value Combinations: 4
2024-04-09 13:22:24.000: Request-28: Value Combinations: 4
2024-04-09 13:22:24.000: Request-29: Value Combinations: 4
2024-04-09 13:22:24.000: Request-30: Value Combinations: 4
2024-04-09 13:22:24.000: Request-31: Value Combinations: 4
2024-04-09 13:22:24.000: Request-32: Value Combinations: 4
2024-04-09 13:22:24.000: Request-33: Value Combinations: 4
2024-04-09 13:22:24.000: Request-34: Value Combinations: 4
2024-04-09 13:22:24.000: Request-35: Value Combinations: 4
2024-04-09 13:22:24.000: Request-36: Value Combinations: 4
2024-04-09 13:22:24.000: Request-37: Value Combinations: 4
2024-04-09 13:22:24.000: Request-38: Value Combinations: 4
2024-04-09 13:22:24.000: Request-39: Value Combinations: 4
2024-04-09 13:22:24.000: Request-40: Value Combinations: 10
2024-04-09 13:22:24.000: Request-41: Value Combinations: 4
2024-04-09 13:22:24.000: Request-42: Value Combinations: 4
2024-04-09 13:22:24.000: Request-43: Value Combinations: 4
2024-04-09 13:22:24.000: Request-44: Value Combinations: 4
2024-04-09 13:22:24.000: Request-45: Value Combinations: 4
2024-04-09 13:22:24.000: Request-46: Value Combinations: 4
2024-04-09 13:22:24.000: Avg. Value Combinations per Request: 4.170212765957447
2024-04-09 13:22:24.000: Median Value Combinations per Request: 4
2024-04-09 13:22:24.000: Min Value Combinations per Request: 2
2024-04-09 13:22:24.000: Max Value Combinations per Request: 10
2024-04-09 13:22:24.000: Total dependencies: 68
2024-04-09 13:22:24.016: Generation: 1 / Sequences Collection Size: 1
(After random-walk Extend)
Exhausted collection...
2024-04-09 13:26:36.350: Generation: 1 / Sequences Collection Size: 0
(After random-walk Render)
2024-04-09 13:26:36.350: Final Swagger spec coverage: 17 / 47
2024-04-09 13:26:36.350: Rendered requests: 17 / 47
2024-04-09 13:26:36.350: Rendered requests with "valid" status codes: 17 / 17
2024-04-09 13:26:36.350: Num fully valid requests (no resource creation failures): 17
2024-04-09 13:26:36.350: Num requests not rendered due to invalid sequence re-renders: 0
2024-04-09 13:26:36.350: Num invalid requests caused by failed resource creations: 0
2024-04-09 13:26:36.350: Total Creations of Dyn Objects: 56
2024-04-09 13:26:36.350: Total Requests Sent: {'gc': 113, 'main_driver': 431}
2024-04-09 13:26:36.350: Bug Buckets: {'main_driver_501': 1}
Testing completed -- below are the final stats:
Setting fuzzing schemes: random-walk
Well after --time_budget
was exceeded.
Steps to reproduce
I've only seen this once so far, and I'm not familiar enough about the implementation details of sequence rendering to understand the preconditions that triggered this issue.
RESTler was using the random-walk
strategy in case that's relevant:
restler.exe fuzz --time_budget 3.8963747535833333 --search_strategy random-walk --disable_checkers useafterfree,invaliddynamicobject,leakagerule,resourcehierarchy,payloadbody,examples,invalidvalue --grammar_file "Compile\grammar.py" --dictionary_file "dict.json" --settings "engine_settings.json"
Expected results
I expected that Timed out...
would be logged after the time budget was exceeded.
Actual results
RESTler kept running forever and logging "Exhausted collection..."
Environment details
RESTler version 9.2.4