fcsonline/drill

Panic when some requests time out

daoleno opened this issue · 6 comments

If a timeout occurs on some requests, the statistics will not be printed out. The program will then panic.

TimeOut Log

Error connecting 'xxxxx': reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("xxxx")), port: None, path: "xxxx", query: Some("xxxx"), fragment: None }, source: TimedOut }

Panic Log

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ValueOutOfRangeResizeDisabled', /Users/daoleno/.cargo/registry/src/github.com-1ecc6299db9ec823/hdrhistogram-7.5.1/src/lib.rs:1855:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

just created #174 to hopefully solve this. if you (or anyone facing this issue) want to clone, build and test my fork to see if this solves your issue, that would be great to confirm that my PR does indeed solve your timeout case!!

based on my findings while working on #174 it doesn't actually appear to be exclusive to timeouts. actually, any request that's >3.6s in duration causes this panic

I got a similar issue. Tried the branch in #174
Was still able to trigger traceback now and then.

git clone https://github.com/rayzr522/drill.git
Cloning into 'drill'...
..loyment/benchmark-drill [main|…1] ❯❯❯ mv drill drill-pr-202303
..loyment/benchmark-drill [main|…1] ❯❯❯ cd drill-pr-202303 
..-drill/drill-pr-202303 [master|✔] ❯❯❯ cargo run --benchmark ../benchmark.yml --stats
...
...
...
Fetch index.html          https://dev.foo.com/nz/index.html 200 OK 1184ms
Error connecting 'https://dev.foo.com/nz/index.html': reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("dev.foo.com")), port: None, path: "/nz/index.html", query: None, fragment: None }, source: TimedOut }
Fetch index.html          https://dev.foo.com/nz/index.html 200 OK 1169ms
Fetch index.html          https://dev.foo.com/nz/index.html 200 OK 1179ms
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ValueOutOfRangeResizeDisabled', /Users/pez/.cargo/registry/src/github.com-1ecc6299db9ec823/hdrhistogram-7.5.2/src/lib.rs:1864:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I was able to work around the panic by adding a timeout of 3 sec, with "--timeout 3"

I got a similar issue. Tried the branch in #174
Was still able to trigger traceback now and then.

git clone https://github.com/rayzr522/drill.git
Cloning into 'drill'...
..loyment/benchmark-drill [main|…1] ❯❯❯ mv drill drill-pr-202303
..loyment/benchmark-drill [main|…1] ❯❯❯ cd drill-pr-202303 
..-drill/drill-pr-202303 [master|✔] ❯❯❯ cargo run --benchmark ../benchmark.yml --stats
...
...
...
Fetch index.html          https://dev.foo.com/nz/index.html 200 OK 1184ms
Error connecting 'https://dev.foo.com/nz/index.html': reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("dev.foo.com")), port: None, path: "/nz/index.html", query: None, fragment: None }, source: TimedOut }
Fetch index.html          https://dev.foo.com/nz/index.html 200 OK 1169ms
Fetch index.html          https://dev.foo.com/nz/index.html 200 OK 1179ms
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ValueOutOfRangeResizeDisabled', /Users/pez/.cargo/registry/src/github.com-1ecc6299db9ec823/hdrhistogram-7.5.2/src/lib.rs:1864:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

you didn't check out the branch that has the fix. the branch with the fix is not the main branch :)

make sure to git checkout fix-histogram-upper-bound! or you can install via cargo directly w cargo install --git https://github.com/rayzr522/drill --branch fix-histogram-upper-bound

Looks like it fixed the issue for me. Sadly the errors are not 100% reproducible, as the timeouts seem to happen unpredictably in my tests, but so far your branch was working fine.

https://github.com/rayzr522/drill --branch fix-histogram-upper-bound seems to be working for me as well.