edicl/drakma

unidentified bug causing short read of chunked data

leafbebop opened this issue · 10 comments

leafbebop commented

The problem: (drakma:http-request "http://digidb.io/digimon-list/") got an input-chunking-unexpected-end-of-file error.

Details:
After I poking around the code, I find that it is indeed a short read of (read-sequence input-buffer inner-stream :start 0 :end chunk-size) in chunga::fill-buffer. (https://github.com/edicl/chunga/blob/cb333cdba178e99b03fa60e2caa8c5d3654201d8/input.lisp#L145)

The number of bytes read is consistent most of time, but sometimes it might vary.

The bytes read is valid, and another (read-sequence big-buffer chunga::inner-stream) in the debugger can successfully read the following content left in the stream.

How to reproduce:
This error is only reproducible on Android, where only ecl is available. Can both be reproduced in cl-repl app (which is available in Play Store), and ecl compiled in Termux (a terminal for android). This error is not reproducible by curl or other tools on the same device.

This error is only observed with remote server - but I only tested with "http://digidb.io/digimon-list/" so I am uncertain about other servers. On the other hand, replaying the response on localhost does not reproduce the error.

This error is only reproducible when using drakma:http-request without :want-stream t. Even if use want-stream,

  1. use read-sequence directly with the returned stream
  2. use drakma:read-body on returned http-stream, or
  3. use drakma::%read-body on the returned stream
    does not reproduce the error (and the request is successful).
stassats commented

The android requirement makes this harder to diagnose (don't have an android device).

leafbebop commented

Even though I have one, the diagnose process so far is really painful.

I am currently run out of idea. I cannot think a single reason why drakma:read-body with :want-stream work, but directly use of drakma:http-request don't.

Do you have any suggestion how can I further investigate this problem?

Thanks

edit: wording

stassats commented

Why does it do only one read-sequence and then give up?

leafbebop commented
stassats commented

read() is not guaranteed to read everything up to EOF. I'll make change to chunga for you to try.

stassats commented

Although I'm not sure how READ-SEQUENCE is implemented on ECL.

stassats commented

Can you try edicl/chunga@e83c804

And I hope I got the logic right, as I don't have anything to test it on.

leafbebop commented
stassats commented

I'm not sure it's the right fix, maybe ECL implements read-sequence incorrectly. CLHS says that read-sequence returns fewer elements only upon reaching EOF.

leafbebop commented