Error when making requests from a Micropython client to a Microdot server

[j4mie]

Hi, huge thanks for Microdot, it's brilliant!

I've been trying to track down an issue introduced in a recent version, but it's been difficult to reproduce as it only happens in very specific circumstances.

My setup is as follows: I have a Microdot server running on a Raspberry Pi 4, with a bunch of Raspberry Pi Picos around my house running Micropython. The Picos send sensor data back via HTTP POSTs to the server using the urequests library that is part of Micropython.

A few days ago, I upgraded the version of Microdot running on the server from 1.2.1 to 1.3.0. Immediately, the Picos stopped reporting data back to the server. I quickly rolled back to 1.2.1 and everything started working again.

I've now had a chance to dig in to the problem, but it's tricky: everything works fine when I make the request with curl on my laptop, but when I use urequests on the Pico, any type of request reliably fails with the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "urequests.py", line 180, in get
  File "urequests.py", line 130, in request
ValueError: HTTP error: BadStatusLine:
[]

This is coming from inside urequests here: https://github.com/micropython/micropython-lib/blob/master/python-ecosys/urequests/urequests.py#L130

I've done a bit of stepping through the urequests code, and it turns out that when making requests to later (broken) versions of Microdot, the l = s.readline() on line 125 of urequests ends up with l being equal to an empty bytestring (b'').

I've cloned Microdot locally and used git bisect to find the problematic commit, and it's this one: d0d358f

Sure enough, if I set microdot.Request.socket_read_timeout to None in my server code (restoring the previous behavior) then everything works fine. Any value bigger than 0.2 seems to work OK, but 0.1 reliably fails.

Note that it doesn't seem to help if I pass the timeout argument to the urequests request methods.

This is an issue and not a PR because I'm not sure what the right fix is here. Should we be increasing the default timeout in Microdot? Should we revert that commit entirely? Should I just add microdot.Request.socket_read_timeout = None to my code and forget about it? Is it a bug in urequests? Or something else?

To Reproduce
Steps to reproduce the behavior:

1. Install Micropython on a Raspberry Pi Pico W
2. Connect to a REPL (possibly using the Thonny IDE)
3. import urequests
4. response = urequests.get("http://some-server-running-microdot")
5. Note ValueError: HTTP error: BadStatusLine

Expected behavior

An HTTP request is completed successfully.

[miguelgrinberg]

I think the problem is that your clients are very slow, so the 0.1s timeout hits before they get to send a complete request.

Disabling the timeout altogether may work for this specific situation, but it has been causing other problems, so I'm not going to get rid of it. But yes, it seems my chosen 0.1s is a bit too quick, so I'll increase it.

[miguelgrinberg]

Changed to 1 second now.

[j4mie]

Thanks for the quick response! 🙂