ESP32 ublox module causes core panic
nseidle opened this issue · 2 comments
tldr; Reducing the I2C buffer length to 32 bytes seems to prevent core panic.
Bonus points to readers: Why does the ESP32 fail to return from a .requestFrom() if the I2C peripheral fails to respond in the middle of a transmission? This failure to return is what causes the core panic. But the ublox module is misbehaving and is the main problem.
I have an ESP32 controlling a ZED-F9P (firmware is v1.13 but I've experienced same behavior with v1.12) over I2C using this lib. After a certain amount of time, and under certain conditions the ESP32 will hard fault which looks like this:
The module is outputting 100bytes of PVT, then ESP32 borks. Looking at the I2C traffic the plot thickens:
Image above: this is the last packet to come across before ESP32 panics. The ESP32 is correctly reading 100 bytes (or at least attempting to). Note in the image above, starting around the mid point of the transmission the data goes to 0xFF (ublox stops transmitting data), and towards the end of the packet the ublox module fails to ack entirely.
Above is the last third of the packet. The ESP32 is still running the I2C clock but ublox module has stopped acking.
Above is the 1st 3rd of the packet. You can see, very oddly, that the 0x4A fails to have an ACK, then NACKs, then a few dozen 0xFF ACKs. The ublox module seems to have lost its mind...
The clock is 100kHz. This board has no physical pullups on the I2C lines. It's hard to say what software pullups are enabled (I'm not very deep into ESP32 land).
Above is the ublox module controlling SDA long before the ESP32 raises the clock so I don't think this is a race condition issue. I don't think this is a pull up issue.
So what's causing the ESP32 to lose its mind? The library is reading the 100 bytes available in checkUbloxI2C(). Does the I2C hardware get lost and block infinitely when the code says to do a requestFrom(0x42, 100) but the ublox module fails to ack the last 35 bytes? Once lost, the ESP32 WDT kicks in and resets the system?
In the above image I added the debug prints around the suspected read command.
Above, we see a good start/end request. Then, a start, then a panic. So, unless we're not printing complete buffers because of que'ing, then I'd say the ublox module is hanging the I2C bus causing heartburn for the ESP32.
The ESP32 I2C buffer length is 128. Does shortening the length to 32 help?
(head in hands) It's as solid as the day is long.
Grumble. Above is the ESP32 hitting the ublox is 32 byte chunks. It works great. ?!?
Good news is it's stable at 32 byte buffer. Bad news is I'm not sure why. I believe I can put the blame on the ublox. Yes, the ESP32 should recover better when a peripheral fails to respond, but the ublox is the peripheral that is going into the weeds. I believe the added delay between 32 byte reads is allowing the ublox to correctly que up the next I2C bytes avoiding a lockup.
Possibly related issues:
- Wemos D1 Mini is ESP8266. Not sure if it's related or not.
I suggest a platform override for the ESP32 to reduce length to 32 (from default of 128). This would cause other libraries to operate differently but there are very few libraries and I2C peripherals to my knowledge that require >32 byte blocks so it should be fairly transparent to end user.
The real question is what is going on with the ublox module and why is the I2C interface folding over?
Possibly related?
#121 (comment)
Transaction size now defined as a global in this commit. Use setI2CTransactionSize and getI2CTransactionSize to increase over default of 32 bytes as needed.