Can't Get Socket due to Out of Memory Error (Adafruit Feather ESP32-S2)
Closed this issue · 19 comments
Adafruit CircuitPython 9.0.0 on 2024-03-19; Adafruit Feather ESP32S2 with ESP32S2
Board ID:adafruit_feather_esp32s2
I have the Adafruit ESP32-S2 feather version with integrated BME280
9.x library bundle from adafruit-circuitpython-bundle-9.x-mpy-20240307
Updated the bootloader to 0.18.1 and circuit python to 9.0 before beginning with the porting project. I backed up my old files before flashing the bootloader. The older version that was working was running
Adafruit CircuitPython 8.0.0-beta.1-29-gfed884738 on 2022-10-11; Adafruit Feather ESP32S2 with ESP32S2
Board ID:adafruit_feather_esp32s2
I have a project on an Adafruit Feather ESP32-S2 that used to work in 8.0 beta (last time I used it). It doesn't even make it to the 2nd API request let alone the 6th. Not sure if this is Connection Manager related, build size related, etc.. but have to start somewhere and since Connection Manager is throwing the error suppose it's as good as anywhere to start.
I've uploaded the full (currently non-working) script here: Multiplexed 7-Segment Social Media Tracker
This is as far as it gets (and its got a much longer way to go before the script completes)
Connecting to WiFi...
✅ Wifi!
| Attempting to GET YouTube JSON...
| ✅ YouTube JSON!
| | Matching Results: 1
| | Request Kind: youtube#channel
| | Channel ID: UCHpvNfMNs7qdsUjOad2_FWg
| | Videos: 273
| | Views: 7183199
| | Subscribers: 11800
| | Response Kind: youtube#channelListResponse
Attempting to GET TWITTER Stats!
===============================
Traceback (most recent call last):
File "code.py", line 224, in <module>
File "adafruit_requests.py", line 597, in get
File "adafruit_requests.py", line 535, in request
File "adafruit_connection_manager.py", line 264, in get_socket
MemoryError:
I've attempted to slim down the script, not use CIRCUITPY_WIFI_SSID (so not to enable web workflow), glittered manual GC collects, etc.. and it just fails on the 2nd request attempt. This might be an S2 specific issue as I'm sure it would work on my S3 but I haven't tested it because the controller is embedded inside an enclosure and not easily modified or replaced. All the details for the build are included in my repo including a video of how it looks like when it was working.
I intend on troubleshooting this more with memfree etc... just want a second set of eyes in case I'm missing something. I have other microcontrollers with more memory I can throw at the problem but that's not the point. Point is this used to work with 8.0 and now it doesn't on 9.0
@DJDevon3 is it possible to test on s S3, just to see if it works on different hardware?
Yeah it just means I have to take it down off my wall and open it up. I wanted to take better pictures of the insides for a learn guide anyway.
Adafruit CircuitPython 9.0.0 on 2024-03-19; Adafruit Feather ESP32S3 4MB Flash 2MB PSRAM with ESP32S3
Board ID:adafruit_feather_esp32s3_4mbflash_2mbpsramSame thing. It gets to about line 350 then fails on the 4th request with gaierror -2.
File "code.py", line 358, in <module>
File "adafruit_requests.py", line 597, in get
File "adafruit_requests.py", line 535, in request
File "adafruit_connection_manager.py", line 236, in get_socket
gaierror: (-2, 'Name or service not known')It's always failing on a requests.get mastodon_response = requests.get(url=MAST_SOURCE) On the S2 is was the 2nd API get, on the S3 it's the 4th API get. No memory error this time. The error line numbers from adafruit_requests and Connection Manager that crashes the script are the same.
If you make that request via the REPL, it works? Since the args are in your env, can you also share the full URL?
I was so tired. Got 5 hours of sleep and yep nice one. This is literally the url.
https://None/api/v1/accounts/109289491812946909/statuses?limit=1
Nothing to see here move along. 🤦
I think checking to ensure the get url is at least not none and throw an error could help.
Mastodon has just changed their API. The PR I submitted 2 weeks ago, was reviewed, and approved... no longer works. That's an API issue and has nothing to do with this issue.
Yes the Adafruit ESP32-S3 Feather fully completes all requests. It has the same 4MB Flash 2MB PSRAM as the Adafruit ESP32-S2. I'll have to go back and take a look at the S2 again now that Twitter (last year) and Mastodon (this week) deprecated some methods and endpoints.
Updated the code in my repo for 9.0 works on S3. Have yet to go back and test the S2.
Loaded the same code as the S3 which worked onto the S2 and the S2 is failing at the same place it did prior. The line number is different because I put a lot of work into getting it working on the S3 but the actual place in the script it was failing is the same place.
Traceback (most recent call last):
File "code.py", line 205, in <module>
File "adafruit_requests.py", line 597, in get
File "adafruit_requests.py", line 535, in request
File "adafruit_connection_manager.py", line 264, in get_socket
MemoryError: mastodon_response = requests.get(url=MAST_SOURCE, headers=MASTODON_HEADER)Mastodon is only the 2nd API and is failing at the requests.get. I double checked the url and it should be working. It's the same script that worked on the S3. It has to be some kind of memory issue because it's not even getting to the 2nd GET request.... unless the S2 only has 1 socket?
I verified the get url is correct and when copy/pasted into a browser does take me a json page. Tried deleting everything after that point to just finish the script and it does finish. Slowly started adding back in 1 API and out of memory again. This might be a memory issue and not connection manager or socket related. It works on the S3 not on the S2.
After more investigation of slowly adding in parts of API's this definitely seems more memory related than socket related. If I reduce the overall memory footprint I can get through all of them. It's the size of the script and additional variables taking up RAM. Might need to reference and shuffle this one to the core devs to look into the build size and stuff.
Glad you were able to track it down. You may be able to correct it by doing some gc.collect() calls. Might be getting too much fragmentation...
My guess is that ConnectionManager is not the culprit here, because it's pretty much just a code refactoring. To test that hypothesis, I suggest testing with an earlier version of adafruit_requests that does not include ConnectionManager. https://github.com/adafruit/Adafruit_CircuitPython_Requests/releases/tag/2.0.5 is the last release before the addition of ConnectionManager.
Are you using .py or .mpy versions of the libraries?
.mpy, unless there is an issue that needs obvious attention the preference is always mpy. I added gc back into the new version of the script after it started crashing yes there's sprinkled everywhere again, no difference.
I also updated the bootloader to 0.18.2 would it matter if I install an older version of circuit python before that bootloader existed to test with earlier versions as suggested?
I figured this might end up being a core issue but connection manager was a good place to start since that's what is throwing the immediate error. Should I raise a new issue with the core and reference this?
Bootloader makes no difference. Try with the earlier version of adafruit_requests as I mentioned. Don't move CircuitPython to an earlier version yet; just use an earlier version of adafruit_requests. That will tell you if there's some issue with ConnectionManager. Assuming that fails similarly (might not be exactly the same due to different RAM use), then we can transfer this issue to core.
I still have my 8.0 beta device backup before updating the bootloader and flashing to 9.0. I've learned the hard way in 7.x/8.x to always make a backup before upgrading. Have plenty of old versions of all my projects to fall back on. Will try that tonight.
Won't allow the 8.0 beta adafruit_requests.mpy to work in 9.0. Probably requires a 9.0 compatible mpy file since I have 9.0 installed and using other 9.0 libraries.
File "code.py", line 14, in <module>
ValueError: incompatible .mpy file
Yes the last time I used this project was 8.0 beta. Even if I had a more recent 8.x mpy it would still likely be incompatible with 9.x
You can use the .py version of the library from the GitHub repo for the older release.
Added gc.memfree and traceback module. I have no idea where the MemoryError is coming from. I cannot catch that exception.
Not worried about including the url for call or username as it's all publicly accessible no token required. The header for Mastodon doesn't matter I've tried removing it. The culprit is stemming from the requests.get/post triggering something it doesn't like.
code.py output:
Prior to Imports: 2052064
After Imports: 2025488
Memory Used: 26576
Prior to Script Configurations: 2025328
After Script Config: 2021088
Memory Used: 4240
Prior to WiFi (in while true): 2021088
Connecting to WiFi...
✅ Wifi!
| Shuffling data to 7-Segment Displays
| This could take a minute...
After Wifi: 2021088
Memory Used: 0
Prior to YouTube (in while true): 2021088
After Youtube: 2016480
Memory Used: 4608
Prior to Mastodon (in while true): 2016480
Mastodon Source: {'Content-Type': 'application/json'} https://hackaday.social/api/v1/accounts/lookup?acct=TreasureDev@hackaday.social
# setup an general traceback handler around entire Mastodon API call and if it errors then skip the section.
Traceback (most recent call last):
File "code.py", line 237, in <module>
File "adafruit_requests.py", line 597, in get
File "adafruit_requests.py", line 535, in request
File "adafruit_connection_manager.py", line 264, in get_socket
MemoryError:
# Fails on requests.get for Mastodon then proceeds to Twitch
Prior to Twitch (in while true): 2016192
Just prior to Twitch requests.post: 2016048
Memory Used: 144
Traceback (most recent call last):
File "code.py", line 287, in <module>
File "adafruit_requests.py", line 601, in post
File "adafruit_requests.py", line 535, in request
File "adafruit_connection_manager.py", line 264, in get_socket
MemoryError:Fails on requests.post for Twitch in the same way. I have 2.1MB of memfree then a MemoryError on requests.get/post. It almost feels like a hard fault except it's not saying it's a safe mode hard fault.
The only thing odd that jumps out is Wifi connection taking 0 bytes.
print("\nConnecting to WiFi...")
while not wifi.radio.ipv4_address:
try:
wifi.radio.connect(ssid, password)
except ConnectionError as e:
print("❌ Connection Error:", e)
print("Retrying in 10 seconds")
gc.collect()
print("✅ Wifi!")It is reporting Memory Used: 0 to execute this portion of the script.
You can use the
.pyversion of the library from the GitHub repo for the older release.
🤦 oh right. that might take a while to track down, will give it a shot.
Snagged a version of adafruit_requests from Oct 2022 which is the timestamp on some of the files in my 8.0 beta backup. Same errors, same places. I have 2.1MB free then a MemoryError.
I have multiple Adafruit Feather ESP32-S2's but aren't soldered up. Will solder another up. Maybe it's the board?
soldered up a 2nd Adafruit Feather ESP32-S2 and it already had 8.2.5 and some files on it including requests (most of my boards have it).
Adafruit CircuitPython 8.2.5 on 2023-09-07; Adafruit Feather ESP32S2 with ESP32S2
Board ID:adafruit_feather_esp32s2I imagine it's using 8.2.5 library bundle or close thereabout.
All I did was fill in settings.toml, paste my script that worked on the S3, changed socketpool and wifi back to 8.2.x (removed connection manager), and it just worked.
Then I hard reset the board and it gave the same error but in an 8.2.5 kind of way.
Just prior to Twitch requests.post: 1994256
Memory Used: 192
Traceback (most recent call last):
File "adafruit_requests.py", line 515, in _get_socket
espidf.MemoryError: So it would seem this memory error is esp-idf related. Odd that 8.2.5 provides more of a clue than all the traceback and gc.memfree stuff I was throwing at it in 9.0.
I did a soft-reload and it still failed. Did a hard reset again and it completed. This is all quite odd.
@DJDevon3 The link I gave above, https://github.com/adafruit/Adafruit_CircuitPython_Requests/releases/tag/2.0.5, has .zip files of both 8.x and 9.x versions of adafruit_requests. This is the last version before ConnectionManager was added. Test the 9.x .mpy with CircuitPython 9.0.1.
Think I've proven connection manager has nothing to do with it. Connection manager didn't exist in 8.2.5 and I was still getting MemoryError on a completely different ESP32-S2 running 8.2.5 with 8.2.5 libraries.
Alright, so would you like to open a new issue on CircuitPython itself, showing the memory issue with a simple test program? We can close this.