Program to Big.... Using BLE libraries!
pbecchi opened this issue ยท 55 comments
Hardware:
Board: ESP32 Dev Module
Core Installation/update date: 31/dec/2017?
IDE name: Arduino IDE 1.8.4
Flash Frequency: ?40Mhz?
Upload Speed: ?115200?
Description:
Using BLE library to add BLE uart connection capability to an existing code, has increased the size of the .bin of more than 630kb and build failed:
Debug Messages:
`Program too big. The size is 1349762 bytes (of a 1310720 byte maximum). Ensure debugging is OFF and/or see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing it.
Build failed for project 'ESP32_OpenSprinkler'
I Never thought I could have hit that limit!!
Any way to decrease size of those libraries??
The BLE library is HUGE. And it is not possible to reduce the size.
I struggled with the same problem and the only solution was to change the partition size to add more available program space by reducing (or eliminating) the SPIFFS and EEPROM partitions.
There is a small tutorial how to Change partition size
Thanks , I know how to change partition size, but this is not the right solution!
The BLE library is huge, there must be a way to reduce it to a reasonable size may be tailoring the services to the application need! Otherwise we keep increasing code size and decreasing flash available for Spiffs and other use!
From what I know that's complete up to Espressif inside the ESP-IDF. I guess you would need to post an issue in the ESP-IDF repo
A posted an issue to nkolban site hoping to get some guidance on how to tailor configuration files.
I think that I don't need a number of Ble services and should be possible , I hope , to compile a reduced size library!
Hi all, I faced the same issue and did not found a easy solution, so I finally forked the library and create a "light version" of it.
Only the BLE server will work. This is a fork of the official esp32 ble library and is stripped to decrease the size. It is only possible to use the ble server features.
I saved about ~30KB program size with this version.
@jjoe64 I am trying your library, but getting this error...
error: 'class BLECharacteristic' has no member named 'notify'
pTxCharacteristic->notify();
Hi all, I faced the same issue and did not found a easy solution, so I finally forked the library and create a "light version" of it.
Only the BLE server will work. This is a fork of the official esp32 ble library and is stripped to decrease the size. It is only possible to use the ble server features.I saved about ~30KB program size with this version.
Just curious - will the BLE_scan example work with this light(er) version of the library? Thanks
@m-menon most of flash is taken by esp-idf bt stack when you init bluetooth. BT stack is very heavy. Try to test it with btStart() if i remember. When i did test few weeks ago it was about about 800kB bin size with just one bt init command.
@chegewara thanks. Sure, I will test it out. The BT stuff is heavy and there is no easy way to shrinking it. I can work around with the partition scheme for now and get things to work. The library itself is excellent and thanks for that.
I faced the same problem SO I fixed mine just to change some settings in Arduino IDE.
Tools -> Partition Scheme:
change "Default" to "Huge APP(3MB No OTA)"
So I selected this Huge App Partition and my huge code got accepted with no freaking errors for my ESP32 Board.
Same here. Thanks for finding this.
Using the device : WeMosยฎ ESP32 WiFi + Bluetooth Ultra Low Power Consumption Dual Core ESP-32
Changed Arduino setting according namanbhardwaj42's fix.
Working great now.
Using Arduino IDE 1.8.8
with the following includes.
BLE, WIFI and MQTT on a WeMosยฎ ESP32 WiFi + Bluetooth-ontwikkelingsraad Ultra Low Power Consumption Dual Core ESP-32 ESP-32S Vergelijkbaar ESP8266
#include "BLEDevice.h"
#include <WiFi.h>
#include <PubSubClient.h>
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
I've used BLE on several other platforms and none of those had libraries that were anywhere near this big. I made a "Simple" app for the ESP32 with a Web Interface to change data and a BLE Interface to read it, and the size is almost 1.5 MB!
The library sizes really limit the applications that the ESP32 can be used for - certainly no real commercial applications. It's a shame as I had great hopes for the ESP32.
@APBeck3rd good for you. For example i am using esp32 with BLE in few projects and none is even close to 1.5MB, it around 1.3MB (my AWSFreeRTOS app, with full BLE stack, mqtt and TFT working on m5stack is 1.17MB, only issue is heap shortage).
But i understand what you are saying and the problem is not library, the problem is that esp-idf bluetooth stack library is for BLE and classic bluetooh and even if you are using only BLE whole library is linked. Maybe esp-idf v4.x will have NimBLE implementer, which is already merged to master branch. NimBLE library is light weight because its only BLE driver. Lets hope someone will add NimBLE support for arduino too (maybe even this year).
Hi @chegewara , it is great knowing esp-idf has a lighter version of BLE library. I am running into the same issue trying to make a application which has the BLUFI and some other sub-tasks for LED driving, GPIOs manipulation etc... It was fine before I added the BLUFI example to my application, but now I am having the same issue that everyone else in this thread have faced. Now thanks to you I have just known about NimBLE, is there any available library that has light footprint like NimBLE but supporting BLUFI.
If not please suggest anything I could do to serve my purpose.
PS: I am developing my application in FreeRTOS not Arduino
Hi @QuangNguyen1412
recently NimBLE has been merged to AWSFreeRTOS, just i dont know if its fully functional already (just didnt test it yet). I think you can ask in AWSFreeRTOS repository about that.
I was thinking the same way, thanks for the info @chegewara
[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
Not stale
[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.
When will NimBLE support be added to Arduino? I noticed the branch, IDF release/v3.3, in this project but it does not include the libnimble.a library that is in IDF v3.3.
@yuetloo it won't be available until it is in a stable release of esp-idf and that release has been integrated to this repo. The only version of esp-idf that has NimBLE today is the master and v4 if esp-idf and neither have a stable release yet, only an early beta/alpha for v4 (which may be from before NimBLE integration).
I'm really struggling with the BLE library size as I need OTA and BLE and using BLE make my sketch about 47% of 3M e.g. no space for 2 X OTA and Factory ...
also I definitely need both as I need to do a OTA over a BLE-
e.g. any help with significantly reducing the BLE lib or how to do a OTA with having only two partitions will be of GREAT HELP
You can use BLE and OTA if you change your partition scheme when uploading, you can use minimal SPIFFS for example
I do not need SPIFFS - and yes I have already changed my partition scheme , but still I have only 3M and my sketch takes 47% of that e.g. no space for standard ะะขะ with 2 x OTA and one Factory - if you point me to a example that use only 2 x ะะขะ without the Factory - then it will be fine for me...
ref : WiFi - my sketch DO USE WIFI - but is also need BLE - e.g. no mater do I use WiFi for OTA ot BLE to transport teh binary is not important the important fact is that I need teh BLE lib and it is so huge that after including it - the standard OTA lib with 2 x OTA and 1 x Factory is not possible to be used.
@vtomanov I dont know what app you are trying to build and why you need factory partition, but usually when you build OTA capable app then factory partition is not required (usually, not always).
This is from one of my apps, where i am using WIFI + BLE and few more libraries, ssd1306, e-paper ...
Here is size of my app and i feel i still have plenty free space:
Sketch uses 1692502 bytes (86%) of program storage space. Maximum is 1966080 bytes.
@vtomanov I dont know what app you are trying to build and why you need factory partition, but usually when you build OTA capable app then factory partition is not required (usually, not always).
This is from one of my apps, where i am using WIFI + BLE and few more libraries, ssd1306, e-paper ...
Here is size of my app and i feel i still have plenty free space:Sketch uses 1692502 bytes (86%) of program storage space. Maximum is 1966080 bytes.
hmm then may be I'm wrong - reading the documentation about the OTA the two default partitioning options are :
โSingle factory app, no OTAโ
โFactory app, two OTA definitionsโ
e.g. the second one is :
Espressif ESP32 Partition Table
Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x4000,
otadata, data, ota, 0xd000, 0x2000,
phy_init, data, phy, 0xf000, 0x1000,
factory, 0, 0, 0x10000, 1M,
ota_0, 0, ota_0, 0x110000, 1M,
ota_1, 0, ota_1, 0x210000, 1M,
Can you suggest custom portioning with only two OTA ?
and also how I can force the loader to load my sketch originally in OTA1 e.g. from example directly Arduino UI to upload to ota_0
Sorry if I'm asking stupid questions... just trying to understand how the things with big projects and OTA can be done
btw: my app size :
Sketch uses 1638378 bytes (48%) of program storage space. Maximum is 3407872 bytes.
Like suggested here you can use Minimal SPIFFS schema. Its 2 OTA partitions 1.9MB each.
You can use BLE and OTA if you change your partition scheme when uploading, you can use minimal SPIFFS for example
[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
Not stale
[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.
Not stale
Why?
I still want to know the answer, I think the issue is still valid. I hope someone eventually updates this thread with a solution that might become available and we can be notified.
My solution was to use custom OTA table with only two OTA partitions without Factory partition ๐
Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x4000,
phy_init, data, phy, 0xD000, 0x1000,
otadata, data, ota, 0xE000, 0x2000,
ota_0, app, ota_0, 0x10000, 1984K,
ota_1, app, ota_1, 0x200000, 1984K,
eeprom, data, 0x99, 0x3F0000, 0x1000,
nvs_key, data, nvs_keys,0x3F1000, 0x4000
spiffs, data, spiffs, 0x3F5000, 32K,
#12K not used
there is also a issue with OTA update of big partitions - especially when BLE is use ( as it consumes CPU) if you try standard OTA update the watchdog restarts the app while partition is erased - the work around is to have a custom partition cleaner on chunks of 512K and next tell teh OTA that partition size is 1024
custom eraser ๐
inline esp_err_t OTAEraseWTD()
{
const esp_partition_t * erase_partition = NULL;
esp_err_t err;
LOG64_SET(F("OTA: Erase WTD..."));
LOG64_NEW_LINE;
erase_partition = esp_ota_get_next_update_partition(NULL);
if (erase_partition == NULL)
{
LOG64_SET(F("OTA: Erase WTD address NULL"));
LOG64_NEW_LINE;
return ESP_ERR_OTA_BASE;
}
else
{
LOG64_SET(F("OTA: Erase WTD ["));
LOG64_SET("0x" + String(erase_partition->address, HEX));
LOG64_SET("Size : " + String(erase_partition->size));
LOG64_SET(F("]"));
LOG64_NEW_LINE;
}
uint32_t blockBegin = 0;
uint32_t blockSize = erase_partition->size;
for (;;)
{
uint32_t eraseBlockSize = OTA_ERASE_SIZE;
if (OTA_ERASE_SIZE > blockSize)
{
//last block to erase
if (blockSize != 0)
{
err = esp_partition_erase_range(erase_partition, blockBegin, blockSize);
if (err != ESP_OK)
{
return err;
}
}
break;
}
err = esp_partition_erase_range(erase_partition, blockBegin, OTA_ERASE_SIZE);
if (err != ESP_OK)
{
return err;
}
blockSize -= OTA_ERASE_SIZE;
blockBegin += OTA_ERASE_SIZE;
delay(64);
}
return ESP_OK;
}
and in OTABegin :
.....
LOG64_SET(F("OTA: Erase WTD Start"));
LOG64_NEW_LINE;
err = OTAEraseWTD();
if (err != ESP_OK)
{
LOG64_SET(F("OTA: Erase WTD ERR ["));
LOG64_SET((uint32_t)err);
LOG64_SET(F("]"));
LOG64_NEW_LINE;
OTAInit();
return;
}
LOG64_SET(F("OTA: Erase WTD Finish : "));
LOG64_SET((uint32_t)(millis() - procedureStart));
LOG64_NEW_LINE;
err = esp_ota_begin(update_partition, 2048, &update_handle);
.....
I still want to know the answer, I think the issue is still valid. I hope someone eventually updates this thread with a solution that might become available and we can be notified.
This issue is valid and will be valid as long as esp32 will be programmed with arduino and has nothing to do with BLE.
Your app is too big to flash with partition table scheme you are using. Change partition table scheme or create partitions.csv file with partitions that will fit and put into project folder.
This question has been answered here few times already and in few more similar issue topics.
Im not trying to be mean, just like i said, its been already answered in that over 2 years period.
I actually agree that the issue is still there - may be not connected with arduino GUI- but still the BLE library is wrongly huge and need to be fixed
Thank you for sharing. So far I have been using minimal spiffs for now, but as I add some more functionality to my project I expect I will come up against the size limit again. I'll give that custom partition table a go. My project does require OTA too, so that potential issue you detailed might save me a whole lot of time and frustration!
My project is 1.6mb with BLE ( 1M BLE ) and the OTA is over BLE - when we started if you try to transport 1.6MB over BLE it was taking about 28 minutes - which is crazy - we managed to get it down to 2 minutes which is OK - but there are a lot of tricks - on which tread to respond - avoid race conditions dealing with WTD and MTU size ... out of the box the BLE lib is only for proof of concept and OTA can work only on WIFI on small projects ..
My project is 1.6mb with BLE ( 1M BLE ) and the OTA is over BLE - when we started if you try to transport 1.6MB over BLE it was taking about 28 minutes - which is crazy - we managed to get it down to 2 minutes which is OK - but there are a lot of tricks - on which tread to respond - avoid race conditions dealing with WTD and MTU size ... out of the box the BLE lib is only for proof of concept and OTA can work only on WIFI on small projects ..
Im having few projects with OTA over BLE, most recent is m5stack factory app with OTA over BLE functionality, which will allow very new users to flash prebuild binaries from android and maybe iOS app too.
I never got any issue with WDT and yes, you have to increase MTU to 500-512 to have acceptable update speed.
Im not saying BLE library size is not a problem, but it is mostly because of esp-idf library size and cant be change. In arduino BLE library we could shrink it by few kB, but it requires a lot of work to remove C++ variable types and replace with plain C types. There is already one PR merged that is optimizing library size and heap usage. If anyone feel confident feel free to create PR with changes.
I'm kind of seriously surprised that you able to get stability from OTA over BLE with big near 2mb bin uploads without issues with the WTD - there were plenty of users in the forums with similar issues - in the beginning we got the OTA in source and fixed it, later decided to just patch it using the the described way - the video is when we were at the middle of the optimisation - from 24min to 15min - currently is 2 minutes..
the MTU that we use is :
// ble constants
#define BLE_DEFAULT_MTU 500
and the safe size for chunk to be erased without WTD is :
//OTA constats
#define OTA_ERASE_SIZE (512 * 1024)
we also have the same app on iOS
also when initialising BLE the following setting are required to work stable on iOS and Android with Bluetooth 4.2 before 4.2 things are not good
bleServer->getAdvertising()->setScanResponse(true);
bleServer->getAdvertising()->setMinPreferred(0x06);
bleServer->getAdvertising()->setMaxPreferred(0x12);
and
class bleServiceCallbacks: public BLEServerCallbacks
{
void onConnect(BLEServer* pServer, esp_ble_gatts_cb_param_t *param)
{
pServer->updateConnParams(param->connect.remote_bda, 0x06, 0x12, 0, 100);
....
also I'm suggesting using flash frequency 40mhz and DIO -to play safe and get production stability
also the fastest way to transport data over BLE is to have request/response type type style e.g. the phone sends , the client replay that it is received the phone sends next chunk - the replay need to be done on the BLE lib callback thread ( and take good care about race conditions and as simple as possible synchronisations) the data sometime get truncated by the BLE lib e.g. len of the sent packet should be the first bytes of the packet - |I will also suggest a simple 1 byte checksum
the speed that we currently get is about 5-7KBytes / sec
I have simplified, maybe not production ready, OTA over BLE code. No data check.
The best (fastest) option is to use notifications, then indications or read depend on esp32 role.
I am also using 500 bytes packets and dont erase flash during OTA update.
From notification/indication callback call function to flash data chunk, thats all.
I dont remember exact numbers now, but IIRC with notifications its possible to achieve about 100-110 kbits per sec.
[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.
This library below might be worth mentioning as a light weight BLE only lib for ESP32;
https://github.com/h2zero/NimBLE-Arduino
I was flashing esp32 using examples->camera webserver. I was getting similar error(sketch is too big).
Just followed the above. And resolved it..........................
"Arduino IDE: Tools -> Partition Scheme:change "Default" to "Huge APP(3MB No OTA)""
A crude workaround for the big Bluetooth library is to employ an ESP32 module with a bigger Flash memory.
There are Wroom modules with 16MB flash which is 4x the standard one, for a few cents more.
A crude workaround for the big Bluetooth library is to employ an ESP32 module with a bigger Flash memory.
There are Wroom modules with 16MB flash which is 4x the standard one, for a few cents more.
I have done dozens ble apps, including AWSfreeRTOS with ble, and i never had to use bigger flash size than 4MB. Always with OTA. Also you can find modules with 8MB flash.
Tools -> Partition sheme -> Huge App (select this Partition)
Tools -> Partition sheme -> Huge App (select this Partition)
This works for me.
I faced the same problem SO I fixed mine just to change some settings in Arduino IDE. Tools -> Partition Scheme: change "Default" to "Huge APP(3MB No OTA)"
So I selected this Huge App Partition and my huge code got accepted with no freaking errors for my ESP32 Board.
<3333333333333333333333333333
Has anyone found a small BT Classic library? I'm having size issues with BLE library + OTA (i need both), I'll not use BLE, only classic bluetooth.