Low performance of NOTIFY or INDICATE
Opened this issue · 9 comments
We are sending cca 120 notifications per second. When we subscribe to indications with nRF Connect Android App then ESP32 (as GATTS) device quickly runs out of heap memory because elements from pending_inds in esp32_bt_gatts.c are removed much less ofthen than they are added.
With notifications only (no indications) it is better but still it fills up. What is interesting that if we subscribe to that notifications with another ESP32 (as GATTC) then the performance of GATTS is even worse.
This can be reproduced by creating project which uses https://github.com/mongoose-os-libs/bt-service-debug, logging 200 messages per second and turnign of indications on the client.
Is there anything we can check?
ok, so two issues here:
- there should be a limit on the number of items we queue so that the system degrades gracefully when under load - start dropping notifications but not crash
- increase performance to be able to process more notifications.
(1) is easy, (2) is more challenging. what kind of performance are you looking for? would just (1) be enough?
Major bottleneck seems to be on the client when it calls mgos_event_trigger_schedule in ESP_GATTC_NOTIFY_EVT case in file esp32_bt_gattc.c. Initially ESP_GATTC_NOTIFY_EVT case is executed 120x per second and then it gets very quickly "saturated" (probably because of rtos task queue full) and is executed cca 4x per second. If we comment mgos_event_trigger_schedule call then the rate of ESP_GATTC_NOTIFY_EVT doesn't decrease.
This also causes problems on the server, because ESP_GATTS_CONF_EVT is then surprisingly delayed and it doesn't catch emptying of pending_inds. I said surprisingly because in case on notifications (unconfirmed) this should be called immediately on sending without asking on response (see espressif/esp-idf#749).
I think that in mgos_bt_gatts_notify there is no need to put anything to queue unless mode == MGOS_BT_GATT_NOTIFY_MODE_INDICATE. It would then prevent that problems on server.
BTW nRF Connect app has no problems reading the data without observed data loss from the ESP32 client (used recording functionality).
Regarding slow mgos_event_trigger_schedule our MGOS_BT_GATTC_EV_NOTIFY event handler was during the test empty.
BTW nRF Connect app has no problems reading the data without observed data loss from the ESP32 client (used recording functionality).
But it also causes memory exhaustion by big pending_inds.
I am now playing with bare ESP-IDF gatt_client and gatt_server examples and there is no problem sending and receiving 100 notifications per second.
...also no problem with 300 notif per second.
Did you make further progress with this, did you end up simply using ESP-IDF?
I am using my hacks directly accessing ESP-IDF API. Also the project is now hibernating (you know, start-ups...).