Disconnect on large websocket message

[Jonty]

Hello!

I'm trying to use this library on an ESP32 against API's that stream very large messages (~40k).

It works fine for the smaller messages, however as soon as a large message (should be) received a WebsocketsEvent::ConnectionClosed event is emitted and the connection is dropped.

I did some brief digging and it looks like the TCP connection stops being available(), but I'm not clear why.

This example code demonstrates the problem:

#include <ArduinoWebsockets.h>
#include <WiFi.h>

const char* ssid = "SSID";
const char* password = "PASSWORD";

const char* websockets_connection_string = "wss://ws.blockchain.info/inv";

using namespace websockets;

void onMessageCallback(WebsocketsMessage message) {
    Serial.print("Got Message: ");
    Serial.println(message.data());
}

WebsocketsClient client;

void onEventsCallback(WebsocketsEvent event, String data) {
    if(event == WebsocketsEvent::ConnectionOpened) {
        Serial.println("ConnectionOpened");
    } else if(event == WebsocketsEvent::ConnectionClosed) {
        Serial.println("ConnectionClosed");
    } else if(event == WebsocketsEvent::GotPong) {
        Serial.println("GotPong");
    }
}

void setup() {
    Serial.begin(115200);

    WiFi.begin(ssid, password);
    for(int i = 0; i < 10 && WiFi.status() != WL_CONNECTED; i++) {
        Serial.print(".");
        delay(1000);
    }

    client.onMessage(onMessageCallback);
    client.onEvent(onEventsCallback);

    client.setCACert(nullptr);
    client.connect(websockets_connection_string);

    // Works - results in "GotPong"
    client.ping();

    // Works - results in "Got Message: {"op":"pong"}"
    client.send("{\"op\":\"ping\"}");

    // FAILS - results in "ConnectionClosed"
    // This will cause ~40k of data to be returned down the websocket
    client.send("{\"op\":\"ping_block\"}");
}

void loop() {
    client.poll();
}

Any help very much appreciated.

[gilmaimon]

I've looked into it abit. This looks like a bug in the library itself, it has nothing to do with memory restrictions on the esp32 or anything like that.

I'm not even sure it has something to do with the message size, I think it's something about the protocol that is not implemented corectly (probably on my end).

I've tried poking around, but nothing too obvious popped out. I'm working on it and will keep you updated.

Thanks,
Gil.

[Jonty]

Oh good, glad it wasn't me being daft! Let me know if there's anything I can do to help debug it.

[gilmaimon]

Thanks again for openning this issue @Jonty!

Look for version 0.4.7, It should resolve your issue. I've tested it locally and it seems to work fine. Iv'e tried running your code on an ESP8266 and it crashed, I guess it makes sense considering how much data is being transferred. (Everything worked fine with an ESP32, after using the new version).

Version 0.4.7 is available as a zip from the release page, and it will be on the Library Manager very soon.

Please make sure everything works as expected on your end.

Best regards,
Gil.

[Jonty]

Just in case this is also relevant, I can't seem to find any way to get the library to reconnect properly when the connection closes like this.

Adding client.connect(websockets_connection_string); to the WebsocketsEvent::ConnectionClosed block does result in the connection reopening, however ping() or send() after that no longer get responses from the server - presumably they're not being sent, or the responses are not being handled.

[Jonty]

Sorry to have to reopen this, but it doesn't appear to be resolved for me!

Running the example code above still results in:

.ConnectionOpened
GotPong
Got Message: {"op":"pong"}
ConnectionClosed

I've fetched the latest zip and installed it - I even removed the library and checked it wouldn't build when it was removed! No dice.

If there's anything I can do to help debug this, let me know.

[Jonty]

Bah, wait, ignore me - this seems to have been a cache problem. It is now reliably crashing so I presume this esp32 doesn't have enough memory!

Thanks.

[gilmaimon]

Well, I get inconsistent crashes. Sometimes the ESP32 does not crash and prints the data correctly. I might try to improve the memory usage of the library, it is far from optimized..

Anyway, I recommend you try another websockets library just to see if it's an optimization problem or a physical restriction.

Let me know if you found a solution (if you don't mind sharing),
Gil.

[Jonty]

I am almost certain that is because the response sizes can vary enormously and it's hit-and-miss as to if the data will fit in memory on a standard ESP32 WROOM module - I've got a much larger WROVER module on the way and will test with that!

[gilmaimon]

Ok cool, good luck!