ESP8266 crashes when is receiving data from websocketserver

Question

ESP8266 crashes when is receiving data from websocketserver

elC0mpa opened this issue 5 years ago · 29 comments

Describe the bug
I am using your library to receive a JSON file from a websocketserver. When it receive the file, it starts printing hexadecimal numbers and after two seconds aproximately the esp8266 restarts.

To Reproduce
Version : 0.4.11
Board : NODEMCU
Only using this library

Expected behavior
I expect to receive the json file, wich is received as a string, and later i want to print it By using serial library

Additional context
I supposed that this problem was related to the receive buffer size, so I execute the same code, but receiving a short json file and it worked as expected, so I think the problem is not related to code.

Answer 1 · 2019-10-10T00:29:44.000Z

Hi Jose,

Sounds weird, let's try to solve it. Can you write what software you use on the server? I'll try to setup a similar server and see if I have issues with long messages as well. If I can access the same server you are using it will be best (if it's a public service).

Also, can you turn on the esp8266 debug logs (from the boards menu) and attach those.

Thanks,
Gil.

Answer 2 · 2019-10-10T02:46:56.000Z

Thanks for your answer @gilmaimon, I can not publish the server code because i don't have it and it is written in php. I need to comunicate with that server wich will be running on a Raspberry pi. Let me ask you something, which is the maxium lenght of received data in this library, because i am pretty sure that this is the problem. Remember it worked with the same code when i changed the size of the json file.

Answer 3 · 2019-10-10T09:05:55.000Z

Hi Jose,

There is no limitation.

How big is the file you are getting, could the esp be out of memory? Do you have an esp32 around?

Answer 4 · 2019-10-10T10:08:04.000Z

Hi Jose,
Let me say few words, perhaps will help you:

1. First of all, you told me you are using PHP backend, which makes me think you are using some websocket implementation found on the internet;
Most of those I found in the beginning, do not really read the headers properly and cannot handle long messages, they just truncate it or throw an error.
Been there, done that, ended up creating my own correct implementation by reading the RFCs, works like a charm now, HOWEVER

2. I did some testing on few Arduino modules, though I work with ESP32. The maximum amount I could safely send was 16kb, working with it as a client. I sometimes reached more (about 24kb), but from what I've tested, this was the safest amount of data to send. Anyway, should be a lot. I'm talking here about one single message, you could send as much as you want in multiple messages, by encoding & chunking it.

I don't think its a limitation of the library (and @gilmaimon confirmed so), but rather of the ESP32's Arduino libraries that handle the strings. Didn't investigate further, as 16kb were enough for me. Even the browsers limit the amount of data you can send/receive (and that's 128kb); seems a bit more, but browsers have much more memory than an micro controller.

Hope it helps a bit,
All the best,
A.

Answer 5 · 2019-10-10T12:47:01.000Z

Thanks for your answer @gilmaimon and @adelin-mcbsoft. First of all, I have no an Esp32 to test if the problem is related to RAM, but i am pretty sure that the file is not so big. I will check it later.

1. First of all, you told me you are using PHP backend, which makes me think you are using some websocket implementation found on the internet;
Most of those I found in the beginning, do not really read the headers properly and cannot handle long messages, they just truncate it or throw an error.
How can i know if this php implementation is correct? Remember that using the same code, I changed only the json file and everything worked as expected. Maybe it is something related to frames, but i am not sure.
Thanks for your answer

Answer 6 · 2019-10-10T13:00:58.000Z

Cool, let us know when you know more about the file size.

Regarding the correctness of the implementation. I assume it is correct, but might require the use of some extension I'm not supporting. Anyway, it shouldn't crash the esp.

If you can use EspExceptionDecoder to decode the crash stacktrace (those hex bytes) it will help identify what exactly is failing.

Also I'll point out again those ESP8266 debug logs, which might help.

Thanks,
Gil.

Answer 7 · 2019-10-12T05:24:15.000Z

Ok @gilmaimon, let me tell you something. First of all, I used the ESPAsyncWebServer to simulate the same behavior as the php websocketserver. By doing this i can identify if the problem is related to the server or to the client. Because of the problem remains, I was able to detect that the problem is not related to the server. The file size is about 4 kB, reason why we can descart that the esp is running out of memory.
I modified the server code to send Just a part of the file, not entirely. Because i send the json file as a string, I just decided to send 500 chars, and it worked. But after that i decided to send 600 chars and the Esp01 which is acting as the websocketclient crashed. Tomorrow i will take a look to EspExceptionDecoder, I hope it could Help me

Answer 8 · 2019-10-12T13:07:27.000Z

Hi Jose,

I don't think this is a library limitation or bug. There is no difference between message of size 500 and of size 600 (considering the code which runs inside the library in each case). Also, the same logic running on my PC can handle 4K messages with no problem - which is also why I don't think this is a logic bug.

Best thing to do is to decode the stack trace, this will let us know if the crash is related to memory or something else. I wouldn't discard running out of memory, the memory usage can always be improved and there are other stuff using your esp RAM. Also, some of the memory is used for the internal TCP buffers (I think around 8K?).

Best steps to take are:

Share the code you are using exactly (if you don't mind sharing)
Decode the crash and print the decoded stack trace (make sure to censor private information like paths on your PC)

I'm sure we will sort this out.
Gil.

Answer 9 · 2019-10-12T14:17:49.000Z

Thanks @gilmaimon, I will follow your advice and later i will tell you how it was. Thanks for your answer

Answer 10 · 2019-10-13T01:24:21.000Z

Hello @gilmaimon, I already used the espExceptionDecoder, and it wasn't succesfull. The thing is, that it looks like my problem is not an exception. When it happens, the serial monitor shows : soft WDT reset and after that, it shows the stack trace, so it doesn't decodify any exception. I will send you the json file, and maybe you could Help me By receiving it using your library. I will be waiting for your answer, thanks in advance

Answer 11 · 2019-10-13T08:23:46.000Z

Jose, look how deep we are into this discussion and this is the first time you mention that the exception is a soft WDT reset...

Please copy and paste the crash log here
Please share the code you are using
Please try again to decode the stack trace
The error you mention is called because esp8266 has a thing called software watchdog. The watchdog will reset your sketch if it is stuck for too long. It is possible that receiving a 4k file just takes too long. It is also possible that you have some other code that is causing the issue.

Helpful resources:

Gil.

Answer 12 · 2019-10-13T14:18:52.000Z

Thanks for sharing these links @gilmaimon, I learned some things By reading them, I decided to share my code and my json file with you. I really hope you could Help me. I followed the steps of how to use the espExceptionDecoder, but it doesn't worked. Despiste this, I am pretty sure that the soft WDT overflows in the line number 22, which is the line where i read the incoming msg from the websocketserver. Thanks in advance
Json file

{
   "clientes": {
     "1": {
       "id": 1,
       "nombre": "Cliente 1",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 4,
       "tiempo_en_area": 4,
       "tiempo_restante_en_area": 0,
       "area": {
         "id": 6,
         "nombre": "Piscina",
         "icono": "fa-support",
         "tiempo_en_area": 4,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-25 15:06:35"
       }
     },
     "2": {
       "id": 2,
       "nombre": "Cliente 2",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 2,
       "tiempo_en_area": 4,
       "tiempo_restante_en_area": 2,
       "area": {
         "id": 6,
         "nombre": "Piscina",
         "icono": "fa-support",
         "tiempo_en_area": 4,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-25 15:06:35"
       }
     },
     "3": {
       "id": 3,
       "nombre": "Cliente 3",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 1,
       "tiempo_en_area": 1,
       "tiempo_restante_en_area": 0,
       "area": {
         "id": 1,
         "nombre": "Habitaci?n Ba?o",
         "icono": "fa-bath",
         "tiempo_en_area": 1,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-24 19:58:55"
       }
     },
     "4": {
       "id": 4,
       "nombre": "Cliente 4",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 1,
       "tiempo_en_area": 1,
       "tiempo_restante_en_area": 0,
       "area": {
         "id": 4,
         "nombre": "Parqueo - Mascota",
         "icono": "fa-github-alt",
         "tiempo_en_area": 1,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-24 19:58:55"
       }
     }
   },
   "preferencias": {
     "id": 1,
     "limite_de_basura": 100,
     "coeficiente_de_basura": 2,
     "tiempo_de_simulacion": 1,
     "dia": "07:00 AM",
     "noche": "07:00 PM",
     "cant_actuadores_energia": 5,
     "cant_actuadores_agua": 5,
     "created_at": "2019-09-24 19:58:55",
     "updated_at": "2019-10-02 16:07:54"
   },
   "simulacion": {
     "basura_acumulada": 100,
     "dia": 176,
     "hora": 18,
     "cantidad_clientes": 4,
     "sesion": "dia"
   }
}

platformio code

#include <Arduino.h>
#include <ArduinoWebsockets.h>

#define SSID "AsyncWebSocketTest"
#define PSWD ""
#define PORT 82
#define IPADDRESS "192.168.0.1"
#define ROUTE "/"
#define TEST true
#define BAUDRATE 115200

#define USE_SERIAL Serial

using namespace websockets;

WebsocketsClient client;

#pragma region  Events

void onMessageCallback(WebsocketsMessage msg)
{
  auto data = msg.data();
  Serial.println(data);
}

void onEventsCallback(WebsocketsEvent event, String data)
{
  if (event == WebsocketsEvent::ConnectionOpened)
  {
    Serial.println("Connection Opened");
  }
  else if (event == WebsocketsEvent::ConnectionClosed)
  {
    Serial.print("Connection Closed: ");
    auto code = client.getCloseReason();
    Serial.println(code);
  }
  else if (event == WebsocketsEvent::GotPing)
  {
    Serial.println("Got a ping");
  }
  else if (event == WebsocketsEvent::GotPong)
  {
    Serial.println("Got a pong");
  }
}
#pragma endregion

void setup() {
  // put your setup code here, to run once:
  USE_SERIAL.begin(BAUDRATE);

	USE_SERIAL.setDebugOutput(true);

	USE_SERIAL.println();
	USE_SERIAL.println();
	USE_SERIAL.println();

	for(uint8_t t = 4; t > 0; t--) {
		USE_SERIAL.printf("[SETUP] BOOT WAIT %d...\n", t);
		USE_SERIAL.flush();
		delay(1000);
	}

  WiFi.mode(WIFI_STA);
	Serial.println("Trying to connect...");
	WiFi.begin(SSID, PSWD);
	Serial.println("Connected to Wifi");
	delay(1000);
	Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
	  
	client.onMessage(onMessageCallback);
  client.onEvent(onEventsCallback);
  
  #if TEST
  client.connect(IPADDRESS, PORT, ROUTE);
  #else
  client.connect("ws://192.168.1.11:6001");
  #endif
  client.ping();
}

void loop() {
  // put your main code here, to run repeatedly:
  client.poll();
}

EDIT: I've allowed myself to edit your comment just so the code is easier to read (@gilmaimon)

Answer 13 · 2019-10-13T23:08:43.000Z

I'm having some troubles with my esp8266, but I think I solved it.
Can you try again with the changes commited to master?

Gil.

Answer 14 · 2019-10-13T23:42:42.000Z

Ok, will try later and tomorrow i will give you the feedback. Could you tell me which was the problem? I would like to know, because despite i have been working with Microcontrollers since a long time, this topic is out of my confort zone. Thanks again

Answer 15 · 2019-10-14T10:16:43.000Z

Sure,

ESP8266 has a watchdog that makes sure that TCP buffers and network operations are done within some set intervals. Those operations occur when calling delay or yield (which is called by delay).

I've added yield in random places in the library and it seems to fix the issue. So I moved the yields from those random places to the classes in the library that wraps WiFiClient and other esp-networking stuff. Every operation now calls yield so I can be certain it is fairly regularly.

With yield called regularly, even when a big message is being read, the watchdog seems to not alert and reboot the program. However as I've said, I'm having some troubles with my esp8266. So let me know if it works for you.

By the way, on esp32 your sketch works perfectly.

Gil.

Answer 16 · 2019-10-14T11:53:41.000Z

Thanks @gilmaimon, I am going to work right now, but when i finsih, I will try it. 👌🏻

Answer 17 · 2019-10-15T02:17:43.000Z

Hello @gilmaimon, let me tell you that now it is worst, when the esp starts executing the code, it restarts and prints the Next message:
ets Jan 8 2013, rst cause:1, boot mode :(3, 6).
I hope you could fix this. Please remember that i want to keep using your library 🤝

Answer 18 · 2019-10-15T04:33:11.000Z

I don't have an esp8266 around me until the weekend, but I can guess what is causing the issue. Can you try again with the changes commited to master?

Answer 19 · 2019-10-15T10:27:43.000Z

Ok @gilmaimon, I will try later and i will update you with the results 👌🏻

Answer 20 · 2019-10-15T23:48:37.000Z

Hello @gilmaimon, I realized that you removed the yield() function from the constructor of the class. The thing is that the problem is still there, Just when the esp8266 starts executing the code, it restarts. Have you another idea of what could be causing this?

Answer 21 · 2019-10-16T05:03:18.000Z

Does it crash immediately or when connecting? I'll keep working on it this weekend.

Answer 22 · 2019-10-16T05:24:02.000Z

I believe the crash is happening when the client c'tor is called. I'm not sure why it crashes, the same exact code runs on esp32 without crashing. There is probably some bug in the esp8266 specific code in my library or in the esp8266 libs themselves.

Any ways, the issue happens because the c'tor is called before setup. If you would change the code such that it delays initialization until setup, things should work. This is not a fix, but a temporary workaround.

Try:

WebsocketsClient* client;

void setup() {
   // stuf...
   client = new WebsocketsClient();
   client->connect(.......);
}

You will need to replace you accessors with -> instead of a . because client is now a pointer.
Let me know if it worked, it should. I will look into it again this weekend and hopefully figure out whats causing the issue.

Gil.

Answer 23 · 2019-10-16T06:22:40.000Z

Does it crash immediately or when connecting? I'll keep working on it this weekend.

It crashes inmediately @gilmaimon

Answer 24 · 2019-10-16T06:26:27.000Z

I believe the crash is happening when the client c'tor is called. I'm not sure why it crashes, the same exact code runs on esp32 without crashing. There is probably some bug in the esp8266 specific code in my library or in the esp8266 libs themselves.

Any ways, the issue happens because the c'tor is called before setup. If you would change the code such that it delays initialization until setup, things should work. This is not a fix, but a temporary workaround.

Try:
WebsocketsClient* client;

void setup() {
   // stuf...
   client = new WebsocketsClient();
   client->connect(.......);
}
You will need to replace you accessors with -> instead of a . because client is now a pointer.
Of course
Let me know if it worked, it should. I will look into it again this weekend and hopefully figure out whats causing the issue.
I will do it and i will comment you about the results
Gil.

Answer 25 · 2019-10-16T21:23:36.000Z

Perfect @gilmaimon, now the esp8266 is able to receive the entire json file without beign restarted because of the software WDT. I will be waiting for you to fix the problem of the constructor of the class. Please notify me when it will be able to download as a New version of the library. If you need me to test it before the release, let me know. 👌🏻

Answer 26 · 2019-10-18T13:52:04.000Z

Hi Jose
I believe I fixed the issue. Seems like there was another function that was called in the c'tor which had a yield in it (which I recently added) and it caused the issue.

Overall, I think it is a bug that calling yield before setup crashes the esp8266 (esp32 can handle it), but I worked around it anyways.

You should be able to revert your code to it's previous state and everything should work fine. Please let me know if it did.

Best wishes,
Gil.

Answer 27 · 2019-10-18T15:42:47.000Z

Hi @gilmaimon, I will download the repo and revert my code to its previous state later, now i am at work. I will let you know when i do it. Thanks in advance 👏🏼

Answer 28 · 2019-10-18T22:10:47.000Z

Bug fixed @gilmaimon. Now it works as expected. You can now publish a New release of the library 👌🏻. Let me ask you a final question : now there is no limit of data to be readed by using this library, is it? 🤓

Answer 29 · 2019-10-19T12:01:59.000Z

Thank you! I've posted the new version and it should be available via the library manager soon.
There should be no limitation beyond the websockets protocol itself (and physical limitations, of course).

Gil.