AsyncWebSocketServer_RP2040W crashes with "[AWS] ERROR: Too many messages queued"

Question

AsyncWebSocketServer_RP2040W crashes with "[AWS] ERROR: Too many messages queued"

drmue opened this issue 2 years ago · 3 comments

Dear Khoih,

Thank you for release 1.3.0. However, the AsyncWebSocket server crashes after a while.
In my current project, a value is transferred to the browser every 20 ms and displayed there in a canvas.
After a while the canvas freezes.
After disconnecting and reconnecting the serial monitor, it says "[AWS] ERROR: Too many messages queued".
I implemented a counter on the website to see after how many transmissions the crash occurs. The number of transfers seems to be random after the crash happens.

Load PicoAsyncSocB onto the Pi pico W
Connect to the socket server with a browser
Wait a moment

As you might have guessed, I've also tried all of this with "ESPAsyncWebServer" on an EspNode board, and nothing crashes there.

Best regards
drmue

PicoWAsyncSockB.zip

Arduino IDE version: 1.8.19
RP2040 core v2.6.0
RASPBERRY_PI_PICO_W Module
OS: Ubuntu 20.04 LTS
Linux xy-Inspiron-3593 5.15.0-48-generic #54~20.04.1-Ubuntu SMP Thu Sep 1 16:17:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Answer 1 · 2022-10-10T15:23:42.000Z

Probably not related, but may be of interest.
I have been trying to get a stable web server to run on my Pico-W for some time, I have tried reverting to the example web server in the Arduino IDE examples for the Pico-W. Even with that, I get random deaths of the server response. Though that generally ends with not managing to get a reconnection to the Pico-W even though it still responds to Pings, and can Ping external targets.
During my investigations, I have also seen variation in the response times for the Pico-W, so wonder if your random failure could be linked with this variability in responses, possibly causing a backlog of messages.
This would point to a lower level of the Pico-W driver code as being a problem.

Answer 2 · 2022-10-11T02:48:08.000Z

Hi all,

There are something you have to take into consideration

WebSockets (WS) has pros and cons,compared to regular TCP regular Sockets. WS must be used correctly, when absolutely necessary in certain use-cases. It's not designed for very high speed because of some trade-offs for two-way communication.

From RFC6455

The WebSocket Protocol enables two-way communication between a client running untrusted code in a controlled environment to a remote host that has opted-in to communications from that code. The security model used for this is the origin-based security model commonly used by web browsers. The protocol consists of an opening handshake followed by basic message framing, layered over TCP. The goal of this technology is to provide a mechanism for browser-based applications that need two-way communication with servers that does not rely on opening multiple HTTP connections (e.g., using XMLHttpRequest or <iframe>s and long polling).

Also check Writing_WebSocket_servers

Also check this thread in Teensy Forum Thread: Teensy 4.1 high rate websocket communication

When i send like 5000 messages in a for loop the Teensy crashes and the connection is lost. If i send 50 messages to the teensy the teensy does not crash and the connection remains.

I’m curious why websockets would be better than a regular socket here. Seems like a little more overhead. According to http://eng.kifi.com/websockets-vs-regular-sockets/, websockets are for browser-to-server communication when you can’t do regular sockets.

Thank you for suggesting sockets! I wrote an implementation and it is so much faster! I can send around 200000 messages per second . This is way more than i need and it is very stable, no more crashing of the Teensy. You made me very happy thank you!

Anyway, I've modified the library to provide a little bit better performance, but don't expect it will permit you sending extremely high number of messages per seconds.

Testing OK using the new version with 30ms between WebSockets messages using the following modified code of yours

#include <AsyncWebServer_RP2040W.h>

#include "webpage.h"

char ssid[] = "your_ssid";        // your network SSID (name)
char pass[] = "12345678";         // your network password (use for WPA, or use as key for WEP), length must be 8+

String JSONtxt;

AsyncWebServer server(80);
AsyncWebSocket ws("/ws");

AsyncWebSocketClient * globalClient = NULL;

void onEvent(AsyncWebSocket * server, AsyncWebSocketClient * client, AwsEventType type, void * arg, uint8_t *data, size_t len)
{
  if (type == WS_EVT_CONNECT) 
  {
    Serial.println("Websocket client connection received");
    globalClient = client;
    //client->text("Hello from Pico W_Server");
  } 
  else if (type == WS_EVT_DISCONNECT) 
  {
    Serial.println("Websocket client connection finished");
    globalClient = NULL;
  }
}

void handleRoot(AsyncWebServerRequest *request)
{
  request->send(200, "text/html", webpageCont);
}

void setup()
{
  Serial.begin(115200);
  delay(2000);
  WiFi.begin(ssid, pass);
  Serial.println(WiFi.localIP());
  Serial.println("");

  ws.onEvent(onEvent);
  server.addHandler(&ws);

  server.on("/", handleRoot);
  server.begin();
}

void heartBeatPrint()
{
  static int num = 1;

  Serial.print(F("."));

  if (num == 80)
  {
    Serial.println();
    num = 1;
  }
  else if (num++ % 10 == 0)
  {
    Serial.print(F(" "));
  }
}

void sendStatus()
{
  static unsigned long sendStatus_timeout  = 1000;
  static unsigned long checkstatus_timeout = 1000;

#define STATUS_CHECK_INTERVAL     10000L

#define SEND_INTERVAL             30L

  // Send status report every SEND_INTERVAL (30) millis
  if (millis() > sendStatus_timeout)
  {
    //if (globalClient != NULL && globalClient->status() == WS_CONNECTED)
    // Sending only when gessage_queue not full
    if (globalClient != NULL && globalClient->status() == WS_CONNECTED && globalClient->canSend()) 
    {
      //String POTvalString = String(analogRead(A0) / 4);
      String POTvalString = String(rand() % 256);
      
      JSONtxt = "{\"POT\":\"" + POTvalString + "\"}";
           
      globalClient->text(JSONtxt);
    }
    
    sendStatus_timeout = millis() + SEND_INTERVAL;
  }

  // Send status report every STATUS_REPORT_INTERVAL (10) seconds
  if (millis() > checkstatus_timeout)
  {
    heartBeatPrint();
    checkstatus_timeout = millis() + STATUS_CHECK_INTERVAL;
  }
}

void loop()
{
  sendStatus();
}

Still running

It's possible there is some internal bug (CYW WiFi Driver and/or arduino-pico core ???) where every 5 seconds, the WiFi slow downs / sleeps => WS messge_queue not processed => messge_queue overflowed

15:13:16.443 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:16.476 -> [AWS] ERROR: Large MsQ
15:13:16.509 -> [AWS] ERROR: Large MsQ
15:13:21.318 -> [AWS] ERROR: Large MsQ
15:13:26.459 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:31.268 -> [AWS] ERROR: Large MsQ
15:13:31.268 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:36.444 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:41.419 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:41.453 -> [AWS] ERROR: Large MsQ
15:13:46.428 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:46.495 -> [AWS] ERROR: Large MsQ
15:13:51.769 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:51.802 -> [AWS] ERROR: Large MsQ
15:13:51.835 -> [AWS] ERROR: Large MsQ
15:13:56.445 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:13:56.478 -> [AWS] ERROR: Large MsQ
15:14:01.421 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:01.454 -> [AWS] ERROR: Large MsQ
15:14:06.397 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:06.430 -> [AWS] ERROR: Large MsQ
15:14:11.770 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:11.803 -> [AWS] ERROR: Large MsQ
15:14:11.836 -> [AWS] ERROR: Large MsQ
15:14:21.289 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:21.322 -> [AWS] ERROR: Large MsQ
15:14:26.431 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:31.540 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:31.573 -> [AWS] ERROR: Large MsQ
15:14:36.414 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:36.481 -> [AWS] ERROR: Large MsQ
15:14:36.514 -> [AWS] ERROR: Large MsQ
15:14:46.463 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:46.463 -> [AWS] ERROR: Large MsQ
15:14:51.338 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:14:51.371 -> [AWS] ERROR: Large MsQ
15:14:56.479 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:15:01.554 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:15:01.554 -> [AWS] ERROR: Large MsQ
15:15:06.429 -> [AWS] ERROR: Large MsQ         <=========== Every 5s, getting error. WiFi Driver slow down ???
15:15:06.495 -> [AWS] ERROR: Large MsQ

The USB power supply for RP2040W is also important because of WiFi. Check if you have enough power. I saw the board crashed if using bad (noisy, lower) USB hubs.
The WiFi router / network is also important. Check and use the strong and reliable router. I also saw the message_queue overflowed more frequently if using old slower model of WiFi Router.

ESP32 is more mature and more powerful (240MHz <-> 133MHz) than RP2040W, especially the WiFi
So I still suggest you to use ESP32 over RP2040W for project requiring reliability / more power, at least for now.

I'm going to release new version v1.3.1 within hrs. Please wait.

Answer 3 · 2022-10-11T03:47:25.000Z

Hi @drmue

Again, the AsyncWebServer_RP2040W v1.3.1 has just been released. Please test and report any more bug / issue

Best Regards,

Release v1.3.1

Improve robustness of AsyncWebSockets server. Check AsyncWebSocketServer_RP2040W crashes with "[AWS] ERROR: Too many messages queued" #6
Add example Async_WebSocketsServer_Xtreme to demo the nearly highest possible WebSockets Server speed