Stream does not get new data when data is pushed

[tickietackie]

Hi,
I am using an Arduino Nano IoT to read a stream in the RTDB with the stream example at the path "/table_config/CUywk9RhoFTWTZTD4KuNcMvFDBC2/mcu_read/123456789"

I am pushing data with the Firebase JS SDK to the database to replace the data at the root node. When doing this, the data does not get received every time I would expect it. Therefore I checked another library also wrapping the Firebase REST API with Python, which should then be comparable to this one, right?

As you can see in the screenshots the data is received every time with python, but not on the Arduino. I also can see the data changing in the web console every time.
I am sorry, but I just do not understand that behaviour, as the Python library https://github.com/thisbejim/Pyrebase also works with the REST API, which should then get the same stream data in my opinion.

Could you please explain what could be a reason for this? Do I really misunderstand something here with this library?
What would also really help me, is a way to further debug this. Is there a possibility?

Below you will find the Arduino code, the python code, js code and the json, that I am pushing.

Thanks in advance for your help.

Arduino code:

/*
 * Created by K. Suwatchai (Mobizt)
 * 
 * Email: k_suwatchai@hotmail.com
 * 
 * Github: https://github.com/mobizt
 * 
 * Copyright (c) 2021 mobizt
 *
*/

//Example shows how to connect to Firebase RTDB and get stream connection

//Required WiFiNINA Library for Arduino from https://github.com/arduino-libraries/WiFiNINA

#include "Firebase_Arduino_WiFiNINA.h"

#define DATABASE_URL "" //<databaseName>.firebaseio.com or <databaseName>.<region>.firebasedatabase.app
#define DATABASE_SECRET ""
#define WIFI_SSID ""
#define WIFI_PASSWORD ""

//Define Firebase data object
FirebaseData fbdo;

FirebaseData stream;

unsigned long sendDataPrevMillis = 0;

String path = "/table_config/CUywk9RhoFTWTZTD4KuNcMvFDBC2/mcu_read/123456789";

int count = 0;

void setup()
{

  Serial.begin(115200);
  delay(2000);
  Serial.println();

  Serial.print("Connecting to Wi-Fi");
  int status = WL_IDLE_STATUS;

  while (status != WL_CONNECTED)
  {
    status = WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
    Serial.print(".");
    delay(3000);
  }
  Serial.println();
  Serial.print("Connected with IP: ");
  Serial.println(WiFi.localIP());
  Serial.println();

  //Provide the autntication data
  Firebase.begin(DATABASE_URL, DATABASE_SECRET, WIFI_SSID, WIFI_PASSWORD);
  Firebase.reconnectWiFi(true);

  if (!Firebase.beginStream(stream, path))
  {
    Serial.println("Can't connect stream, " + stream.errorReason());
    Serial.println();
  }
}

void loop()
{

  
  if (!Firebase.readStream(stream))
  {
    Serial.println("Can't read stream, "+ stream.errorReason());
  }

  if (stream.streamTimeout())
  {
    //Serial.println("Stream timed out, resuming...");
  }

  if (stream.streamAvailable())
  {
    count++;
    if (stream.dataType() == "null")
      count = 0;

    Serial.println("Stream data received... ");
    Serial.println("stream path: " + stream.streamPath());
    Serial.println("event path: " + stream.dataPath());
    Serial.println("data type: " + stream.dataType());
    Serial.println("event type: " + stream.eventType());
    Serial.print("value: ");
    if (stream.dataType() == "int")
      Serial.println(stream.intData());
    if (stream.dataType() == "int64")
      Serial.println(stream.int64Data());
    if (stream.dataType() == "uint64")
      Serial.println(stream.uint64Data());
    else if (stream.dataType() == "double")
      Serial.println(stream.doubleData());
    else if (stream.dataType() == "float")
      Serial.println(stream.floatData());
    else if (stream.dataType() == "boolean")
      Serial.println(stream.boolData() == 1 ? "true" : "false");
    else if (stream.dataType() == "string")
      Serial.println(stream.stringData());
    else if (stream.dataType() == "json")
      Serial.println(stream.jsonData());
    else if (stream.dataType() == "array")
      Serial.println(stream.arrayData());
    Serial.println();
  }
}

Python code:

import pyrebase

config = {
    "apiKey": "",
    "authDomain": "",
    "databaseURL": "",
    "storageBucket": ""
}

firebase = pyrebase.initialize_app(config)

# Get a reference to the database service
db = firebase.database()
auth = firebase.auth()
user = auth.sign_in_with_email_and_password("tester@pongate.de", "12345678")

# Pass the user's idToken to the push method
#results = db.child("users").push(data, user['idToken'])


def stream_handler(message):
    print("event: " + message["event"])

    # >> put

    print("path: " + message["path"])

    # >> /-K7yGTTEp7O549EzTYtI

    print(message["data"])
    print("")
    # >> {'title': 'firebase', "body": "Python interface to the Google's Firebase REST APIs"}


my_stream = db.child("table_config").child("CUywk9RhoFTWTZTD4KuNcMvFDBC2").child(
    "mcu_read").child("123456789").stream(stream_handler, user['idToken'])

Javascript sdk using push function:

const sendConfig = (tableConfig: InitialTableState, tableId: string | number | null):
    Promise<ISendApiRequestResponse> => {
    return new Promise((resolve, reject) => {
        const userId = auth.currentUser?.uid
        const idPath = tableId !== null ? tableId : "general";
        const now = new Date();
        tableConfig.last_updated = Math.round(now.getTime() / 1000);

        db.ref(`/table_config/${userId}/mcu_read/${idPath}`).
            push(tableConfig).then((data) => {
                if (__DEV__) console.log("Updated whole config in fb db.")
                resolve({
                    error: false,
                    code: 0,
                });
            }).catch((error) => {
                console.log('error ', error)
                reject({
                    error: true,
                    code: -1,
                    message: error.message,
                });
            })
    })
}

The JSON:

{
    "configuration_per_state": {
        "Idle": {
            "allowed_effects": [
                0,
                2,
                6,
                1,
                3,
                5,
                4
            ],
            "brightness": 31,
            "default_effect": {
                "type": 0
            }
        },
        "In_Game": {
            "allowed_effects": [
                2,
                6,
                1,
                3,
                5,
                4
            ],
            "brightness": 57,
            "duration": 1200,
            "field_master": {
                "default_effect": {
                    "type": 2
                }
            },
            "field_slave": {
                "default_effect": {
                    "type": 2
                }
            },
            "limit_game_duration": true,
            "remaining_duration": 1200,
            "state": 0
        }
    },
    "id": "123456789",
    "last_updated": 1645558291,
    "name": "test",
    "state": "Idle",
    "wifi_mode": 1
}

[mobizt]

I can confirm you that the library works normally for stream.

From intensive test with 100,000 to 500,000 continuous data sending in loop without delay test in Android and Web clients, and use library to listen the stream, no data was missed from routine test.

I don't know what is the cause of your issue other than network issue but I did not see the timed out.

You should check your stream path in all clients.

[mobizt]

The RTDB stream does not have quality of service as MQTT that can resume the data when it failed to push to client.

All stream data received on client are the server changes on the stream listening path.

You should make sure that your client making some change the data under streaming path.

There is nothing blocking in the stream operation of this library as the socket connection is always kept open to listening the stream and has the method to check the timeout by looking for the keep-alive event data to trigger the timeout error when it is missing more than 40 seconds.

[tickietackie]

I got it working! Finally!
And the issue is as dumb as it could be. The size of the data is simply too big.
I updated the FIREBASE_RESPONSE_SIZE to 700 and suddenly everything worked ...
That's also the reason, some data came through, as then the data was simply small enough.
Thanks for your help so far. I think this is also good for other people, seeing the limits, that are possible here with the capabilities of the Arduino.