Keep the connection open-Persistent connection

[garudaonekh]

Hi,
I need to publish the params of user GET/POST request params to a topic and listen for response(1 to 5 seconds) from another topic and echo the response to the user. I think that make new connection to MQTT Server for every request is not a good idea and maybe slow, thus I wish to keep the connection open forever and reconnect when it's lost. However, there might be another issue of concurrency when multiple user request and using the same shared MQTT Server connection.

Any suggestion is much appreciated.

This might be a dummy question since I am from the Arduino world.
Related to: #71

[Namoshek]

You are right that opening a new connection per request is not the best solution, although it depends on the traffic level. If we are talking about 5-10 new connections per second, it won't be a problem at all. If we are talking thousands, this can become an issue quite fast.

Before I can go into more details what solution might be suitable, I need to know whether you really need to respond with the details in the response of the request (because it's an API) or if you need this information for a UI where you could instead also show a loading icon while the details are fetched?

[garudaonekh]

You are right that opening a new connection per request is not the best solution, although it depends on the traffic level. If we are talking about 5-10 new connections per second, it won't be a problem at all. If we are talking thousands, this can become an issue quite fast.

Before I can go into more details what solution might be suitable, I need to know whether you really need to respond with the details in the response of the request (because it's an API) or if you need this information for a UI where you could instead also show a loading icon while the details are fetched?

While online, each user will make AJAX request per 5 seconds. So I want to scale to 100 concurrent users at most, which should be 20 to 30 requests per seconds. The maximum response size is 4KB. Average size is 1.2KB

This PHP script will act as a middleware. We have a separate Cordova app which send request to this PHP script. And it's another iOT device which response to this request. In our current testing it look around 2-5 seconds for the response to come back to the script. Thus the PHP script will loop and wait for response for 10 seconds before it timeout.

[Namoshek]

There are multiple issues with this approach, ranging from timeouts to scalability. Your numbers are by no means an issue though, as most MQTT brokers are used to have frequent connects and disconnects due to low-power IoT devices which go into standby or deep-sleep mode.

As long as you need a specific request and response topic per HTTP request, moving the logic to a queue worker with persistent connections is not really an improvement either. It will still block for multiple seconds while waiting for a response. And you introduce additional challenges, e.g. how to get the response from the queue worker to the client.

That said, however, the cleanest and most scalable design in my opinion would be:

• Let the app make a request which creates a database entry with a unique process id as well as a queue job for the process id. Then the app immediately receives a response with the process id.
• The queue job simply performs the MQTT action, but since the queue worker has a permanently open connection, calling unsubscribe after receiving a response is important. When the queue worker receives the MQTT response, it updates the process entry in the database.
• While all this happens, the app periodically (every 500-1000ms) fetches updates with the process id (short-polling).
• You probably would also need a cronjob which periodically prunes old process entries from the database.

Obviously, that makes the whole stateless application rather complex and introduces additional components. So I would advise against it as long as you don't need it from a scalability point of view.

[garudaonekh]

Your approach of creating response id is great. Somehow, this need me to update the app front end which I rather avoid and no developer resources to do it as it's javascript(cordova) and also a lot more development on the PHP side.

My idea is to wait for response in the PHP script and within 10 seconds, if no response, send timeout to the front-end app.

I think i will just try the simple approach of open MQTT connection for each request if 20-30 connections per seconds not an issue.

Thanks for your quick response and analysis.

[Namoshek]

It may depend on your broker, but in case of doubt you can simply benchmark your broker using mosquitto_pub and a bash for loop for example:

for i in {0..1000}; do mosquitto_pub -h 'localhost' -p 1883 -t 'my/topic' -m 'My message' -d; done

Good luck!

[garudaonekh]

It took 3 seconds to run this 1000 loop request. I think it's more than enough to handle 50 connections per seconds, right?

[Namoshek]

That is more than enough, and mostly limited due to sequential execution and TCP handshakes. Since requests are executed concurrently, the actual time it takes may be much lower. 👍