Best way to reconnect?
JSteunou opened this issue · 28 comments
Hi!
I was wondering what should be the best way to reconnect after losing connection. I'm facing a case where an HTML5 app lose the connection when the mobile is going on hold.
I though calling a connect again in a connection error callback could work, but it does not. Freeze on "Opening Web Socket..."
I managed it by calling for a new connection in connection error callback. But I'm losing all my subscription so I have to store those in order to subscribe again. Not great.
I'm thinking about another thing: is heartbeat could prevent losing connection?
heartbeats are designed to keep a connection open yes.
Just re-connecting should be fine. Any idea why its hanging? What does your broker say?
No reason, no clue, no log :( I'm using stompjs + sockjs + RabbitMQ.
After testing, heartbeats does not work for my case. On iOS, on hold, the system shut down network connection and others things. I still lose the connection to my broker.
what do you mean by "On iOS, on hold"?
On iOS, the network connection will be closed when the application goes into background.
So is the client object being garbage collected?
@jmesnil iPhone 4s, iOS 6 & 7, when the mobile is on sleep / hibernate (maybe a better word than on hold).
The user use safari or chrome, the application is an HTML application. After a time of inactivity or when pressing the on/off button the mobile goes to sleep / hibernate.
When the user wake up the mobile, the browser is still opened on the application, and the application works well, but the stomp client is disconnected. If I call connect in the error callback it's stuck on "Opening Web Socket..."
@rfox90 I dont think objects are garbage collected. Mobile put on hold everything, even running JavaScript in browser when going to sleep / hibernate but usually all things restart smoothly on wake up.
have you solved this?
Yes, I manually store all topics I subscribed to and when I catch a socket error I re-create a connection and re-subscribe to all my topics.
My implementation, try to reconnect every 1 second after disconnected.
let ws = new WebSocket(socketUrl);
let client = webstomp.over(ws);
client.connect({}, (frame) => {
successCallback();
}, () => {
reconnect(socketUrl, successCallback);
});
function reconnect(socketUrl, successCallback) {
let connected = false;
let reconInv = setInterval(() => {
ws = new WebSocket(socketUrl);
client = webstomp.over(ws);
client.connect({}, (frame) => {
clearInterval(reconInv);
connected = true;
successCallback();
}, () => {
if (connected) {
reconnect(socketUrl, successCallback);
}
});
}, 1000);
}
How you are making sure that you are not losing any messages when you "re-create a connection and re-subscribe to all my topics."
my take on this problem https://bushbaby.nl/solving-websockets-subscriptions-and-disconnects/
@bfwg thanx for your solution, it works like a charm.
However I have a question, in reconnect function, in error callback I don't understand why this is if (connected) and not if (!connected), I have tried with if (!connected) but the socket keeps reconnecting even if it is already connected.
Hey @mouss-delta , the connected is the flag to prevent a recursive hell. The logic is: we only want to call the reconnect function when there is a successful connection.
@mouss-delta you are welcome. The code I'm using in my code base currently looks like the below, as you can see the logic is much cleaner.
let ws = new WebSocket(socketUrl);
let client = webstomp.over(ws);
connectAndReconnect(socketInfo, successCallback);
private connectAndReconnect(socketInfo, successCallback) {
ws = new WebSocket(socketUrl);
client = webstomp.over(ws);
client.connect({}, (frame) => {
successCallback();
}, () => {
setTimeout(() => {
this.connectAndReconnect(socketInfo, successCallback);
}, 5000);
});
}@bfwg Can you explain the use for the socketInfo parameter to connectAndReconnect? I see it being passed in, but it's never used.
@brendanbenson it's just the ws URL.
Hi, @bfwg avec you had any issue with socket reconnexion on Chrome. The given algorithm works just fine with firefox but non in Chrome, it is like the error callback of client.connect is not even taken into account
Hey, @mouss-delta no the algorithm works fine for me on Chrome.
File: node_modules/@stomp/stompjs/lib/stomp.js
Line: 352
There is a method name is onclose for websocket. In line 352 it's handling this. Real method is:
this.ws.onclose = (function(_this) {
return function() {
var msg;
msg = "Whoops! Lost connection to " + _this.ws.url;
if (typeof _this.debug === "function") {
_this.debug(msg);
}
_this._cleanUp();
if (typeof errorCallback === "function") {
errorCallback(msg);
}
return _this._schedule_reconnect();
};
})(this);
We can access this from our app. I'm writing here my usage:
connect() {
var self = this;
//self.socket = new SockJS(self.getConnectionUrl());
var token = window.myApp.getCookie("token");
self.socket = new SockJS(self.getChatWebsocketConnectionUrl());
self.socket.onheartbeat = function() {
console.log("heartbeat");
};
self.stompClient = Stomp.over(self.socket);
self.stompClient.connect(self.getDefaultHeaders(), function(frame) {
self.setConnected(true);
console.log("Connected: " + frame);
});
// here is important thing. We must get current onclose function for call it later.
var stompClientOnclose = self.stompClient.ws.onclose;
self.stompClient.ws.onclose = function() {
console.log("Websocket connection closed and handled from our app.");
self.setConnected(false);
stompClientOnclose();
};
}
If websocket server down or a network problem occured than we can handle close event.
let ws = new WebSocket(socketUrl);
let client = webstomp.over(ws);
connectAndReconnect(socketInfo, successCallback);
private connectAndReconnect(socketInfo, successCallback) {
ws = new WebSocket(socketUrl);
client = webstomp.over(ws);
client.connect({}, (frame) => {
successCallback();
}, () => {
setTimeout(() => {
this.connectAndReconnect(socketInfo, successCallback);
}, 5000);
});
}
when my code is like this. I shutdown the server process，the errorcallback is called twice，what's the problem？？@bfwg
@super86 The problem with this function is that, it is possible the Websocket's CONNECTING phase might last more than 1 second. In this case another setInterval will be called/initialized, and if the process repeats it might come to multiple setIntervals instead of one.
What I did was the adding two checks inside setInterval ( in your case inside connectAndReconnect), so it looks like like this:
var reconInt = setInterval(function() {
if (client.ws.readyState === client.ws.CONNECTING) {
return;
}
if (client.ws.readyState === client.ws.OPEN) {
clearInterval(reconInt);
return;
}
var ws = new WebSocket(config.stomp.url);
client = new Stomp.over(ws);
client.heartbeat.outgoing = 30000;
client.heartbeat.incoming = 30000;
client.debug = null;
client.connect(
credentials,
() => {
clearInterval(reconInt);
connected = true;
parent.connectCallback(callbackParams, client, accountUid);
},
() => {
if (connected) {
parent.errorCallback(
callbackParams,
config,
client,
credentials,
accountUid,
parent
);
}
}
);
}, 1000);
}
Notice the following part at the top:
if (client.ws.readyState === client.ws.CONNECTING) {
return;
}
if (client.ws.readyState === client.ws.OPEN) {
clearInterval(reconInt);
return;
}
Here in the first part I am checking whether the previous setInterval client status in in CONNECTING phase state, which is possible if the client.connect lasts more than 1 second. In this case it is still possible that the CONNECTING phase fails, hence I am not removing the interval but having a second check.
Here I am checking if the state is open, meaning some previous interval already opened a connection, then clear the interval and exit from the current one.
You can also check the code here: https://github.com/inplayer-org/inplayer.js/blob/master/src/Socket/index.js
How do you guys/girls handle the fact that if you have a lot of concurrent users and Stomp fails at some point, every single one of them will try to reconnect at the same time and the server will probably spam and block the servers?
We had just 6k concurrent users, and are using Stomp over WebSocket + RabbitMQ. However during an event, there might've been some interruption and the WS connection failed for all of them. Afterwards all the users flooded our servers with the 1 second callback to reconnect(we had like a lot of requests pending and stacked), so we had to restart everything.
One solution I can think of is to have something like a Fibonacci sequence for the setInterval(), time call, e.g. -> setTimeout(). But still, the initial 1-5 seconds a lot of requests will come at once. Plus the fact that I would have to have an infinite loop blocking everything, util it connects...
Anyone has any better idea to optimize this?
@GoranGjorgievski Basically you need to add a random amount of starting delay + a backoff delay.
For instance, you randomize the initial wait time somewhere between 5-30 seconds, after which you will do exponential backoff (or fib, if you please). Make sure to test your server behavior in a heavy load case to make sure it will eventually recover.
Even better (and related to the solution I'm looking for) the server should be able to close connections with a certain close code, and then the client can reconnect accordingly (server can tell the client how long to wait, for instance).
@milosonator Yes thanks! Thanks what I actually did, having a random starting setTimeout + something like incremental 'Fibonacci like' timed callback: https://github.com/inplayer-org/inplayer.js/blob/master/src/Socket/index.js
errorCallback(
...,
timeoutStart = 0
) {
...
if (timeoutStart === 0) {
timeoutStart =
(Math.floor(Math.random() * MAX_INITIAL_INTERVAL) + 1) * 1000; //get a random start timeout between 1-max
}
setTimeout(function() {
if (
client.ws.readyState === client.ws.CONNECTING ||
client.ws.readyState === client.ws.OPEN
) {
return;
}
var ws = new WebSocket(config.stomp.url);
client = new Stomp.over(ws);
...
client.connect(
credentials,
() => {
parent.connectCallback(callbackParams, client, accountUid);
//reset the timeoutStart
timeoutStart =
(Math.floor(Math.random() * MAX_INITIAL_INTERVAL) + 1) *
1000; //get a random start timeout between 1-max
},
() => {
parent.errorCallback(
...
timeoutStart
);
}
);
}, timeoutStart);
if (timeoutStart >= MAX_INTERVAL) {
//if more than 10 minutes reset the timer
timeoutStart =
(Math.floor(Math.random() * MAX_INITIAL_INTERVAL) + 1) * 1000; //get a random start timeout between 1-max
} else {
timeoutStart += Math.ceil(timeoutStart / 2);
}
}
In addition, I also added a TOP limit for the timeout for 10 minutes.
e.g. If the timeout interval reaches 600000 milliseconds, it should reset back again to the random initial number. This way I am avoiding having some unreasonable high re-connection times for the client.
Still in testing phases and thinking about somehow getting the number of active socket connections on OPEN, and then when it fails just scale the random MAX_INITIAL_INTERVAL number depending on that number of connections. Once I manage to improve it, I will make sure to post it here of course.
You can do it with ↓
var sock = new SockJS('/my_prefix');
sock.onheartbeat = function() {
console.log('heartbeat');
};
https://github.com/sockjs/sockjs-protocol/wiki/Heartbeats-and-SockJS
After long search, I was able to achieve this properly with this code
var socket = new SockJS(this.websocketUrl);
var stompClient = Stomp.over(this.socket);
var isconnected = false;
initNetworkEvents();
connectToWebsocket();
initNetworkEvents() {
const that = this;
window.addEventListener('offline', function () {
that.triggerReconnect('offline')
});
window.addEventListener('online', function () {
that.triggerReconnect('online')
});
}
triggerReconnect(eventName: string) {
const that = this;
console.log(eventName, "event received")
// This will trigger the "onWebSocketClose" method in stompClient
that.socket.close();
}
connectToWebsocket() {
var that = this;
console.log('Trying to connect to Websocket Server...');
// Check the list of methods available in this stompClient object
console.log('StompClient - ', stompClient);
// Switch off debug
stompClient.debug = () => { };
// Error handlers. There are many other methods also
that.stompClient.onDisconnect = () => { that.reConnectWebSocket() }
that.stompClient.onStompError = () => { that.reConnectWebSocket() }
// This is the most important
that.stompClient.onWebSocketClose = () => { that.reConnectWebSocket() }
that.stompClient.connect({}, function () {
console.log('Websocket connection established...');
that.isconnected = true;
// Start all the subscriptions
}, function (error) {
console.log("Web socket error", error);
that.reConnectWebSocket();
});
}
reConnectWebSocket() {
var that = this;
const retryTimeout = 2;
that.isconnected = false;
console.log('Re-connecting websocket after', retryTimeout * 1000, 'secs...')
// Call the websocket connect method
setTimeout(function(){ that.connectToWebsocket(); }, retryTimeout * 1000);
}
The above method will automatically re-connect for the following cases:-
- When browser goes offline
- When browser comes online
- When exiting connection gets disconnected due to a problem
you can reference websocket-auto-reconnect
this js has already deal with reconnection
simply use as
const ws = new ReconnetWebsocket(
'/yourUrl',
)
ws.connect(
stompClient => {
console.log('ws 连接成功')
stompClient.subscribe(
'/yourChannel',
res => {
this.processData(res)
},
)
},
e => {
console.log('ws 连接失败', e)
}
}
ws.onReconnect = () => {
console.log('reconnected')
}