Restart loop detected - adapter crash while internet connection is down
Thomas-Doc opened this issue · 7 comments
ioBroker/ioBroker.js-controller#1217 (comment)
I have still the same issue:
Question: It is really good feature to detect adapter restart loops and stop restarting after 3 crashes. But can we also implement maybe some additional delay after which js-controller will make yet another try to start adapter?
E.g. my system uses mobile internet and it sometimes happens that internet is down for 15-30 minutes. During this time some adapters (like Telegram) crashes while attempting to connect to the server and js-controller stops them..
Really it is not what I'm expecting from home control system. e.g. Telegram stopped by system makes me think that everything is Ok as I'm not receiving any messages from IoBroker..
I understand that adapters should be updated to try-catch such crashes, but do you think it is really possible for all the adapters to be updated to catch all the possible crashes that may happen?
Response: "In fact yes the adapter would need to take care here ... can you please open another issue (or update existing) on relevant adapters so that we can try to fix there"
Basically you are right with your expected response ... in case of Telegram we need more logs because we did several tries already to catch all such cases and feedback so far was that all cases where handled ... so provide logs so that we can fix it :-)
I've exactly the same Issue. The thing is that I turn off the Internet/WLAN in the night when I don't need it.
2021-09-29 05:04:32.757 - �[32minfo�[39m: telegram.0 (18651) starting. Version 1.10.0 in /opt/iobroker/node_modules/iobroker.telegram, node: v16.8.0, js-controller: 3.3.18
2021-09-29 05:04:43.087 - �[31merror�[39m: telegram.0 (18651) Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().
2021-09-29 05:04:43.088 - �[31merror�[39m: telegram.0 (18651) unhandled promise rejection: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org
2021-09-29 05:04:43.090 - �[31merror�[39m: telegram.0 (18651) RequestError: Error: getaddrinfo EAI_AGAIN api.telegram.org
at new RequestError (/opt/iobroker/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/opt/iobroker/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/opt/iobroker/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/opt/iobroker/node_modules/request/request.js:185:22)
at Request.emit (node:events:394:28)
at Request.onRequestError (/opt/iobroker/node_modules/request/request.js:877:8)
at ClientRequest.emit (node:events:394:28)
at TLSSocket.socketErrorListener (node:_http_client:447:9)
at TLSSocket.emit (node:events:394:28)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
2021-09-29 05:04:43.091 - �[31merror�[39m: telegram.0 (18651) Exception-Code: EFATAL: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org
2021-09-29 05:04:43.600 - �[32minfo�[39m: telegram.0 (18651) terminating
2021-09-29 05:04:43.603 - �[33mwarn�[39m: telegram.0 (18651) Terminated (UNCAUGHT_EXCEPTION): Without reason
2021-09-29 05:04:53.592 - �[31merror�[39m: host.raspberrypi instance system.adapter.telegram.0 terminated with code 6 (UNCAUGHT_EXCEPTION)
2021-09-29 05:04:53.593 - �[32minfo�[39m: host.raspberrypi Restart adapter system.adapter.telegram.0 because enabled
2021-09-29 05:04:54.658 - �[32minfo�[39m: host.raspberrypi instance system.adapter.telegram.0 started with pid 18711
2021-09-29 05:04:57.067 - �[32minfo�[39m: telegram.0 (18711) starting. Version 1.10.0 in /opt/iobroker/node_modules/iobroker.telegram, node: v16.8.0, js-controller: 3.3.18
2021-09-29 05:05:07.403 - �[31merror�[39m: telegram.0 (18711) Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().
2021-09-29 05:05:07.405 - �[31merror�[39m: telegram.0 (18711) unhandled promise rejection: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org
2021-09-29 05:05:07.406 - �[31merror�[39m: telegram.0 (18711) RequestError: Error: getaddrinfo EAI_AGAIN api.telegram.org
at new RequestError (/opt/iobroker/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/opt/iobroker/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/opt/iobroker/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/opt/iobroker/node_modules/request/request.js:185:22)
at Request.emit (node:events:394:28)
at Request.onRequestError (/opt/iobroker/node_modules/request/request.js:877:8)
at ClientRequest.emit (node:events:394:28)
at TLSSocket.socketErrorListener (node:_http_client:447:9)
at TLSSocket.emit (node:events:394:28)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
2021-09-29 05:05:07.407 - �[31merror�[39m: telegram.0 (18711) Exception-Code: EFATAL: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org
2021-09-29 05:05:07.915 - �[32minfo�[39m: telegram.0 (18711) terminating
2021-09-29 05:05:07.918 - �[33mwarn�[39m: telegram.0 (18711) Terminated (UNCAUGHT_EXCEPTION): Without reason
2021-09-29 05:05:17.925 - �[31merror�[39m: host.raspberrypi instance system.adapter.telegram.0 terminated with code 6 (UNCAUGHT_EXCEPTION)
2021-09-29 05:05:17.926 - �[32minfo�[39m: host.raspberrypi Restart adapter system.adapter.telegram.0 because enabled
2021-09-29 05:05:17.926 - �[33mwarn�[39m: host.raspberrypi Do not restart adapter system.adapter.telegram.0 because restart loop detected
Entschuldige bitte die späte Rückmeldung.
Der Fehler lässt sich am schnellsten reproduzieren, wenn man die Internet-Verbindung trennt (Kabel zieht) und dann den telegram-Adapter bei fehlender Internet-Konnektivität neustartet. Der Adapter versucht dann mehrfach zu starten, bis er geblockt wird ("restart loop detected"). Tritt aber auch unter anderen Umständen bei gestörtem Internet auf. Der Adapter sollte möglichst wieder sauber laufen, wenn die Internet-Verbindung wiederhergestellt ist. Diese "restart loop detected" ist in diesem Fall sehr störend. War vorher besser.
telegram.0 | 2021-10-01 23:48:02.595 | warn | Terminated (UNCAUGHT_EXCEPTION): Without reason
telegram.0 | 2021-10-01 23:48:02.593 | info | terminating
telegram.0 | 2021-10-01 23:48:02.087 | error | Exception-Code: EFATAL: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org
telegram.0 | 2021-10-01 23:48:02.087 | error | RequestError: Error: getaddrinfo EAI_AGAIN api.telegram.org
at new RequestError (/opt/iobroker/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/opt/iobroker/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/opt/iobroker/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/opt/iobroker/node_modules/request/request.js:185:22)
at Request.emit (events.js:314:20)
at Request.onRequestError (/opt/iobroker/node_modules/request/request.js:877:8)
at ClientRequest.emit (events.js:314:20)
at TLSSocket.socketErrorListener (_http_client.js:427:9)
at TLSSocket.emit (events.js:314:20)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
telegram.0 | 2021-10-01 23:48:02.086 | error | unhandled promise rejection: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org
telegram.0 | 2021-10-01 23:48:02.085 | error | Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within the next 7 days. Please check if the issue is still relevant in the most current version of the adapter and tell us. Also check that all relevant details, logs and reproduction steps are included and update them if needed. Thank you for your contributions.
Dieses Problem wurde automatisch als veraltet markiert, da es in letzter Zeit keine Aktivitäten gab. Es wird geschlossen, wenn nicht innerhalb der nächsten 7 Tage weitere Aktivitäten stattfinden. Bitte überprüft, ob das Problem auch in der aktuellsten Version des Adapters noch relevant ist, und teilt uns dies mit. Überprüft auch, ob alle relevanten Details, Logs und Reproduktionsschritte enthalten sind bzw. aktualisiert diese. Vielen Dank für Eure Unterstützung.
In 1.11.0 the telegram lib was updated. Please retest the behaviour.
v.1.11.1
Confirm, keeps crashing
normal start:
20:21:09.588 debug Schedule restart: 4 5 * * *
20:21:09.652 debug Start polling with: 300(number) ms interval
20:21:09.774 debug getMe: {"id":###,"is_bot":true,"first_name":"###","username":"###","can_join_groups":true,"can_read_all_group_messages":false,"supports_inline_queries":false}
20:21:09.775 debug Send message to "###": Started!
error when trying to start without internet connection
20:22:55.694 debug Schedule restart: 4 5 * * *
20:22:55.751 debug Start polling with: 300(number) ms interval
20:22:55.776 error getMe Error:Error: EFATAL: Error: getaddrinfo ENOTFOUND api.telegram.org
20:22:55.779 error Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().
20:22:55.779 error unhandled promise rejection: EFATAL: Error: getaddrinfo ENOTFOUND api.telegram.org
20:22:55.779 error RequestError: Error: getaddrinfo ENOTFOUND api.telegram.org at new RequestError (C:\iobroker\node_modules\request-promise-core\lib\errors.js:14:15) at Request.plumbing.callback (C:\iobroker\node_modules\request-promise-core\lib\plumbing.js:87:29) at Request.RP$callback [as _callback] (C:\iobroker\node_modules\request-promise-core\lib\plumbing.js:46:31) at self.callback (C:\iobroker\node_modules\request\request.js:185:22) at Request.emit (events.js:400:28) at Request.onRequestError (C:\iobroker\node_modules\request\request.js:877:8) at ClientRequest.emit (events.js:400:28) at TLSSocket.socketErrorListener (_http_client.js:475:9) at TLSSocket.emit (events.js:400:28) at emitErrorNT (internal/streams/destroy.js:106:8) at emitErrorCloseNT (internal/streams/destroy.js:74:3) at processTicksAndRejections (internal/process/task_queues.js:82:21)
20:22:55.779 error Exception-Code: EFATAL: EFATAL: Error: getaddrinfo ENOTFOUND api.telegram.org
20:22:55.781 error getMe (reconnect #0) Error:Error: EFATAL: Error: getaddrinfo ENOTFOUND api.telegram.org
20:22:55.783 debug Send message to "###": Restarting...
Can someone try to reproduce it with the GitHub version??