Reconnection problems when connection goes down
vict2019 opened this issue · 5 comments
vict2019 commented
I've downloaded the latest package and I don't see the Unauthorized access error anymore but it looks like a problem still happens when trying to reconnect. I'm seeing the following error message:
The connection was inactive for more than the allowed 60000 milliseconds and is closed by container...
And then the receivers don't reconnect and eventually get the message that something is busted. Here are more logs:
azure:event-hubs:error [connection-1] Error (context.connection.error) occurred on the amqp connection: c {
azure:event-hubs:error value:
azure:event-hubs:error [ Typed { type: [Object], value: 'amqp:connection:forced' },
azure:event-hubs:error Typed {
azure:event-hubs:error type: [Object],
azure:event-hubs:error value: 'The connection was inactive for more than the allowed 60000 milliseconds and is closed by container \'LinkTracker\'. TrackingId:b8f8f89478fc4cc3beea90f3e89ef207_G5, SystemTracker:gateway5, Timestamp:2019-01-06T10:35:43' },
azure:event-hubs:error Typed { type: [Object], value: null } ] } +45m
azure:event-hubs:error [connection-1] state: { wasConnectionCloseCalled: false,
azure:event-hubs:error numSenders: 0,
azure:event-hubs:error numReceivers: 24 } +2ms
azure:event-hubs:error [connection-1] connection.close() was not called from the sdk and there were some sender or receiver links or both. We should reconnect. +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +301ms
rhea-promise:error [connection-1] The receiver is open ? -> false +45m
azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +2ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '759a93a3-ed16-46f6-b1fd-88b2f560bb2a' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/2'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +0ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +7ms
rhea-promise:error [connection-1] The receiver is open ? -> false +8ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '759a93a3-ed16-46f6-b1fd-88b2f560bb2a' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/2'. +0ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +6ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +1ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +1ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +1ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +1ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +1ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +1ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +0ms
rhea-promise:error [connection-1] The session is open ? -> false +1ms
azure:event-hubs:error [connection-1] Something is busted. State of Receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0' is: { wasCloseInitiated: true,
azure:event-hubs:error receiverError: c { value: [ [Object], [Object], [Object] ] },
azure:event-hubs:error _receiver:
azure:event-hubs:error Receiver {
azure:event-hubs:error domain: null,
azure:event-hubs:error _events:
azure:event-hubs:error { message: [Function],
azure:event-hubs:error receiver_error: [Function],
azure:event-hubs:error receiver_close: [Function] },
azure:event-hubs:error _eventsCount: 3,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error type: 'receiver',
azure:event-hubs:error _session:
azure:event-hubs:error Session {
azure:event-hubs:error domain: null,
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 2,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error _connection: [Object],
azure:event-hubs:error _session: [Object] },
azure:event-hubs:error _link:
azure:event-hubs:error Receiver {
azure:event-hubs:error session: [Object],
azure:event-hubs:error connection: [Object],
azure:event-hubs:error name: 'e102e390-0e1f-478f-9295-a0125c1221d7',
azure:event-hubs:error options: [Object],
azure:event-hubs:error state: [Object],
azure:event-hubs:error issue_flow: false,
azure:event-hubs:error local: [Object],
azure:event-hubs:error remote: [Object],
azure:event-hubs:error delivery_count: 17686,
azure:event-hubs:error credit: 814,
azure:event-hubs:error observers: [Object],
azure:event-hubs:error drain: false,
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 7 },
azure:event-hubs:error linkOptions:
azure:event-hubs:error { name: 'e102e390-0e1f-478f-9295-a0125c1221d7',
azure:event-hubs:error autoaccept: true,
azure:event-hubs:error source: [Object],
azure:event-hubs:error credit_window: 1000,
azure:event-hubs:error onMessage: [Function],
azure:event-hubs:error onError: [Function],
azure:event-hubs:error onClose: [Function],
azure:event-hubs:error onSessionError: [Function],
azure:event-hubs:error onSessionClose: [Function] } } } +14ms
azure:event-hubs:error [connection-1] Something is busted. State of Receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1' is: { wasCloseInitiated: true,
azure:event-hubs:error receiverError: c { value: [ [Object], [Object], [Object] ] },
KyotoFox commented
I am seeing the same issue, no reconnection is attempted when the connection times out.
I am connecting to an event hub using an IoT Hub connection string.
It is very easily reproduced by e.g. disabling your WiFi connection when running locally, which causes the connection to time out.
Here are some logs, starting after I have turned my WiFi connection off (nothing happens until 2 to 3 minutes has passed). After the following log, nothing happens, even when the WiFi connection is re-enabled.
Running node v10.15.0 on macOS 10.14 with @azure/event-hubs@1.0.8
DEBUG=azure*,rhea*
rhea:frames [connection-2]:0 -> empty +2m
rhea:raw [connection-2] SENT: 8 0000000802000000 +2m
rhea:events [connection-2] Connection got event: disconnected +3m
rhea-promise:connection [connection-2] connection got event: 'disconnected'. Re-emitting the translated context. +3m
rhea-promise:translate [connection-2] Translating the context for event: 'disconnected'. +3m
azure:event-hubs:error [connection-2] Error (context.error) occurred on the amqp connection: { Error: read ETIMEDOUT
azure:event-hubs:error at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' } +3m
azure:event-hubs:error [connection-2] state: { wasConnectionCloseCalled: false,
azure:event-hubs:error numSenders: 0,
azure:event-hubs:error numReceivers: 4 } +0ms
azure:event-hubs:error [connection-2] connection.close() was not called from the sdk and there were some sender or receiver links or both. We should reconnect. +1ms
azure:event-hubs:error [connection-2] calling detached on receiver '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/0'. +304ms
rhea-promise:error [connection-2] The receiver is open ? -> false +3m
azure:event-hubs:error [connection-2] calling detached on receiver '8816bda2-91f2-430b-9f04-722d630bfbdb' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/1'. +1ms
rhea-promise:error [connection-2] The receiver is open ? -> false +0ms
azure:event-hubs:error [connection-2] calling detached on receiver '7345a09f-bf55-4a69-9592-951607f47e63' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/2'. +1ms
rhea-promise:error [connection-2] The receiver is open ? -> false +1ms
azure:event-hubs:error [connection-2] calling detached on receiver 'e82a21a8-b6b8-4254-b426-736a79b8db57' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/3'. +0ms
rhea-promise:error [connection-2] The receiver is open ? -> false +0ms
rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +3m
rhea-promise:error [connection-2] The session is open ? -> false +0ms
rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +0ms
rhea-promise:error [connection-2] The session is open ? -> false +0ms
rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +0ms
rhea-promise:error [connection-2] The session is open ? -> false +0ms
rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +0ms
rhea-promise:error [connection-2] The session is open ? -> false +0ms
azure:event-hubs:linkEntity [connection-2] StreamingReceiver '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/0' closed. +3m
azure:event-hubs:linkEntity [connection-2] StreamingReceiver '8816bda2-91f2-430b-9f04-722d630bfbdb' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/1' closed. +0ms
azure:event-hubs:linkEntity [connection-2] StreamingReceiver '7345a09f-bf55-4a69-9592-951607f47e63' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/2' closed. +0ms
azure:event-hubs:linkEntity [connection-2] StreamingReceiver 'e82a21a8-b6b8-4254-b426-736a79b8db57' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/3' closed. +1ms
azure:event-hubs:error [connection-2] Something is busted. State of Receiver '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/0' is: { wasCloseInitiated: true,
azure:event-hubs:error receiverError:
azure:event-hubs:error { Error: read ETIMEDOUT
azure:event-hubs:error at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
azure:event-hubs:error _receiver:
azure:event-hubs:error Receiver {
azure:event-hubs:error _events:
azure:event-hubs:error [Object: null prototype] {
azure:event-hubs:error message: [Function],
azure:event-hubs:error receiver_error: [Function],
azure:event-hubs:error receiver_close: [Function] },
azure:event-hubs:error _eventsCount: 3,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error type: 'receiver',
azure:event-hubs:error _session:
azure:event-hubs:error Session {
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 2,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error _connection: [Connection],
azure:event-hubs:error _session: [Session] },
azure:event-hubs:error _link:
azure:event-hubs:error Receiver {
azure:event-hubs:error session: [Session],
azure:event-hubs:error connection: [Connection],
azure:event-hubs:error name: '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e',
azure:event-hubs:error options: [Object],
azure:event-hubs:error state: [EndpointState],
azure:event-hubs:error issue_flow: false,
azure:event-hubs:error local: [Object],
azure:event-hubs:error remote: [Object],
azure:event-hubs:error delivery_count: 2,
azure:event-hubs:error credit: 998,
azure:event-hubs:error observers: [EventEmitter],
azure:event-hubs:error drain: false,
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 7 },
azure:event-hubs:error linkOptions:
azure:event-hubs:error { name: '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e',
azure:event-hubs:error autoaccept: true,
azure:event-hubs:error source: [Object],
azure:event-hubs:error credit_window: 1000,
azure:event-hubs:error onMessage: [Function],
azure:event-hubs:error onError: [Function],
azure:event-hubs:error onClose: [Function],
azure:event-hubs:error onSessionError: [Function],
azure:event-hubs:error onSessionClose: [Function] } } } +1ms
azure:event-hubs:error [connection-2] Something is busted. State of Receiver '8816bda2-91f2-430b-9f04-722d630bfbdb' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/1' is: { wasCloseInitiated: true,
azure:event-hubs:error receiverError:
azure:event-hubs:error { Error: read ETIMEDOUT
azure:event-hubs:error at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
azure:event-hubs:error _receiver:
azure:event-hubs:error Receiver {
azure:event-hubs:error _events:
azure:event-hubs:error [Object: null prototype] {
azure:event-hubs:error message: [Function],
azure:event-hubs:error receiver_error: [Function],
azure:event-hubs:error receiver_close: [Function] },
azure:event-hubs:error _eventsCount: 3,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error type: 'receiver',
azure:event-hubs:error _session:
azure:event-hubs:error Session {
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 2,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error _connection: [Connection],
azure:event-hubs:error _session: [Session] },
azure:event-hubs:error _link:
azure:event-hubs:error Receiver {
azure:event-hubs:error session: [Session],
azure:event-hubs:error connection: [Connection],
azure:event-hubs:error name: '8816bda2-91f2-430b-9f04-722d630bfbdb',
azure:event-hubs:error options: [Object],
azure:event-hubs:error state: [EndpointState],
azure:event-hubs:error issue_flow: false,
azure:event-hubs:error local: [Object],
azure:event-hubs:error remote: [Object],
azure:event-hubs:error delivery_count: 8,
azure:event-hubs:error credit: 992,
azure:event-hubs:error observers: [EventEmitter],
azure:event-hubs:error drain: false,
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 7 },
azure:event-hubs:error linkOptions:
azure:event-hubs:error { name: '8816bda2-91f2-430b-9f04-722d630bfbdb',
azure:event-hubs:error autoaccept: true,
azure:event-hubs:error source: [Object],
azure:event-hubs:error credit_window: 1000,
azure:event-hubs:error onMessage: [Function],
azure:event-hubs:error onError: [Function],
azure:event-hubs:error onClose: [Function],
azure:event-hubs:error onSessionError: [Function],
azure:event-hubs:error onSessionClose: [Function] } } } +1ms
azure:event-hubs:error [connection-2] Something is busted. State of Receiver '7345a09f-bf55-4a69-9592-951607f47e63' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/2' is: { wasCloseInitiated: true,
azure:event-hubs:error receiverError:
azure:event-hubs:error { Error: read ETIMEDOUT
azure:event-hubs:error at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
azure:event-hubs:error _receiver:
azure:event-hubs:error Receiver {
azure:event-hubs:error _events:
azure:event-hubs:error [Object: null prototype] {
azure:event-hubs:error message: [Function],
azure:event-hubs:error receiver_error: [Function],
azure:event-hubs:error receiver_close: [Function] },
azure:event-hubs:error _eventsCount: 3,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error type: 'receiver',
azure:event-hubs:error _session:
azure:event-hubs:error Session {
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 2,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error _connection: [Connection],
azure:event-hubs:error _session: [Session] },
azure:event-hubs:error _link:
azure:event-hubs:error Receiver {
azure:event-hubs:error session: [Session],
azure:event-hubs:error connection: [Connection],
azure:event-hubs:error name: '7345a09f-bf55-4a69-9592-951607f47e63',
azure:event-hubs:error options: [Object],
azure:event-hubs:error state: [EndpointState],
azure:event-hubs:error issue_flow: false,
azure:event-hubs:error local: [Object],
azure:event-hubs:error remote: [Object],
azure:event-hubs:error delivery_count: 0,
azure:event-hubs:error credit: 1000,
azure:event-hubs:error observers: [EventEmitter],
azure:event-hubs:error drain: false,
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 7 },
azure:event-hubs:error linkOptions:
azure:event-hubs:error { name: '7345a09f-bf55-4a69-9592-951607f47e63',
azure:event-hubs:error autoaccept: true,
azure:event-hubs:error source: [Object],
azure:event-hubs:error credit_window: 1000,
azure:event-hubs:error onMessage: [Function],
azure:event-hubs:error onError: [Function],
azure:event-hubs:error onClose: [Function],
azure:event-hubs:error onSessionError: [Function],
azure:event-hubs:error onSessionClose: [Function] } } } +1ms
azure:event-hubs:error [connection-2] Something is busted. State of Receiver 'e82a21a8-b6b8-4254-b426-736a79b8db57' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/3' is: { wasCloseInitiated: true,
azure:event-hubs:error receiverError:
azure:event-hubs:error { Error: read ETIMEDOUT
azure:event-hubs:error at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
azure:event-hubs:error _receiver:
azure:event-hubs:error Receiver {
azure:event-hubs:error _events:
azure:event-hubs:error [Object: null prototype] {
azure:event-hubs:error message: [Function],
azure:event-hubs:error receiver_error: [Function],
azure:event-hubs:error receiver_close: [Function] },
azure:event-hubs:error _eventsCount: 3,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error type: 'receiver',
azure:event-hubs:error _session:
azure:event-hubs:error Session {
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 2,
azure:event-hubs:error _maxListeners: undefined,
azure:event-hubs:error actionInitiated: 0,
azure:event-hubs:error _connection: [Connection],
azure:event-hubs:error _session: [Session] },
azure:event-hubs:error _link:
azure:event-hubs:error Receiver {
azure:event-hubs:error session: [Session],
azure:event-hubs:error connection: [Connection],
azure:event-hubs:error name: 'e82a21a8-b6b8-4254-b426-736a79b8db57',
azure:event-hubs:error options: [Object],
azure:event-hubs:error state: [EndpointState],
azure:event-hubs:error issue_flow: false,
azure:event-hubs:error local: [Object],
azure:event-hubs:error remote: [Object],
azure:event-hubs:error delivery_count: 4,
azure:event-hubs:error credit: 996,
azure:event-hubs:error observers: [EventEmitter],
azure:event-hubs:error drain: false,
azure:event-hubs:error _events: [Object],
azure:event-hubs:error _eventsCount: 7 },
azure:event-hubs:error linkOptions:
azure:event-hubs:error { name: 'e82a21a8-b6b8-4254-b426-736a79b8db57',
azure:event-hubs:error autoaccept: true,
azure:event-hubs:error source: [Object],
azure:event-hubs:error credit_window: 1000,
azure:event-hubs:error onMessage: [Function],
azure:event-hubs:error onError: [Function],
azure:event-hubs:error onClose: [Function],
azure:event-hubs:error onSessionError: [Function],
azure:event-hubs:error onSessionClose: [Function] } } } +1ms
KyotoFox commented
Is anything being done to address this issue?
ramya-rao-a commented
ramya-rao-a commented
@vict2019 and @KyotoFox Just giving an update here that we have found the root cause as why we are not retrying the connection and resuming the receiver
- Host not found is a non retryable error. Unfortunately Nodejs gives the same
ENOTFOUNDerror for both host not found and the network being down scenarios. Therefore, we were treating the network being down as a non retryable error as well, and so recevier didnt get resumed. - rhea is the library we use to create the AMQP connection and AMQP links for senders/receivers. The indicator for suggesting if the link was explicitly closed by us or due to external factors had a bug. This is what caused the logs you see for
Something is busted
The second issue has been now fixed in rhea. We are in the process of fixing the first issue.
We are targeting sometime next week to publish an update with the fixes.
Thanks for your patience
ramya-rao-a commented
@vict2019, @KyotoFox Thanks for your patience
We have just released a new version 2.0.0 for the Event Hubs library which includes the fix for the connection related problems being discussed in this issue.
We will be re-connecting and resume receiving of messages when the network goes down and then comes back up
Please keep in mind that there is a major version update in this release. This is because of a few breaking changes for which you may have to react to and make some changes in your code. Please see the changelog for the list of breaking changes and bug fixes.