Event hub freezes and is not responding after a long run.

[tikyau]

Hi there, the Event hub freezes and not responding after a long run. Could anyone advise how we could eliminate this timeout issues?

{ Error: The link 'G10:5173588:/shioeventhub/Partitions/0_e1af6238-00e5-40f5-b822-7206730cdbeb' is force detached by the broker due to errors occurred in publisher(link7376012). Detach origin: AmqpEventHubPublisher.IdleTimerExpired: Idle timeout: 00:30:00. TrackingId:51cec16f0000539e00708c8c5a600968_G10_B25, SystemTracker:shioeventhub:eventhub:shioeventhub16383, Timestamp:1/18/2018 3:20:02 PM
at Object.translateError [as translate] (/opt/project-tcp-redirect-to-eventhub/node_modules/azure-event-hubs/lib/errors.js:37:15)
at SenderLink.onErrorReceived (/opt/project-tcp-redirect-to-eventhub/node_modules/azure-event-hubs/lib/sender.js:27:39)
at emitOne (events.js:121:20)
at SenderLink.emit (events.js:211:7)
at SenderLink.Link._detached (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/link.js:221:10)
at SenderLink._detached (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/sender_link.js:236:41)
at SenderLink.Link._detachReceived (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/link.js:204:8)
at Session._processDetachFrame (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/session.js:335:45)
at Session._processFrame (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/session.js:302:61)
at Connection._processFrameEH (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/session.js:186:49)
at emitOne (events.js:121:20)
at Connection.emit (events.js:211:7)
at Connection._receiveAny (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/connection.js:426:12)
at Connection._receiveData (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/connection.js:357:8)
at TlsTransport. (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/connection.js:515:38)
at emitOne (events.js:116:13)
transport:
{ AmqpProtocolError: amqp:link:detach-forced:The link 'G10:5173588:/shioeventhub/Partitions/0_e1af6238-00e5-40f5-b822-7206730cdbeb' is force detached by the broker due to errors occurred in publisher(link7376012). Detach origin: AmqpEventHubPublisher.IdleTimerExpired: Idle timeout: 00:30:00. TrackingId:51cec16f0000539e00708c8c5a600968_G10_B25, SystemTracker:shioeventhub:eventhub:shioeventhub16383, Timestamp:1/18/2018 3:20:02 PM
at Object.errors.wrapProtocolError (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/errors.js:10:12)
at SenderLink.Link._detached (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/link.js:221:42)
at SenderLink._detached (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/sender_link.js:236:41)
at SenderLink.Link._detachReceived (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/link.js:204:8)
at Session._processDetachFrame (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/session.js:335:45)
at Session._processFrame (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/session.js:302:61)
at Connection._processFrameEH (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/session.js:186:49)
at emitOne (events.js:121:20)
at Connection.emit (events.js:211:7)
at Connection._receiveAny (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/connection.js:426:12)
at Connection._receiveData (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/connection.js:357:8)
at TlsTransport. (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/connection.js:515:38)
at emitOne (events.js:116:13)
at TlsTransport.emit (events.js:211:7)
at TLSSocket. (/opt/project-tcp-redirect-to-eventhub/node_modules/amqp10/lib/transport/tls_transport.js:29:49)
at emitOne (events.js:116:13)
name: 'AmqpProtocolError',
message: 'amqp:link:detach-forced:The link 'G10:5173588:/shioeventhub/Partitions/0_e1af6238-00e5-40f5-b822-7206730cdbeb' is force detached by the broker due to errors occurred in publisher(link7376012). Detach origin: AmqpEventHubPublisher.IdleTimerExpired: Idle timeout: 00:30:00. TrackingId:51cec16f0000539e00708c8c5a600968_G10_B25, SystemTracker:shioeventhub:eventhub:shioeventhub16383, Timestamp:1/18/2018 3:20:02 PM',
condition: 'amqp:link:detach-forced',
description: 'The link 'G10:5173588:/shioeventhub/Partitions/0_e1af6238-00e5-40f5-b822-7206730cdbeb' is force detached by the broker due to errors occurred in publisher(link7376012). Detach origin: AmqpEventHubPublisher.IdleTimerExpired: Idle timeout: 00:30:00. TrackingId:51cec16f0000539e00708c8c5a600968_G10_B25, SystemTracker:shioeventhub:eventhub:shioeventhub16383, Timestamp:1/18/2018 3:20:02 PM',
errorInfo: {} } }

[michaelinipop]

I am seeing similar behavior!!

In an implementation using this library I have two micro-services receiving messages.

With one service - I am receiving messages at 10 - 15 minute periods.
With another server - I am receiving messages every 10-30 seconds.

In the case that I am receiving messages every 10-30 seconds , I am not able to see this library freeze.

HOWEVER with the micro service that is operating at 10 -15 minutes I am seeing that library get suck into a frozen state.

Take note I am using the latest version of this library
I have been seeing this issue for the past three months. Every time I restarted the services everything would go back to normal operation.

@tikyau
The quick fix for me is the restart the connection every 30 minutes. I have been running with this strategy for three days and my problems have been resolved.

Any chance the issue could be from how the library is handling session timeout and auto re-authentication ?

[amarzavery]

@tikyau @michaelinipop - We have a new preview version "0.1.0" of the event hubs client. This client was written from scratch all up. Please try the new version and let us know if you still see issues. You can find examples over here.

[michaelinipop]

@amarzavery I will give it a look!! LOVE THE async and await approach!!

Thanks.