Symphony Bdk Datafeed service fails to reconnect in java.net.SocketException
vladokrsymphony opened this issue · 1 comments
vladokrsymphony commented
Bug Report
As observed we found that if the BOT is running and the Agent gets restarted or upgraded the BOT will then be not able to reconnect successfully to the data feed in case of java.net.SocketException
Expected Result:
Once the Agent is back the BOT should reconnect automatically to the Data Feed.
Actual Result:
Bot is not reconnecting to the data feed and it continues to retry and retry. At the end customers need to restart the BOT so that it reconnects successfully.
Environment:
Java BDK 2.9.0
Additional Context:
2022-08-24T09:59:37,892Z INFO [SymphonyBdk_Datafeed-77] com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery : YQs7gf - Retry in 2.0s...
2022-08-24T09:59:39,980Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:39,980Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:40,065Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:40,065Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:40,148Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:40,148Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:40,284Z INFO [SymphonyBdk_Datafeed-77] com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery : YQs7gf - Retry in 4.0s...
2022-08-24T09:59:44,292Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:44,293Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:44,299Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:44,299Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:44,306Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:44,306Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:44,313Z INFO [SymphonyBdk_Datafeed-77] com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery : YQs7gf - Retry in 8.0s...
2022-08-24T09:59:52,321Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:52,321Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:52,330Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:52,330Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:52,337Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - I/O exception (java.net.SocketException) caught when processing request to {s}->https://some-pod.com:443: Connection reset
2022-08-24T09:59:52,337Z INFO [SymphonyBdk_Datafeed-77] org.apache.http.impl.execchain.RetryExec : YQs7gf - Retrying request to {s}->https://some-pod.com:443
2022-08-24T09:59:52,345Z INFO [SymphonyBdk_Datafeed-77] com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery : YQs7gf - Retry in 16.0s...
ystxn commented
Extensive investigations were performed by @yinan-symphony and myself that were inconclusive in determining a root cause of this issue. It would appear to be an edge case that happens only in very specific network environments. Closing for now and we can revisit when more information is available on how this can be prevented.