StreamClosedByServerException: Stream closed by server
Depi1993 opened this issue · 13 comments
Launch Darkly Configuration:
implementation group: 'com.launchdarkly', name: 'launchdarkly-java-server-sdk', version: '6.0.4'
With Config:
new LDConfig.Builder() .http(Components.httpConfiguration() .connectTimeout(Duration.ofSeconds(LD_CONNECT_TIMEOUT_SECONDS)) .socketTimeout(Duration.ofSeconds(LD_SOCKET_TIMEOUT_SECONDS))) .events(Components.sendEvents().flushInterval(Duration.ofSeconds(FLUSH_INTERVAL))) .build();
new LDClient(credentials.get(ApiConstants.LD_CLIENT_CREDS_KEY), ldConfig);
After a while of running, started getting - [Multiple Times]
com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571) at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:188)
Error in stream connection (will retry): com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
|
Hello @Depi1993,
Would you be able to create a support ticket through https://support.launchdarkly.com ? We likely need to ask you for more information to debug this behavior.
We are facing the same issue with java-sdk
Launch Darkly Configuration:
implementation("com.launchdarkly:launchdarkly-java-server-sdk:6.1.0")
ERR log
20:39:22.028 [LaunchDarkly-streaming] INFO com.launchdarkly.sdk.server.LDClient.DataSource - Waiting 852 milliseconds before reconnecting
20:39:22.028 [LaunchDarkly-streaming] WARN com.launchdarkly.sdk.server.LDClient.DataSource - Error in stream connection (will retry): com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
at java.lang.Thread.run(Unknown Source) ~[?:?]
at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:199) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272) ~[reverse-sync.jar:?]
com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
20:39:22.027 [LaunchDarkly-streaming] WARN com.launchdarkly.sdk.server.LDClient.DataSource - Encountered EventSource error: {}We are having the same issue Error in stream connection (will retry): com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server, and we can see these logs multiple times.
+1 for this. Seeing the same issue. Also seeing the below along with it
com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamIOException: java.net.SocketTimeoutException: timeout
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:132) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
Using launchdarkly-java-server-sdk-6.0.4
Functionality does not seem to be affected since flag evaluations are working just fine. But the exception lights up our alerts and had to explicitly ignore it. Would be nice if the SDK itself suppresses/handles that exception
We are hitting the same issue for multiple services everyday. I've raised support ticket also
We use - compile group: 'com.launchdarkly', name: 'launchdarkly-java-server-sdk', version: '5.10.7'
com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272)
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130)
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109)
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598)
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58)
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571)
at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:188)
at java.lang.Thread.run(Thread.java:750)
I reached out to support and they mentioned to upgrade to latest SDK. but I am seeing issues with even 6.2.1
We are hitting the same issue for multiple services everyday. I've raised support ticket also We use - compile group: 'com.launchdarkly', name: 'launchdarkly-java-server-sdk', version: '5.10.7'
com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571) at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:188) at java.lang.Thread.run(Thread.java:750)
Hello @sourikghosh, @saurabhygk, @aravindparappil46, @prashil-g,
If you don't mind, can you all tell me your server location? I am wondering if is this a geographical-related issue.
Typically, this will happen when there is a deployment on the LaunchDarkly side, which will stop the SDK's connection temporarily, and SDK will retry and re-establish the connection and not impact any flag usage (The SDK will keep serving flags using the data already in memory, and will update once the new connection is up).
the service where we use sdk are in multiple regions - eu-west-1, eu-central-1, us-west-2, us-east-2. we see issue in all of them
I also am having this same issue. LaunchDarkly will be working fine, but occasionally these errors are popping up in the logs. Even when running the application locally from my IDE. I'm using Java SDK 6.2.1 in a Spring Boot application.
I have discussed this with the team, and while this is not a sign of a real issue, Java is not handling the behavior at the correct level and creating a much noisier log, probably causing annoyance with people who try to directly have the logs triggering alerts. We will look into a way to capture this information in a less noisy way (as the SDK will correctly reconnect when it senses a disconnect).
@louis-launchdarkly This is causing operational issue atleast for us. we see that after this error happens, SDK is not reconnecting on its own and new variations are not being pulled
LaunchDarkly data source outage - updates have been unavailable for at least 1 minute with the following errors: NETWORK_ERROR (1 time)
Hello @prashil-g, can you create a ticket via through https://support.launchdarkly.com/ ? Your error looks different from the others - in particular, it doesn't seem like the Stream closed by server issue. We need more information like your SDK configuration and your network topology to debug.