Connection closed after a while

[JoostFranssen]

I can make many normal requests via your component, but if I leave it running idly for a while, and then make some more requests, I keep getting ‘SocketException: Connection reset’. (Note, not reset by peer, so it's not as if pokeapi.co is cutting me off or anything.)

See the stacktrace below.

pokeapi-reactor-connection-reset-stacktrace.log

Note how first a bunch of requests are made at around 13:11. Then I let it idle for 20 minutes or so, and a couple more requests go successfully around 13:32, but very quickly I then get the SocketException from Netty. I don't think I'm doing anything peculiar and if I run the same requests right when starting by application, everything works fine; it's not like specific requests fail.

Any idea?

Thanks!

[SirSkaro]

Hmm, some quick Googling leads me to believe this may an issue with the provided HttpClient bean. Perhaps the connection is stale? This could be the result of manually providing an HttpClient without specifying any configuration to invalidate connections - like I suggest to do in the README. Are you using the configuration recommended in the README?

Could you perhaps try to change the configuration of your HttpClient bean? I found this Stackoverflow article on how to configure a HttpClient with a HttpClientConnectionManager that closes stale connections automatically. And here are the docs on configuring a HttpClient.

I can experiment with it too. If you are using the configuration in the README and find a better configuration that solves this issue, please share it here and I'll update the README.

[JoostFranssen]

Yes, that's very much what it sounds like! With the help of your links and some more googling on stale connections in Netty, and some debugging, I've got it working (so it seems) now. Here is the relevant configuration:

@Bean
public ConnectionProvider connectionProvider() {
    return ConnectionProvider.builder("pokeapi-reactor")
            .maxIdleTime(Duration.ofSeconds(10))
            .maxConnections(500)
            .pendingAcquireMaxCount(-1)
            .build();
}

@Bean
public HttpClient httpClient(ConnectionProvider connectionProvider) {
    return HttpClient.create(connectionProvider)
            .compress(true)
            .resolver(DefaultAddressResolverGroup.INSTANCE);
}

The name of the builder is arbitrary and does not matter. The maxIdleTime is the main important part; the exact value probably does not matter, but this is what I found in an example and it works.

With your default configuration (HttpClient.create()) I found by debugging that the maxConnections was set to 500, so I just used that (for me it defaulted to 16, which is twice the amount of logical cores). Since I had many parallel requests, I got some exception about pendingAcquireMaxCount being exceeded (which according to the documentation defaults to twice the maxConnections, so 32 in my case). I could not find any kind of limit (or any value at all) for the default configuration, and -1 means no limit, so I just used that.

The rest of the configuration I have left as is.

Thank you for your investigation and advice! It seems to have been solved with this.

[SirSkaro]

Excellent - thank you for figuring that out and sharing! Very much appreciate you explaining the parameters as well. I've updated the README to include your configuration as an example @Configuration

[JoostFranssen]

Great! Glad we got this figured out so quickly :)