(short issue description)

Question

(short issue description)

MichaelJHaywood opened this issue a year ago · 2 comments

Describe the bug

I'm getting a Connection pool shut down exception when using a LambdaClient. The LambdaClient is created each time I need to call the lambda as the config can change.

After what appears to be a random number of calls it starts failing and throwing a Connection pool shut down exception. It will then fail with every subsequent call and is only resolved by restarting the service.

It's strange as we're creating the LambdaClient each time and it should call .close() as it's using a try-with-resource statement. It's as though there is something stored globally that is being shut down which means any new LambdaClients will fail to make a connection.

I've also noticed this error seems to occur after the service has been running for 60 minutes (I can't be certain). It works fine for many requests and then starts failing every time.

try (LambdaClient client = LambdaClient.builder()
        .region(region)
        .credentialsProvider(DefaultCredentialsProvider.create())
        .overrideConfiguration(
                builder -> builder.apiCallTimeout(Duration.ofSeconds(118))
                        .apiCallAttemptTimeout(Duration.ofSeconds(118))
                        .retryPolicy(RetryPolicy.none()))
        .httpClientBuilder(ApacheHttpClient.builder()
                .maxConnections(10)
                .socketTimeout(Duration.ofSeconds(130))
                .connectionTimeout(Duration.ofSeconds(130))
        )
        .build()) {
    res = client.invoke(request);
    lambdaResponse = res.payload().asUtf8String();
}

Exception

2023-12-20T16:23:27.765Z ERROR 1 --- [undedElastic-23] p.h.GlobalResponseEntityExceptionHandler : Run Time Exception: Connection pool shut down

java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.16.jar:4.4.16]
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:103) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.Mono.block(Mono.java:1712) ~[reactor-core-3.5.11.jar:3.5.11]
		at 
	4) ~[app/:na]
		at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:73) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:32) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:227) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.5.11.jar:3.5.11]
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
		at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler 
Original Stack Trace:
		at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.16.jar:4.4.16]
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.14.jar:4.5.14]
		at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57) ~[apache-client-2.21.11.jar:na]
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.14.jar:4.5.14]
		at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:89) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:52) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:37) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.21.11.jar:na]

Why is the connection pool closed for a new LambdaClient?

Expected Behavior

A LambdaClient should be able to be created and disposed without affecting other LambdaClients.

Current Behavior

A newly created LambdaClient throws an exception due to the connection pool being shutdown.

Reproduction Steps

This isn't easily reproduced. We only have this piece of code that creates a LambdaClient, makes a request and closes and this starts failing after the service has been running for some time.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.21.11

JDK version used

Amazon Corretto: 17

Operating System and version

Amazon Linux: 2

Answer 1 · 2024-01-03T22:13:08.000Z

It's strange as we're creating the LambdaClient each time and it should call .close() as it's using a try-with-resource statement.

Is it possible that some part of the system still tries to use the client after it got closed?

For context, the SDK does not close connection pools when the client is in use, the SDK only closes the connection pool when close() is invoked directly or indirectly (by something outside the SDK, like Spring @Bean destroyMethod() for example).

Have you look into this error?

	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:103) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.Mono.block(Mono.java:1712) ~[reactor-core-3.5.11.jar:3.5.11]

Answer 2 · 2024-01-09T00:05:45.000Z

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.