googleapis/java-storage

com.google.cloud.storage.StorageException: java.lang.IllegalStateException: Connection pool shut down when trying to check if bucket exists

ihudedi opened this issue · 5 comments

ihudedi commented

Hi
I am getting this error when I am trying to check if bucket exists.

2023-08-16 18:18:08,789 ERROR [Thread-3158083] (GCSWrapper:318) -  Error in GCSWrapper::connect - connection with account's gcs credentials failed : java.lang.IllegalStateException: Connection pool shut down
**com.google.cloud.storage.StorageException: java.lang.IllegalStateException: Connection pool shut down**
	at com.google.cloud.storage.StorageException.getStorageException(StorageException.java:85) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	at com.google.cloud.storage.StorageException.coalesce(StorageException.java:100) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	at com.google.cloud.storage.Retrying.run(Retrying.java:54) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1374) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:263) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	at com.bmc.cm.aft.client.wrapper.GCSWrapper.doesBucketExist(GCSWrapper.java:560) ~[classes!/:?]
	at com.bmc.cm.aft.client.wrapper.GCSWrapper.connect(GCSWrapper.java:316) ~[classes!/:?]
	at com.bmc.cm.aft.client.ConnectionManager.connectToHost(ConnectionManager.java:63) ~[classes!/:?]
	at com.bmc.cm.aft.client.SubtaskRunner.connectToHost(SubtaskRunner.java:738) ~[classes!/:?]
	at com.bmc.cm.aft.client.SubtaskRunner.reconnectToHost(SubtaskRunner.java:788) ~[classes!/:?]
	at com.bmc.cm.aft.client.SubtaskRunner.run(SubtaskRunner.java:259) ~[classes!/:?]
	at com.bmc.cm.aft.client.FTCHandler.runSubTasks(FTCHandler.java:412) ~[classes!/:?]
	at com.bmc.cm.aft.client.FTCHandler.runJob(FTCHandler.java:209) ~[classes!/:?]
	at com.bmc.cm.aft.jobmanagement.JobObjectImpl$JobRunnerThread.run(JobObjectImpl.java:368) ~[classes!/:?]
**Caused by: java.lang.IllegalStateException: Connection pool shut down**
	at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.13.jar!/:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.13.jar!/:4.5.13]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar!/:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar!/:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
	at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73) ~[google-http-client-apache-v2-1.41.4.jar!/:?]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.41.4.jar!/:1.41.4]
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:564) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) ~[guava-31.0.1-jre.jar!/:?]
	at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:170) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:967) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) ~[google-auth-library-oauth2-http-1.5.3.jar!/:?]
	at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159) ~[google-cloud-core-http-2.5.6.jar!/:2.5.6]
	at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109) ~[google-cloud-core-http-2.5.6.jar!/:2.5.6]
	at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:91) ~[google-http-client-1.41.4.jar!/:1.41.4]
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:404) ~[google-api-client-1.33.2.jar!/:1.33.2]
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514) ~[google-api-client-1.33.2.jar!/:1.33.2]
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455) ~[google-api-client-1.33.2.jar!/:1.33.2]
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565) ~[google-api-client-1.33.2.jar!/:1.33.2]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:422) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	at com.google.cloud.storage.StorageImpl.lambda$get$4(StorageImpl.java:264) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[gax-2.12.2.jar!/:2.12.2]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[google-cloud-core-2.5.6.jar!/:2.5.6]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[google-cloud-core-2.5.6.jar!/:2.5.6]
	at com.google.cloud.storage.Retrying.run(Retrying.java:51) ~[google-cloud-storage-2.4.5.jar!/:2.4.5]
	... 11 more

Thanks,
Itay

BenWhitehead commented

Can you please provide the following:

  1. Version of com.google.cloud:google-cloud-storage you are using?
  2. How you are constructing the instance of StorageOptions? (The apache http client is non-default and we will need to know your configuration in order to help diagnose any issue.)
  3. How you are constructing your instance of Credentials? (The stack trace looks to be failing while trying to refresh the token)
ihudedi commented

Hi @BenWhitehead
google-cloud-storage-2.4.5.jar - this is my version
This is my code
private static final HttpTransport HTTP_TRANSPORT = new ApacheHttpTransport();
private static final HttpTransportFactory HTTP_TRANSPORT_FACTORY = new DefaultHttpTransportFactory();
HttpTransportFactory httpTransportFactory = HTTP_TRANSPORT_FACTORY;
try (InputStream inputStream = StringUtils.isEmpty(gcsServiceAccountKey) ? null : IOUtils.toInputStream(gcsServiceAccountKey, AS2Utils.ENCODING_UTF_8)) {
googleCredentials = GoogleCredentials.fromStream(inputStream, httpTransportFactory))
//.createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"));
.createScoped(Lists.newArrayList("https://www.googleapis.com/auth/devstorage.full_control"));
}
//build retrySettings
int gcsRequestRetryDelay = mftPropertiesConfig.getIntProperty(PropertyData.gcsRequestRetryDelay, GCS_REQUEST_RETRY_DELAY);
// taken from ServiceOptions.getDefaultRetrySettingsBuilder
RetrySettings retrySettings = RetrySettings.newBuilder()
.setMaxAttempts(gcsRequestMaxAttempts)
.setInitialRetryDelay(Duration.ofMillis(gcsRequestRetryDelay))
.setMaxRetryDelay(Duration.ofMillis(gcsRequestRetryDelay))
.setRetryDelayMultiplier(1.0)
.setTotalTimeout(Duration.ofMillis(60_000L))
.setInitialRpcTimeout(Duration.ofMillis(60_000L))
.setRpcTimeoutMultiplier(1.0)
.setMaxRpcTimeout(Duration.ofMillis(60_000L)).build();
int gcsReadTimeout = mftPropertiesConfig.getIntProperty(PropertyData.gcsReadTimeout, GCS_READ_TIMEOUT);
int gcsConnectionTimeout = mftPropertiesConfig.getIntProperty(PropertyData.gcsConnectionTimeout, GCS_CONNECTION_TIMEOUT);
TransportOptions transportOptions = HttpTransportOptions.newBuilder().
setHttpTransportFactory(httpTransportFactory).
setConnectTimeout(gcsConnectionTimeout).
setReadTimeout(gcsReadTimeout).
build();
storage = StorageOptions.newBuilder().setCredentials(googleCredentials).setRetrySettings(retrySettings).setTransportOptions(transportOptions).build().getService();

ihudedi commented

Hi @BenWhitehead
Any updates regarding this issue?
Thanks,
Itay

BenWhitehead commented

I've spent some time trying to reproduce the exception, and they only way I am able to reproduce it is by calling ApacheHttpTransport#shutdown() before calling a Storage method.

In version 2.4.5, Storage doesn't implement AutoClosable and does not close anything (there are no resources held onto outside individual requests to cleanup).

Perhaps there is a race in your code that is somehow shutting down the Apache pool during Storage calls?

This is my minimized repro in case it might help with your investigation:

import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.apache.v2.ApacheHttpTransport;
import com.google.auth.http.HttpTransportFactory;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.TransportOptions;
import com.google.cloud.http.HttpTransportOptions;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

final class GH2172 {
  private static final HttpTransport HTTP_TRANSPORT = new ApacheHttpTransport();

  public static void main(String[] args) throws Exception {
    HttpTransportFactory httpTransportFactory = () -> HTTP_TRANSPORT;
    GoogleCredentials googleCredentials = GoogleCredentials.getApplicationDefault(httpTransportFactory);
    TransportOptions transportOptions = HttpTransportOptions.newBuilder()
        .setHttpTransportFactory(httpTransportFactory).
        build();
    StorageOptions options = StorageOptions.newBuilder()
        .setCredentials(googleCredentials)
        .setTransportOptions(transportOptions)
        .build();

    Storage storage = options.getService();
    storage.get("<my-bucket>");
    HTTP_TRANSPORT.shutdown(); // Calling this causes any subsequent call to throw the exception
    storage.get("<my-bucket>");
  }
}
BenWhitehead commented

Closing due to inactivity