HikariPool failing to initialize because Secret Manager cant find secret when secret exist in AWS Secret Manager
salerak opened this issue · 3 comments
salerak commented
I'm using Hikari with SpringBoot with Secret Manager and I have my application.yml as
datasource:
writer:
url: ${DATASOURCE_URL}
username: ${DATASOURCE_SECRET_KEY}
maximum-pool-size: ${DB_WRITER_MAX_POOL_SIZE}
minimum-idle: ${DB_WRITER_MINIMUM_IDLE}
driver-class-name: ${DATASOURCE_DRIVER_CLASS_NAME}
reader:
url: ${DATASOURCE_RO_URL}
username: ${DATASOURCE_SECRET_KEY}
maximum-pool-size: ${DB_READER_MAX_POOL_SIZE}
minimum-idle: ${DB_READER_MINIMUM_IDLE}
driver-class-name: ${DATASOURCE_DRIVER_CLASS_NAME}
With my ENV params as
DATASOURCE_URL=jdbc-secretsmanager:postgresql://172.17.0.1:5555/transactionhistory
DATASOURCE_RO_URL=jdbc-secretsmanager:postgresql://172.17.0.1:5555/transactionhistory
DATASOURCE_SECRET_KEY=jenkins/transactionhistoryapi/postgres
DATASOURCE_DRIVER_CLASS_NAME=com.amazonaws.secretsmanager.sql.AWSSecretsManagerPostgreSQLDriver
In my jenkins build its failing as it cant find the specified secret.
2019-09-23 18:55:16,238 [main] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Exception during pool initialization.
com.amazonaws.services.secretsmanager.model.ResourceNotFoundException: Secrets Manager can’t find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: d9a3ed64-39f2-4a99-903c-2a8e4a8c46e3)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.doInvoke(AWSSecretsManagerClient.java:2625)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.invoke(AWSSecretsManagerClient.java:2594)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.invoke(AWSSecretsManagerClient.java:2583)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.executeDescribeSecret(AWSSecretsManagerClient.java:895)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.describeSecret(AWSSecretsManagerClient.java:866)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheItem.executeRefresh(SecretCacheItem.java:102)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheItem.executeRefresh(SecretCacheItem.java:32)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheObject.refresh(SecretCacheObject.java:188)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheObject.getSecretValue(SecretCacheObject.java:286)
at com.amazonaws.secretsmanager.caching.SecretCache.getSecretString(SecretCache.java:123)
at com.amazonaws.secretsmanager.sql.AWSSecretsManagerDriver.connectWithSecret(AWSSecretsManagerDriver.java:321)
at com.amazonaws.secretsmanager.sql.AWSSecretsManagerDriver.connect(AWSSecretsManagerDriver.java:384)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:136)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at org.flywaydb.core.internal.util.jdbc.JdbcUtils.openConnection(JdbcUtils.java:51)
at org.flywaydb.core.internal.database.DatabaseFactory.createDatabase(DatabaseFactory.java:70)
at org.flywaydb.core.Flyway.execute(Flyway.java:1227)
at org.flywaydb.core.Flyway.migrate(Flyway.java:910)
at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:66)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1765)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1702)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:304)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:859)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:139)
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:108)
at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:246)
at org.spockframework.spring.SpringTestContextManager.prepareTestInstance(SpringTestContextManager.java:50)
at org.spockframework.spring.SpringInterceptor.interceptSetupMethod(SpringInterceptor.java:42)
at org.spockframework.runtime.extension.AbstractMethodInterceptor.intercept(AbstractMethodInterceptor.java:28)
at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:87)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:472)
at org.spockframework.runtime.BaseSpecRunner.runSetup(BaseSpecRunner.java:375)
at org.spockframework.runtime.BaseSpecRunner.runSetup(BaseSpecRunner.java:370)
at org.spockframework.runtime.BaseSpecRunner.doRunIteration(BaseSpecRunner.java:323)
at org.spockframework.runtime.BaseSpecRunner$6.invoke(BaseSpecRunner.java:309)
at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:481)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:464)
at org.spockframework.runtime.BaseSpecRunner.runIteration(BaseSpecRunner.java:288)
at org.spockframework.runtime.BaseSpecRunner.initializeAndRunIteration(BaseSpecRunner.java:278)
at org.spockframework.runtime.BaseSpecRunner.runSimpleFeature(BaseSpecRunner.java:269)
at org.spockframework.runtime.BaseSpecRunner.doRunFeature(BaseSpecRunner.java:263)
at org.spockframework.runtime.BaseSpecRunner$5.invoke(BaseSpecRunner.java:246)
at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:481)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:464)
at org.spockframework.runtime.BaseSpecRunner.runFeature(BaseSpecRunner.java:238)
at org.spockframework.runtime.BaseSpecRunner.runFeatures(BaseSpecRunner.java:188)
at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:98)
at org.spockframework.runtime.BaseSpecRunner$1.invoke(BaseSpecRunner.java:84)
at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:481)
at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:464)
at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:76)
at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:67)
at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
2019-09-23 18:55:16,240 [main] WARN o.s.w.context.support.GenericWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Secrets Manager can’t find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: d9a3ed64-39f2-4a99-903c-2a8e4a8c46e3)
Using the AWS SDK and manually calling the getSecret
// Use this code snippet in your app.
// If you need more information about configurations or implementing the sample code, visit the AWS docs:
// https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-samples.html#prerequisites
public static void getSecret() {
String secretName = "jenkins/transactionhistoryapi/postgres";
String region = "us-east-1";
// Create a Secrets Manager client
AWSSecretsManager client = AWSSecretsManagerClientBuilder.standard()
.withRegion(region)
.build();
// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
// We rethrow the exception by default.
String secret, decodedBinarySecret;
GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
.withSecretId(secretName);
GetSecretValueResult getSecretValueResult = null;
try {
getSecretValueResult = client.getSecretValue(getSecretValueRequest);
} catch (DecryptionFailureException e) {
// Secrets Manager can't decrypt the protected secret text using the provided KMS key.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (InternalServiceErrorException e) {
// An error occurred on the server side.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (InvalidParameterException e) {
// You provided an invalid value for a parameter.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (InvalidRequestException e) {
// You provided a parameter value that is not valid for the current state of the resource.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (ResourceNotFoundException e) {
// We can't find the resource that you asked for.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
}
// Decrypts secret using the associated KMS CMK.
// Depending on whether the secret is a string or binary, one of these fields will be populated.
if (getSecretValueResult.getSecretString() != null) {
secret = getSecretValueResult.getSecretString();
}
else {
decodedBinarySecret = new String(Base64.getDecoder().decode(getSecretValueResult.getSecretBinary()).array());
}
// Your code goes here.
}
it is able to get the specified secret, below is a sample of what the secret looks like.
aws secretsmanager get-secret-value --secret-id jenkins/transactionhistoryapi/postgres --profile saml
{
"ARN": "arn:aws:secretsmanager:us-east-1:123456789012:secret:jenkins/transactionhistoryapi/postgres-A3BCef",
"Name": "jenkins/transactionhistoryapi/postgres",
"VersionId": "3493d4a0-b7q2-67f6-b24a-9dca3493d4f2",
"SecretString": "{\"username\":\"postgres\",\"password\":\"abc123\",\"engine\":\"postgres\",\"host\":\"111.171.10.11\",\"port\":1115,\"dbname\":\"mydbname\",\"hostReadOnly\":\"111.171.10.11\"}",
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": 1568224188.993
}
Is there any logging I can look at to see what type of secret aws-secretsmanager-jdbc is calling?
Thanks
salerak commented
config issue, hikari with 2 datasource and springboot and it works
CarlosBarbosaFilho commented
Hey guys...
No solution for the error?