AWS S3 - Getting java.net.UnknownHostException: document-storage-service.s3.ap-south-1.amazonaws.com intermittently

[iLoveOctoCat]

Upcoming End-of-Support

• I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

Hello Team,

We are getting UnknownHostException for amazon s3 bucket with bucket name document-storage-service.s3.ap-south-1.amazonaws.com intermittently in our spring boot application.

Till now we have tried following things:

1. Amazon s3 tech support but they asked us to raise the issue on github
2. We tried setting Security.setProperty("networkaddress.cache.ttl", "0"); in main method before spring boot app start
3. We are able to ping successfully this host from container where this app is running

Let us know in case any more details are required.

Expected Behavior

spring boot app should be able to resolve s3 host name successfully

Current Behavior

Application is facing UnknownHostException intermittently and downloading the document from AWS S3 fails

Stacktrace of exception:
2024-06-18 17:29:12.228 Caused by: java.net.UnknownHostException: document-storage-service.s3.ap-south-1.amazonaws.com 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801) 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509) 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1367) 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1301) 2024-06-18 17:29:12.228 at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) 2024-06-18 17:29:12.228 at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) 2024-06-18 17:29:12.228 at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) 2024-06-18 17:29:12.228 at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) 2024-06-18 17:29:12.228 at jdk.internal.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) 2024-06-18 17:29:12.228 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2024-06-18 17:29:12.228 at java.base/java.lang.reflect.Method.invoke(Method.java:568) 2024-06-18 17:29:12.228 at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) 2024-06-18 17:29:12.228 at com.amazonaws.http.conn.$Proxy34.connect(Unknown Source) 2024-06-18 17:29:12.228 at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) 2024-06-18 17:29:12.228 at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) 2024-06-18 17:29:12.228 at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) 2024-06-18 17:29:12.228 at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) 2024-06-18 17:29:12.228 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) 2024-06-18 17:29:12.228 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) 2024-06-18 17:29:12.228 at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) 2024-06-18 17:29:12.228 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1331) 2024-06-18 17:29:12.228 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) 2024-06-18 17:29:12.228 ... 99 common frames omitted

Reproduction Steps

NA

Possible Solution

No response

Additional Information/Context

We have tried disabling the network address cache in our application with following code
public static void main(String[] args) { log.info("Disabling network address cache"); Security.setProperty("networkaddress.cache.ttl", "0"); SpringApplication.run(BootApplication.class, args); }
And logs for the same stating that this property was initialized before application and tomcat server start
2024-06-18 17:17:35.194 11:47:35.178 [main] INFO com.pchf.documentconsumerservice.BootApplication -- Disabling network address cache 2024-06-18 17:17:37.368 2024-06-18 17:17:37.368 . ____ _ __ _ _ 2024-06-18 17:17:37.368 /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ 2024-06-18 17:17:37.368 ( ( )\___ | '_ | '_| | '_ \/ _ | \ \ \
2024-06-18 17:17:37.368 \/ )| |)| | | | | || (| | ) ) ) )
2024-06-18 17:17:37.368 ' || .__|| ||| |_, | / / / /
2024-06-18 17:17:37.368 =========||==============|/=///_/
2024-06-18 17:17:37.368 :: Spring Boot :: (v3.1.6)
2024-06-18 17:17:37.368
2024-06-18 17:17:37.521 2024-06-18 11:47:37 | INFO | [,] | main | c.p.d.BootApplication | Starting BootApplication v1.0.0-SNAPSHOT using Java 17.0.1 with PID 1 (/app/service.jar started by root in /app)
2024-06-18 17:17:37.522 2024-06-18 11:47:37 | INFO | [,] | main | c.p.d.BootApplication | No active profile set, falling back to 1 default profile: "default"`

AWS Java SDK version used

1.12.261

JDK version used

17

Operating System and version

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[debora-ito]

The SDK resolves hosts via InetAddress.getAllByName(), so it sounds like a DNS caching issue in the JVM or the OS. Being an intermittent error also points to a caching issue.

Check this comment from a past UnknownHostException issue: #1503 (comment)