SNOW-974631: Socket timeout errors from 3.14.x version onwards

Question

SNOW-974631: Socket timeout errors from 3.14.x version onwards

Closed this issue 8 months ago · 23 comments

Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!

What version of JDBC driver are you using?
3.14.x
What operating system and processor architecture are you using?
Linux, 3.10.0-1160.102.1.el7.x86_64
What version of Java are you using?
11.0.12
What did you do?
Earlier, we were using 3.13.x version of the snowflake JDBC JAR. Things were all good. The moment we switch to 3.14.0 then we started seeing this failure:
com.snaplogic.api.ExecutionException: Error while waiting for PUT command to finish at com.snaplogic.snaps.snowflake.BulkLoad.execute(BulkLoad.java:549) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:812) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:586) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:877) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:436) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:120) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: AWS operation failed: Operation=listObjects, Error message=Unable to execute HTTP request: Connection reset at java.base/java.util.concurrent.FutureTask.report(Unknown Source) at java.base/java.util.concurrent.FutureTask.get(Unknown Source) at com.snaplogic.snaps.snowflake.AbstractBulkSnap.waitForPutChunksCompletion(AbstractBulkSnap.java:978) at com.snaplogic.snaps.snowflake.BulkLoad.execute(BulkLoad.java:547) ... 11 more Caused by: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: AWS operation failed: Operation=listObjects, Error message=Unable to execute HTTP request: Connection reset at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.handleS3Exception(SnowflakeS3Client.java:740) at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.handleStorageException(SnowflakeS3Client.java:676) at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.filterExistingFiles(SnowflakeFileTransferAgent.java:2327) at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.execute(SnowflakeFileTransferAgent.java:1326) at net.snowflake.client.core.SFStatement.executeFileTransfer(SFStatement.java:783) at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:129) at net.snowflake.client.core.SFStatement.execute(SFStatement.java:769) at net.snowflake.client.core.SFStatement.execute(SFStatement.java:677) at net.snowflake.client.jdbc.SnowflakeStatementV1.executeInternal(SnowflakeStatementV1.java:320) at net.snowflake.client.jdbc.SnowflakeStatementV1.execute(SnowflakeStatementV1.java:392) at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94) at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) at com.snaplogic.snaps.snowflake.AbstractBulkSnap.putChunks(AbstractBulkSnap.java:944) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at com.snaplogic.cc.snap.common.SnapHolder.lambda$ensureThreadGroup$3(SnapHolder.java:428) ... 1 more Caused by: net.snowflake.client.jdbc.internal.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:902) at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.listObjects(SnowflakeS3Client.java:274) at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.filterExistingFiles(SnowflakeFileTransferAgent.java:2313) ... 15 more Caused by: java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at net.snowflake.client.jdbc.internal.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) at net.snowflake.client.jdbc.internal.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) at net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at net.snowflake.client.jdbc.internal.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at jdk.internal.reflect.GeneratedMethodAccessor103.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at net.snowflake.client.jdbc.internal.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at net.snowflake.client.jdbc.internal.amazonaws.http.conn.$Proxy163.connect(Unknown Source) at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at net.snowflake.client.jdbc.internal.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at net.snowflake.client.jdbc.internal.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ... 29 more
What did you expect to see?
3.14.0 version should also work like earlier versions.
Can you set logging to DEBUG and collect the logs?
Not now as it happens in customer's environment.
What is your Snowflake account identifier, if any? (Optional)
Snaplogic

Note: We tried using all the released 3.14.x versions as of today so far but still the same problem keeps occuring. We even tried setting maxHttpRetries to zero in the URL properties of the jdbc string but still the same issue persists.

Answer 1 · 2023-11-20T11:58:11.000Z

Can you pls help us with how to get past this issue with the 3.14.x version pls? We don't want our customer integrations to fail due to this upgrade.

Answer 2 · 2023-11-21T15:12:32.000Z

Hi @Syed-SnapLogic,
Thanks for reaching out.
We're going to require a bit more information for your particular scenario.

Have you noticed this happening only with PUT statements?

We're going to need debugging logs from your environment to help us understand the nature of the issue. Typically, when you see this type of error:

Caused by: java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at

That's an issue occurring at the network layer. In this error stack, it suggests that the client attempted to initiate a TLS connection by starting a handshake, and then the remote side forcefully closed the connection by sending a RST packet.

I'm not sure why you would only see this in the 3.14.x versions in comparison to the 3.13.x versions, but that's something that would require a deeper dive in your environment if it's not anything we're able to reproduce.

Can I ask you to open a support ticket and reference this GitHub issue, please?

Can you be sure to enable JDBC debugging when reproducing this issue and attach the log when opening the ticket, please?

Answer 3 · 2023-11-22T06:14:10.000Z

@sfc-gh-wfateem The issue is occuring only when we enable the proxy, and i see there is a fix made related to some proxy in 3.14.0 https://github.com/snowflakedb/snowflake-jdbc/pull/1475/files this might be cause of regression?

Answer 4 · 2023-11-22T15:24:11.000Z

@muskaan62 the changes you're referring to is specifc to the connections made later on to the cloud provider's storage server when you get to the point where you need to download resultsets. For that to happen, your first connection in order to login has to work, is that happening? Based on the following in the stack, it would seem that this is occurring when the driver is trying to download data:

net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346)

I recall there was an issue where the proxy settings passed to the JDBC driver weren't take affect when connecting to the S3 service and that's because the S3 client was picking up environment variables. You can refer to the details of how the AWS S3 SDK library determines which proxy settings to use in PR #1475.

How are you passing your proxy settings to the JDBC driver, and which parameters are you using?

Answer 5 · 2023-11-23T12:47:59.000Z

@sfc-gh-wfateem

yes, we are able to login
The proxy setting are done via jvm options as
http.proxyHost = host
http.proxyUser = user
http.proxyPassword = pwd
http.proxyPort = portno
with the same settings it was working till 3.13.34 version and its not working with driver 3.14.x

Answer 6 · 2023-11-23T16:01:57.000Z

Right, so if you're able to login with those proxy settings but it's failing to download data then it sounds like the proxy settings are not taking effect for S3 connections.

I did a quick test by using:

-Dhttp.proxyHost=127.0.0.1
-Dhttp.proxyPort=234
-Dhttp.nonProxyHosts="*.snowflakecomputing.com"

That's a non-existing proxy server, so any connections going through the proxy server should fail. I bypassed only connections going to *.snowflakecomputing.com, so if the S3 connections were in fact picking up the proxy configurations then I would expect those to fail. However, all connections were working, which pretty much confirms that these proxy settings are ignored by the S3 client.

As a workaround, just add the following JVM argument:
-Dhttp.useProxy=true

Answer 7 · 2023-11-24T07:42:28.000Z

@sfc-gh-wfateem I tried the workaround but its not working(below i s the stack trace with the workaround)
Note: i replace the hostname with host in stacktrace
com.snaplogic.api.ExecutionException: Error while waiting for PUT command to finish
at com.snaplogic.snaps.snowflake.BulkLoad.execute(BulkLoad.java:549)
at com.snaplogic.snaps.snowflake.AbstractBulkSnap.executeForSuggest(AbstractBulkSnap.java:1353)
at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:807)
at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeForSuggest(SnapRunnableImpl.java:648)
at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:865)
at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:436)
at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:120)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: AWS operation failed: Operation=listObjects, Error message=Unable to execute HTTP request: host.amazonaws.com
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at com.snaplogic.snaps.snowflake.AbstractBulkSnap.waitForPutChunksCompletion(AbstractBulkSnap.java:978)
at com.snaplogic.snaps.snowflake.BulkLoad.execute(BulkLoad.java:547)
... 12 more
Caused by: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: AWS operation failed: Operation=listObjects, Error message=Unable to execute HTTP request: host.amazonaws.com
at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.handleS3Exception(SnowflakeS3Client.java:740)
at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.handleStorageException(SnowflakeS3Client.java:676)
at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.filterExistingFiles(SnowflakeFileTransferAgent.java:2327)
at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.execute(SnowflakeFileTransferAgent.java:1326)
at net.snowflake.client.core.SFStatement.executeFileTransfer(SFStatement.java:783)
at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:129)
at net.snowflake.client.core.SFStatement.execute(SFStatement.java:769)
at net.snowflake.client.core.SFStatement.execute(SFStatement.java:677)
at net.snowflake.client.jdbc.SnowflakeStatementV1.executeInternal(SnowflakeStatementV1.java:320)
at net.snowflake.client.jdbc.SnowflakeStatementV1.execute(SnowflakeStatementV1.java:392)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94)
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at com.snaplogic.snaps.snowflake.AbstractBulkSnap.putChunks(AbstractBulkSnap.java:944)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at com.snaplogic.cc.snap.common.SnapHolder.lambda$ensureThreadGroup$3(SnapHolder.java:428)
... 1 more
Caused by: net.snowflake.client.jdbc.internal.amazonaws.SdkClientException: Unable to execute HTTP request: host.amazonaws.com
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397)
at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928)
at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:902)
at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.listObjects(SnowflakeS3Client.java:274)
at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.filterExistingFiles(SnowflakeFileTransferAgent.java:2313)
... 15 more
Caused by: java.net.UnknownHostException: host.amazonaws.com
at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source)
at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at net.snowflake.client.jdbc.internal.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
at net.snowflake.client.jdbc.internal.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
at net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at net.snowflake.client.jdbc.internal.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at jdk.internal.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at net.snowflake.client.jdbc.internal.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at net.snowflake.client.jdbc.internal.amazonaws.http.conn.$Proxy162.connect(Unknown Source)
at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at net.snowflake.client.jdbc.internal.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at net.snowflake.client.jdbc.internal.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346)
at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
... 29 more

Answer 8 · 2023-11-25T20:36:07.000Z

@Syed-SnapLogic It might have just been a coincidence that this occurred after you tried the proposed workaround, but that's a different issue. This is a plain DNS lookup issue, and shouldn't be related to the JDBC driver upgrade:

Caused by: java.net.UnknownHostException: host.amazonaws.com
at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source)
at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at net.snowflake.client.jdbc.internal.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
at net.snowflake.client.jdbc.internal.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)

Are you consistently getting a java.net.UnknownHostException? If so, what is the output of the following command from that same machine hosting your JDBC driver:
nslookup host.amazonaws.com

Answer 9 · 2023-11-27T16:34:19.000Z

@Syed-SnapLogic have you been able to figure out that DNS issue you were facing?

Answer 10 · 2023-11-28T11:01:46.000Z

@sfc-gh-wfateem Give us some time,to respond to your query.

Answer 11 · 2023-12-04T17:34:05.000Z

@muskaan62 just checking if you still needed help with this.
I'm reclassifying this issue as an enhancement where ideally we shouldn't need to pass the JVM argument
-Dhttp.useProxy=true once a user has already configured the proxy by passing the other JVM arguments like
-Dhttp.proxyHost and -Dhttp.proxyPort

Answer 12 · 2023-12-05T12:42:47.000Z

@sfc-gh-wfateem That unknown host issue is not yet resolve from my end..i'll let you know by tomorrow if the provided workaround work for us or not. Thanks for your quick responses.

Answer 13 · 2023-12-07T12:29:31.000Z

@sfc-gh-wfateem https://github.com/snowflakedb/snowflake-jdbc/pull/1475/files

As per the release notes Fixed an issue where the JDBC driver used the wrong proxy settings for S3 clients.
does this mean that previous version of jar were not considering the proxy settings for the driver and now with the new driver(from 3.14.0 and above) it is picking up the proxy settings and hence we are encountering this unknownhost or connection reset error.?
could it be the reason of regression??

Answer 14 · 2023-12-07T15:22:35.000Z

@Syed-SnapLogic I think the connection reset was actually caused by the fact that your S3 connections were not going through your proxy server in your environment, which resulted in that connection reset. We use the AWS S3 client library for data transfers to and from the S3 service. The issue that we discovered that needed to be addressed with that change you're referring to is that the S3 client (only) was picking up proxy settings from environment variables (HTTP_PROXY, HTTPS_PROXY, NO_PROXY). That's counter to what all the other components do, where none of them consider environment variables.

That change addresses that particular issue, which might have not been a problem for you, however, with that change it became necessary to add the JVM argument -Dhttp.useProxy=true

You could consider it a "regression," but if you think about it, simply reverting that change would be yet another regression, but more severe than the one you're experiencing here. The issue you're facing is easily addressed using that JVM argument. The UnknownHostException you're experiencing should be unrelated to any of the changes we're discussing here because that's purely a DNS issue.

I've been able to reproduce your issue and confirm everything I'm explaining here, including the workaround I proposed. The workaround does not result in an UnknownHostException in my case because in either scenario, I'm able to perform a DNS lookup on the S3 host regardless of what configurations I'm using.

If you're saying that you're consistently getting an UknownHostException with the new release but that immediately goes away once you revert back to the older release, then that tells me there's some other change in the S3 host format used between the older and the newer release, and that your DNS server doesn't have a record for the S3 host in the new format being used. For instance, for the S3 bucket name "support-example" in US West-2, you can have the following valid formats:

support-example.s3.amazonaws.com
support-example.s3.us-west-2.amazonaws.com
support-example.s3-us-west-2.amazonaws.com

You might have a DNS record for one of those but not the other two. In any other environment all hosts will resolve correctly, like in my case, which would be the only reason in my mind why I'm not seeing the UnknownHostException and you are.

Are you consistently getting an UnknownHostException when you use the -Dhttp.useProxy JVM argument? Have you confirmed if you're able to perform a DNS lookup on that S3 host that's resulting in an UnknownHostException?

You can capture the TCP/IP packets to analyze the DNS queries sent by the client using the old and new JDBC driver releases to compare those and see which S3 host is the driver trying to establish a connection with.

Answer 15 · 2023-12-11T13:16:24.000Z

@sfc-gh-wfateem Thanks for your valuable time and detailed explanation, I really appreciate it. we are facing some challenges to reproduce this and seeing some different behaviour..If its possible for you I would like to connect and have a short call with you..please let me know if it is feasible for you to connect.
if yes, please let me know the possible date and time of yours to schedule a call..Thanks

Answer 16 · 2023-12-11T13:28:34.000Z

@muskaan62 Sure thing.
Can you open a support case and just reference this GitHub issue, please?

Answer 17 · 2023-12-13T09:22:45.000Z

@sfc-gh-wfateem , we are not able to open a support case. Is it fine if we can proceed with a short call without any support ticket? Let me know few of your convenient time slots (along with your time zone) so I can pick the one that works best for both of us.

Answer 18 · 2023-12-13T14:23:23.000Z

@muskaan62 you can use this link to book a time for us.
I look forward to speaking with you soon.

Answer 19 · 2023-12-14T09:30:47.000Z

@sfc-gh-wfateem I have schedule a call on Friday(15 Dec) at 9 AM CST..Looking forward to see you there, Thanks.

Answer 20 · 2023-12-15T16:20:44.000Z

@Syed-SnapLogic I met with @muskaan62 and it's what I was suspecting might have been happening in your environment. There are two things that are happening here:

1.) In order to get the S3 connections to go through the proxy server, you have to add the JVM argument
-Dhttp.useProxy=true

That addressed your customer's particular problem with the connection reset error, and in their environment they're able to resolve the S3 hostnames without any issues.

2.) In your environment, however, you're getting an UnknownHostException because you can only resolve one of the S3 hostname formats, and not the format *.s3.us-west-2.amazonaws.com

Which basically means, there's some other change that happened between the 3.13.x and 3.14.x JDBC driver where the S3 host format used has changed and it's specifically a problem in your environment.

That's something I'll need to look into a bit closer.

Answer 21 · 2024-01-11T13:06:26.000Z

I compared the hostname format for the S3 hosts when connecting using the JDBC driver version 3.13.34 and 3.14.3. There weren't any differences, so the issue seems to be specific to your environment.

As I pointed out earlier, you simply need to add the -Dhttp.useProxy=true JVM argument as stated in our docs.

I'll go ahead and close this off since you have confirmed in the support ticket with us that adding the JVM argument
-Dhttp.useProxy=true resolved your issue.

Answer 22 · 2024-02-07T10:41:36.000Z

@sfc-gh-wfateem I have one query is this property "-Dhttp.useProxy=true" exclusive to snowflake??

Answer 23 · 2024-02-07T13:04:08.000Z

@muskaan62 yes