NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J

Question

NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J

beena-yatin-kanyal opened this issue 3 years ago · 32 comments

beena-yatin-kanyal commented 3 years ago

We are using some Google cloud dependencies in our rest-assured project and suddenly they have started to break due to the below error. Can anyone please take a look and advice?

IDE - IntelliJ
Gradle - 7.0

Error -

java.lang.NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J

	at com.google.api.client.http.ConsumingInputStream.close(ConsumingInputStream.java:40)
	at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227)
	at java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._closeInput(UTF8StreamJsonParser.java:252)
	at com.fasterxml.jackson.core.base.ParserBase.close(ParserBase.java:369)
	at com.google.api.client.json.jackson2.JacksonParser.close(JacksonParser.java:50)
	at com.google.api.client.json.JsonParser.parse(JsonParser.java:363)
	at com.google.api.client.json.JsonParser.parse(JsonParser.java:335)
	at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79)
	at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73)
	at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:451)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:447)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:145)
	at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:603)
	at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
	at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
	at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:91)
	at com.google.datastore.v1.client.Datastore.lookup(Datastore.java:95)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:171)
	at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:416)
	at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:413)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:412)
	at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.loadResults(DatastoreImpl.java:387)
	at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.<init>(DatastoreImpl.java:383)
	at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:373)
	at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:336)
	at com.google.cloud.datastore.DatastoreHelper.get(DatastoreHelper.java:45)
	at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:321)

Dependencies -

dependencies {
	compile(group: 'com.jayway.jsonpath', name: 'json-path', version: '2.0.0')
	compile group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
//see https://mvnrepository.com/artifact/com.google.cloud/libraries-bom/11.1.0
	compile group: 'com.google.auth', name: 'google-auth-library-oauth2-http', version: '0.21.1'//Jul, 2020
	compile group: 'com.google.cloud', name: 'google-cloud-datastore', version: '1.105.0'//(Sep, 2020)
	compile group: 'com.google.cloud', name: 'google-cloud-storage', version: '1.113.1'//Sep, 2020
	compile group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.108.1' //Aug, 2020
	compile group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '1.120.0' //Sep, 2020
}

configurations.all {
	resolutionStrategy {
		force 'com.google.protobuf:protobuf-java:3.13.0'
		force 'com.google.api-client:google-api-client:1.30.11' //Oct, 2020
	}
}

Answer 1 · 2021-10-25T12:40:31.000Z

When updated the dependencies to the latest -

compile group: 'com.google.auth', name: 'google-auth-library-oauth2-http', version: '1.2.2'
compile group: 'com.google.cloud', name: 'google-cloud-datastore', version: '2.1.3'
compile group: 'com.google.cloud', name: 'google-cloud-storage', version: '2.1.9'
compile group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.114.7'
compile group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '2.3.2'

configurations.all {
	resolutionStrategy {
		force 'com.google.protobuf:protobuf-java:3.19.0'
		force 'com.google.api-client:google-api-client:1.32.2'
	}
}

still getting the same error. Reference logs are attached.

referencelogs.txt

Answer 2 · 2021-10-25T17:42:50.000Z

related issue googleapis/google-api-java-client#1906

Answer 3 · 2021-10-25T19:58:04.000Z

https://stackoverflow.com/questions/61597801/java-lang-nosuchmethoderror-com-google-common-io-bytestreams-exhaustljava-io-i

Suggests you need to include Guava.

That said, the problem is that you probably should be using implementation rather than compile which is deprecated. As mentioned in https://tomgregory.com/gradle-implementation-vs-compile-dependencies/

I'm not a gradle user these days, but w/ maven, compile for a dependency means that it will only be used for compilation, but not added to the imported artifacts that are needed for the java run line. This is used, for example, when running inside a servlet engine that might provide a custom implementation of Servlet, etc.

Answer 4 · 2021-10-26T05:08:26.000Z

https://stackoverflow.com/questions/61597801/java-lang-nosuchmethoderror-com-google-common-io-bytestreams-exhaustljava-io-i

Suggests you need to include Guava.

That said, the problem is that you probably should be using implementation rather than compile which is deprecated. As mentioned in https://tomgregory.com/gradle-implementation-vs-compile-dependencies/

I'm not a gradle user these days, but w/ maven, compile for a dependency means that it will only be used for compilation, but not added to the imported artifacts that are needed for the java run line. This is used, for example, when running inside a servlet engine that might provide a custom implementation of Servlet, etc.

@lesv - I think you were in a hurry to close the ticket rather than providing a proper resolution to the mess created by the GCP libraries. I have already gone through the link before posting anything here and it is of no help. Also if read the link it does not ask you to include the guava dependency. Secondly performed the change with the implementation tag and the same issue.

Opening a new ticket to resolve this blunder.

Answer 5 · 2021-10-26T05:48:32.000Z

thank you @lesv.

I updated the dependencies to the latest and reworked on the suggestion given in the StackOverflow page but the same issue. I would appreciate it if anyone can take a look at the issue.

locallogs.txt

Answer 6 · 2021-10-26T05:50:17.000Z

One of us will look at it in the AM here. If we don't get it right, just comment on the issue or if you can, reopen it yourself. We want things to work for you.

Answer 7 · 2021-10-26T05:53:10.000Z

thanks, @lesv. I am trying different ways to get it to work but no luck. However, it works perfectly fine in the Jenkins for some strange reasons. here is the original grade file I am using :-

dependencies {
	frameworkJars('com.myoffice.jar:APICore:1.0.0'){transitive=false}
	compile(group: 'com.jayway.jsonpath', name: 'json-path', version: '2.0.0')
	compile group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
//see https://mvnrepository.com/artifact/com.google.cloud/libraries-bom/11.1.0
	compile group: 'com.google.auth', name: 'google-auth-library-oauth2-http', version: '0.21.1'//Jul, 2020
	compile group: 'com.google.cloud', name: 'google-cloud-datastore', version: '1.105.0'//(Sep, 2020)
	compile group: 'com.google.cloud', name: 'google-cloud-storage', version: '1.113.1'//Sep, 2020
	compile group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.108.1' //Aug, 2020
	compile group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '1.120.0' //Sep, 2020
}

configurations.all {
	resolutionStrategy {
		force 'com.google.protobuf:protobuf-java:3.13.0'
		force 'com.google.api-client:google-api-client:1.30.11' //Oct, 2020
	}
}

Note - Tried clearing Gradle cache almost 5 times but still same error.

Answer 8 · 2021-10-26T18:27:55.000Z

@beena-yatin-kanyal Just as a work around while we investigate this, have you tried adding:

implementation 'com.google.guava:guava:28.2-jre'

To see if that fixes the problem and gets you going. I've asked @Neenu1995 To take a look at this and give a more definitive answer.

Answer 9 · 2021-10-26T23:53:39.000Z

@beena-yatin-kanyal You shouldn't need to include the oauth library. And, if you are using the libraries-BOM, you shouldn't need to be including versions for your dependencies. The whole point is that you get the dependencies from the BOM. You also shouldn't need protobuf or api-client unless you are using an APIary library.

I'm going to be OoO for a few days, and I won't be able to write an example for you. Perhaps @eaball35 or @Neenu1995 can take a look for me.

Answer 10 · 2021-10-27T14:57:33.000Z

Hello, @lesv thanks for your comments.

[1] - I worked on the BOM suggestion (used the latest version) but still have issues getting our rest-assured test code up and running on the environment. here is the snippet from the Gradle file and the fresh error logs.

Error logs -
freshlogs.txt

Dependency snippet from Gradle -

dependencies {
    frameworkJars('com.myoffice.jar:APICore:1.0.0') { transitive = false }
    implementation(group: 'com.jayway.jsonpath', name: 'json-path', version: '2.0.0')
    implementation group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
    implementation(platform('com.google.cloud:libraries-bom:23.1.0'))
    implementation('com.google.auth:google-auth-library-oauth2-http')
    implementation('com.google.cloud:google-cloud-datastore')
    implementation('com.google.cloud:google-cloud-storage')
    implementation('com.google.cloud:google-cloud-pubsub')
    implementation('com.google.cloud:google-cloud-bigquery')

//	implementation group: 'com.google.cloud', name: 'google-cloud-datastore', version: '1.105.0'//(Sep, 2020)
//	implementation group: 'com.google.cloud', name: 'google-cloud-storage', version: '1.113.1'//Sep, 2020
//	implementation group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.108.1' //Aug, 2020
//	implementation group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '1.120.0' //Sep, 2020
//	implementation 'com.google.guava:guava:28.2-jre'
}

configurations.all {
	resolutionStrategy {
		force 'com.google.protobuf:protobuf-java:3.19.0'
		//force 'com.google.api-client:google-api-client:1.32.2' //Oct, 2020
	}
}

[2] - protobuf - we are reading some messages from the google cloud pub/sub hence we require this library. If I remove it I get some strange errors. Error snippet below for reference -

GeneratedMessageV3
          System.out.println("Id: " + message.getMessageId());
                                             ^
  class file for com.google.protobuf.GeneratedMessageV3 not found

[3] - google-api-client - If I uncomment it original error returns.

[4] - implementation 'com.google.guava:guava:28.2-jre' - no luck same issue.

I would appreciate it if you or someone can take a look at this on priority. It has started to impact our development area.

Answer 11 · 2021-10-28T17:00:20.000Z

Hello @beena-yatin-kanyal,

I'm looking into your issue while @lesv is out, hopefully we can find a solution. If I'm understanding correctly when you use the latest bom you are running into this error:

java.lang.NoSuchMethodError: com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(Ljava/lang/String;)Lcom/google/api/client/googleapis/services/AbstractGoogleClient$Builder;
	at com.google.api.services.storage.Storage$Builder.setBatchPath(Storage.java:11209)
	at com.google.api.services.storage.Storage$Builder.<init>(Storage.java:11188)
        ...

This could be because setBatchPath was introduced in google-api-client 1.23.0 and later. Possibly google-api-client is getting overridden elsewhere? The original issue with guava and this issue with protobuf also seem like potentially similar version collision issues. Have you tried looking at the dependency tree?

Answer 12 · 2021-10-28T18:01:45.000Z

Hello @eaball35
Thanks for reaching out. Yes I did and it somewhere uses guava-v21 but I have overridden it in Gradle using the force keyword. Still no luck :'(

Please find the dependency tree (attached) and the Gradle dependency config (below) I am using to run the code on the dev environment.

tree.txt

dependencies {
	frameworkJars('com.myoffice.jar:APICore:1.0.0') { transitive = false }
	implementation(group: 'com.jayway.jsonpath', name: 'json-path', version: '2.0.0')
	implementation group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
	implementation(platform('com.google.cloud:libraries-bom:23.1.0'))
	implementation('com.google.auth:google-auth-library-oauth2-http')
	implementation('com.google.cloud:google-cloud-datastore')
	implementation('com.google.cloud:google-cloud-storage')
	implementation('com.google.cloud:google-cloud-pubsub')
	implementation('com.google.cloud:google-cloud-bigquery')

//	implementation group: 'com.google.cloud', name: 'google-cloud-datastore', version: '1.105.0'//(Sep, 2020)
//	implementation group: 'com.google.cloud', name: 'google-cloud-storage', version: '1.113.1'//Sep, 2020
//	implementation group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.108.1' //Aug, 2020
//	implementation group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '1.120.0' //Sep, 2020
//	implementation 'com.google.guava:guava:28.2-jre'
}

configurations.all {
	resolutionStrategy {
		force 'com.google.protobuf:protobuf-java:3.19.0'
		force 'com.google.api-client:google-api-client:1.32.2' //Oct, 2020
	}
}

Answer 13 · 2021-10-28T20:37:28.000Z

Looks like Guava is downgraded to 21.0 in both the compile classpath and the runtime classpath.

|    +--- com.google.guava:guava:30.1.1-jre -> 21.0 (c)

There are many ways to resolve a diamond dependency conflict in Gradle. What if you use enforcedPlatform() instead of platform() for libraries-bom? (There is a chance that this might cause other problems though.)

Another definite workaround while using platform() is to force Guava like

implementation('com.google.guava:guava:30.1.1-jre') { force = true }

Answer 14 · 2021-10-29T04:21:23.000Z

Looks like Guava is downgraded to 21.0 in both the compile classpath and the runtime classpath.
|    +--- com.google.guava:guava:30.1.1-jre -> 21.0 (c)
There are many ways to resolve a diamond dependency conflict in Gradle. What if you use enforcedPlatform() instead of platform() for libraries-bom? (There is a chance that this might cause other problems though.)

Another definite workaround while using platform() is to force Guava like
implementation('com.google.guava:guava:30.1.1-jre') { force = true }

@chanseokoh - thanks for your inputs. I added the com.google.guava:guava:30.1.1-jre jar explicitly and forced set it to guava-v30.1.1-jre but it is still messed up. I am really not sure that what's wrong.

Below are the logs for reference.

com.google.cloud.datastore.DatastoreException: I/O error

	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:123)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:173)
	at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:416)
	at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:413)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:412)
	at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.loadResults(DatastoreImpl.java:387)
	at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.<init>(DatastoreImpl.java:383)
	at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:373)
	at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:336)
	at com.google.cloud.datastore.DatastoreHelper.get(DatastoreHelper.java:45)
	at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:321)
	 -- work stuff omitted --
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:62)
	at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:385)
	at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:321)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:176)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
	at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:172)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:118)
	at com.google.datastore.v1.client.Datastore.lookup(Datastore.java:93)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:171)
	... 29 more
Caused by: java.io.IOException: Unexpected error refreshing access token
	at com.google.auth.oauth2.OAuth2Credentials.unwrapDirectFuture(OAuth2Credentials.java:309)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:156)
	at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:1022)
	at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
	at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
	at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:95)
	... 31 more
Caused by: java.lang.NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J
	at com.google.api.client.http.ConsumingInputStream.close(ConsumingInputStream.java:40)
	at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227)
	at java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
	at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
	at sun.nio.cs.StreamDecoder.implClose(StreamDecoder.java:378)
	at sun.nio.cs.StreamDecoder.close(StreamDecoder.java:193)
	at java.io.InputStreamReader.close(InputStreamReader.java:199)
	at com.google.gson.stream.JsonReader.close(JsonReader.java:1224)
	at com.google.api.client.json.gson.GsonParser.close(GsonParser.java:51)
	at com.google.api.client.json.JsonParser.parse(JsonParser.java:363)
	at com.google.api.client.json.JsonParser.parse(JsonParser.java:335)
	at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79)
	at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73)
	at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:463)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:611)
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:243)
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:240)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
	at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:567)
	at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:206)
	... 38 more

Answer 15 · 2021-10-29T13:18:54.000Z

Did you try enforcedPlatform()? What's the dependency tree when using enforcedPlatform()? (Just let me know the tree even if it causes failures. The goal is to check if it affects the tree.)

+--- com.google.guava:guava:30.1.1-jre -> 21.0 (c)

(c) indicates that it's due to a dependency constraint. It's likely that you (your project setup) are imposing it but you're not aware of it.

Always check the dependency tree. The stack trace is not important, as we already know that you're pulling in Guava 21. It's obvious that it'll fail as long as you're using Guava 21. It's mostly on you to figure out what's wrong and do a proper Gradle setup, so keep trying. I can help.

Answer 16 · 2021-10-29T14:38:13.000Z

thanks, @chanseokoh. Appreciate your help. So I did the following after reading your suggestion carefully. please find the details below with the dependency tree attached for each case.

[Scenario1] - updated the Gradle file to use the enforcedPlatform() but still there was a dependency constraint as the com.google.guava:guava:30.1.1-jre was referring to v21.0. attaching the dependency tree for your reference.

tree1.txt

[Scenario2] - after going a bit deep into the Gradle I stumbled upon the force keyword. so I added the following tag to my Gradle (along with enforcedPlatform() tag) and the dependency constraint problem vanished but still no luck. attaching the dependency tree and error logs (just for your to take a look at) for your reference.

Gradle file snippet (used in scenario2) -

dependencies {
    frameworkJars('some.qa.automation:APICore:1.0.0') { transitive = false }
    implementation(group: 'com.jayway.jsonpath', name: 'json-path', version: '2.0.0')
    implementation group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
    implementation(enforcedPlatform('com.google.cloud:libraries-bom:23.1.0'))
    implementation('com.google.auth:google-auth-library-oauth2-http')
    implementation('com.google.cloud:google-cloud-datastore')
    implementation('com.google.cloud:google-cloud-storage')
    implementation('com.google.cloud:google-cloud-pubsub')
    implementation('com.google.cloud:google-cloud-bigquery')

//	implementation group: 'com.google.cloud', name: 'google-cloud-datastore', version: '1.105.0'//(Sep, 2020)
//	implementation group: 'com.google.cloud', name: 'google-cloud-storage', version: '1.113.1'//Sep, 2020
//	implementation group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.108.1' //Aug, 2020
//	implementation group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '1.120.0' //Sep, 2020
//	implementation 'com.google.guava:guava:28.2-jre'
}

configurations.all {
    resolutionStrategy {
        force 'com.google.protobuf:protobuf-java:3.19.0'
        force 'com.google.api-client:google-api-client:1.32.2'
        force 'com.google.guava:guava:30.1.1-jre'
    }
}

tree2.txt

errorlogs.txt

I tried to look at all the dependencies that I am using but they all look good and have tried multiple ways to tweak the dependencies in Gradle but not sure how and why the code is breaking even after forcefully setting the guava to v30.x.x

Answer 17 · 2021-10-29T15:33:48.000Z

At least it fixed one thing, so it's a step forward. It shows that Guava 30 is on both the compile classpath and the runtime classpath. It's good.

Now, this doesn't guarantee that the runtime classpath required to run your application as specified in and identify by Gradle is the exact same runtime classpath you end up using at runtime when running java -cp <runtime classpath> com.example.your.Main. For whatever reasons, It's possible that the guava jar supplied at runtime is not v30 but something old. You said that the error doesn't manifest on Jenkins ("for some strange reasons"), so it already shows that there are different runtime classpaths being used in multiple runtime platforms you use.

How are you packaging up your application and how do you run it for the platform where this error occurs? Packaged as a container? If so, what's the entrypoint of the image? Did you put all the dependency JAR files reported by Gradle to the image? And what's the runtime platform?

Answer 18 · 2021-10-29T16:19:41.000Z

And actually, ByteStreams.exhaust(InputStream) exists in Guava 20 but not in Guava 19. So whether it's using Guava 21 or 30, it shouldn't matter for that error. It's likely that at runtime, even a pre-20 Guava JAR is being supplied (or it comes first in the runtime classpath before another Guava JAR) somehow.

Answer 19 · 2021-10-30T03:31:00.000Z

[1] - I am running the setup on the windows environment. here is the quick snippet of I am running this code -

a) In the above run configuration I also added an extra step (i.e. build) before the run but no luck
b) May be Jenkins it is running fine every time because it creates a fresh workspace, downloads the libs again .... but it is not the case with the local environment.

[2] - I gave a thought to your suggestion in the second comment and it is really a strange thing. I am a little surprised that nowhere I find the pre-20 guava dependency both at runtime and compile time. secondly, there were conflicts constraints (i.e. +--- com.google.guava:guava:30.1.1-jre -> 21.0 (c) but pre-20 guava was no nowhere). I am not sure from where it is picking up during the runtime run and messing up the whole dev environment.

Answer 20 · 2021-11-01T15:19:28.000Z

Never heard of TestNG, but based on my quick search, I'll assume that it's just another test framework for unit tests that is very similar to JUnit and can be substituted for JUnit. (If it does something more than that, or it can fork a JVM process to launch your test code, or you set up your project to do so, etc., then the following arguments may not apply verbatim.)

In any case, I think this is just your local environment and configuration issue.

For compiling test source code and running the compiled tests, you should check the dependency trees of test compile classpath and the test runtime classpath, not the app classpaths. Look for testCompileClasspath and testRuntimeClasspath from the output of ./gradlew dependencies (for your sub-module). However, I don't really think there's an issue in the test classpaths, since you said it works fine on Jenkins. And make sure you are using the exact same code on Jenkins and your local machine.
For troubleshooting, don't run TestNG inside IDE but on the command-line. IDEs and IDE plugins maintain its own classpaths, so it just adds another layer of complexity. There's a good chance that it works on the command-line but not inside IDE. Your goal is to make the command-line work and verify it.
If running tests on the command-line still doesn't work, all I can think of is that TestNG does not correctly run your test code with the right set of test runtime classpath or somehow it interferes with it. You should seek help from the TestNG community.

Answer 21 · 2021-11-01T16:09:43.000Z

@chanseokoh - It is something similar to the selenium framework and yes understanding is correct about the TestNG framework (sort of an automation suite). please find my answers below -

[1] - yes the code is the same. I just tried the git pull origin some_branch and no updates were received.
[2] - I am not sure if it will be a good idea but I will try just to check if something new comes up
[3] - I don't think it is an issue with the TestNG set up as the community will be responsible for their issues, not the ones triggered by google libs. Sorry to say but the libs have messed up by already working environment and now the same issue has started to haunt not just one by multiple developers.

I have cleared the entire Gradle cache from my system and let us see (fingers crossed). hopefully, it should work as it will be redownloading all jars (specified in the project) again.

Answer 22 · 2021-11-01T16:35:39.000Z

I think you have a misunderstanding on the nature of this issue. I am spending my own time to debug your own personal project setup issue out of my curiosity and also as a good citizen. I don't see any evidence that this is an issue caused by Google libs. In my perspective, evidence is pointing to that you (your project setup, environment, configuration, TestNG framework, and what not) are not just correctly using the test runtime classpath. So, all you have to do is to make it do so.

Answer 23 · 2021-11-01T17:03:32.000Z

thanks and I appreciate it @chanseokoh and apologies if my comment was a bit rude. anyways coming back to the issue. I am not sure if I am correct but no matter whichever version of google lib I download the guava always remains 21 (_ I think this is internal to how the jar is bonded_)?

This is not something I am putting it manually. let me see what else I can do now. once again thanks.

Answer 24 · 2021-11-01T17:25:01.000Z

No worries, nothing to apologize. Sorry if you felt that way 😅

That datastore may define the Guava 21 (UPDATE: it's v31) dependency, but don't be too obsessed with it. Because, whether using Maven or Guava, in the end, you can always enforce whatever Guava version you want to use in your project. (And this is what you are actually doing, as v21 is being forced as a dependency constraint, and in #1090 (comment) you were actually able to override to v30.) If you ask how you are supposed to know which Guava version is the best one to use with all GCP libraries, the best way would be to use the libraries BOM (probably with enforcedPlatform()). However, for now, you should just focus on resolving the issue.

That said, even if it were using Guava 21, I don't think that is a problem for the issue, because v21 does have the method of our interest as I said.

As long as the dependency trees that Gradle reports are correct, you should be good. (There is a case where Gradle may report a wrong tree, but I don't think it applies here.) The fact that it also works on Jenkins makes me believe that there are differences at runtime somehow. I emphasize again that for troubleshooting, work on the command-line first while excluding IDE. And I think there's also a possibility that the TestNG framework (or the way you set it up) is playing a role.

Answer 25 · 2021-11-01T17:53:32.000Z

But to answer your question

no matter whichever version of google lib I download the guava always remains 21 (_ I think this is internal to how the jar is bonded_)?

No. datastore declares Guava 31. Check the following log as an example.

~$ cd /tmp
/tmp$ mkdir a
/tmp$ cd a
/tmp/a$ cat > build.gradle <<EOF
plugins { id 'java' }
repositories { mavenCentral() }
dependencies { implementation 'com.google.cloud:google-cloud-datastore:2.1.3' }
EOF
/tmp/a$ gradle dependencies --configuration runtimeClasspath

> Task :dependencies

------------------------------------------------------------
Root project 'a'
------------------------------------------------------------

runtimeClasspath - Runtime classpath of source set 'main'.
\--- com.google.cloud:google-cloud-datastore:2.1.3
     ...
     +--- com.google.api:gax:2.6.1
     +--- com.google.guava:guava:31.0.1-jre
     +--- com.google.guava:failureaccess:1.0.1
     +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
     +--- org.checkerframework:checker-qual:3.12.0
     ...

As I said, it is you (your project setup) that pins the Guava version to v21.

Answer 26 · 2021-11-02T04:55:17.000Z

@chanseokoh - is there any dependency tree/UI that can help find which library is compatible with which? I cleared the entire cache just to be sure that there is no guava-21 lib in the system.

[1] - cleared the entire Gradle cache
[2] - redownloaded the Gradle libs with the new version. worked no guava21 lib.
[3] - failed with a new error - java.lang.NoSuchMethodError: com.google.api.client.http.HttpTransport.isMtls()Z

dependency tree with latest google libs (even forced a few just to be sure)

dependencies {
    frameworkJars('com.myoffice.automation:APICore:1.0.0') {
        transitive = false
    }
    implementation(group: 'com.jayway.jsonpath', name: 'json-path', version: '2.0.0')
    implementation(group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1')
    implementation(group: 'com.google.auth', name: 'google-auth-library-oauth2-http',
            version: '1.2.2')
    implementation(group: 'com.google.cloud', name: 'google-cloud-datastore', version: '2.1.3')
    implementation(group: 'com.google.cloud', name: 'google-cloud-storage', version: '2.1.9')
    implementation(group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.114.7')
    implementation(group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '2.3.3')
}

configurations.all {
    resolutionStrategy {
        force 'com.google.guava:guava:31.0.1-jre'
        force 'com.google.protobuf:protobuf-java:3.19.1'
        force 'com.google.api-client:google-api-client:1.32.2'
    }
}

error

java.lang.NoSuchMethodError: com.google.api.client.http.HttpTransport.isMtls()Z
	at com.google.api.services.storage.Storage$Builder.chooseEndpoint(Storage.java:11151)
	at com.google.api.services.storage.Storage$Builder.<init>(Storage.java:11184)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.<init>(HttpStorageRpc.java:118)
	at com.google.cloud.storage.StorageOptions$DefaultStorageRpcFactory.create(StorageOptions.java:58)
	at com.google.cloud.storage.StorageOptions$DefaultStorageRpcFactory.create(StorageOptions.java:52)
	at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:561)
	at com.google.cloud.storage.StorageOptions.getStorageRpcV1(StorageOptions.java:146)
	at com.google.cloud.storage.StorageImpl.<init>(StorageImpl.java:115)
	at com.google.cloud.storage.StorageOptions$DefaultStorageFactory.create(StorageOptions.java:48)
	at com.google.cloud.storage.StorageOptions$DefaultStorageFactory.create(StorageOptions.java:42)
	at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:541)
(GoogleStorage.java:34)

With the above Gradle implementation, the dependency tree looks clean with no google-api-client is being used. I have explicitly specified google-api-client:1.32.1 (to include one of my choices and remove any constraints).

errorlog.txt

It's a little frustrating due to some strange changes we're revolving round and round with no progress. it is getting a bit frustrating to figure out what are the best dependencies to be used so that the dev team could have it running.

Answer 27 · 2021-11-02T05:06:46.000Z

errorlog.txt represents the dependency tree for the below.

dependencies {
    frameworkJars('com.myoffice.automation:APICore:1.0.0') {
        transitive = false
    }
    implementation(group: 'com.jayway.jsonpath', name: 'json-path', version: '2.0.0')
    implementation(group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1')
    implementation(group: 'com.google.auth', name: 'google-auth-library-oauth2-http',
            version: '1.2.2')
    implementation(group: 'com.google.cloud', name: 'google-cloud-datastore', version: '2.1.3')
    implementation(group: 'com.google.cloud', name: 'google-cloud-storage', version: '2.1.9')
    implementation(group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '1.114.7')
    implementation(group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '2.3.3')
}

configurations.all {
    resolutionStrategy {
        force 'com.google.guava:guava:31.0.1-jre'
        force 'com.google.protobuf:protobuf-java:3.19.1'
        force 'com.google.api-client:google-api-client:1.32.2'
    }
}

note - all the dependencies are set to the latest now.

Answer 28 · 2021-11-02T15:23:13.000Z

is there any dependency tree/UI that can help find which library is compatible with which?

it is getting a bit frustrating to figure out what are the best dependencies to be used

You shouldn't be doing that. Google provides a solution for this: BOM. We ensure compatibility between different Google libraries and some other common dependencies declared in our library BOMs. We have been saying above that

And, if you are using the libraries-BOM, you shouldn't need to be including versions for your dependencies. The whole point is that you get the dependencies from the BOM.

If you ask how you are supposed to know which Guava version is the best one to use with all GCP libraries, the best way would be to use the libraries BOM (probably with enforcedPlatform()).

That is, don't specify versions of the libraries yourself. (You are doing it.) For how to properly use the Google Cloud Libraries BOM in Gradle, see this doc.

Answer 29 · 2021-11-02T15:24:44.000Z

Note the doc says

When we publish a version of Google Cloud Platform Libraries BOM, we ensure that the libraries in the BOM do not have dependency conflicts, that would manifest as NoSuchMethodError or NoClassDefFoundError. By using the Libraries BOM, you can ensure your projects always have compatible versions of these several hundred different artifacts.

It's there to solve the exact problem you are having.

Answer 30 · 2021-11-02T16:44:48.000Z

got it @chanseokoh. let me try that and see if I can make things work now. thanks much for your help. we have a multi-module project so multiple teams work on it and it usually becomes hard to keep track of the BOM versions unless we come up with some common being derived from the parent.

Answer 31 · 2021-11-10T00:47:24.000Z

I'm going to close this since it seems solved. If not, please feel free to reopen or ask us to do so.

Answer 32 · 2021-11-12T15:01:02.000Z

@lesv - thanks. appreciate your help.