googleapis/java-storage

grpc compatibility issue 2.27.1

patelprateek opened this issue · 9 comments

patelprateek commented

recreating the ticket #2231 which was pre-maturely marked as complete without any verification
cloud storage 2.27.0 and 2.27.1 , running into different dependency issues

using gcs cloud storage 2.27.0 and grpc-java 1.58.0

@maven//:com_google_cloud_google_cloud_storage
@maven//:io_grpc_grpc_inprocess
.-> @maven//:io_grpc_grpc_core
| @maven//:io_grpc_grpc_util
`-- @maven//:io_grpc_grpc_core

OR

@maven//:com_google_cloud_google_cloud_storage
@maven//:io_grpc_grpc_netty_shaded
.-> @maven//:io_grpc_grpc_core
| @maven//:io_grpc_grpc_util
`-- @maven//:io_grpc_grpc_core

cyclic depdendency issue

If i use an older version of gfprc-java 1.56.0 or older then
noSuchMethodError
"java.lang.NoSuchMethodError: 'void io.grpc.internal.AbstractManagedChannelImplBuilder.(java.lang.String)'"

=> java.lang.NoSuchMethodError: 'void io.grpc.internal.AbstractManagedChannelImplBuilder.(java.lang.String)'

io.grpc.netty.NettyChannelBuilder.(NettyChannelBuilder.java:147)

io.grpc.netty.NettyChannelBuilder.(NettyChannelBuilder.java:142)
I saw that the particular method has been removed in 1.58.0

What is the correct way to handle this cyclic dependency or which version of grpc java should i be using ?

BenWhitehead commented

A few possible options I can think of

First option

Leverage libraries-bom it will manage the versions of grpc and storage to ensure there is only one declared version and everything is compatible. This works very well for both Maven and Gradle.

Second option

An alternative, if you are not actively using StorageOptions.grpc() as part of any private preview access to the GCS gRPC API, you can safely exclude all grpc related dependencies from google-cloud-storage

    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-storage</artifactId>
      <exclusions>
        <exclusion>
          <groupId>io.grpc</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

Third option

If you are part of the private preview with access to the GCS gRPC API the following versions are what version of gRPC we depend on

  • com.google.cloud:google-cloud-storage:2.27.0 depends on io.grpc:*:1.56.1
  • com.google.cloud:google-cloud-storage:2.27.1 depends on io.grpc:*:1.58.0

Aligning with these versions should ensure you don't have any incompatibilities.

Forth option

If you are wanting to change the version of grpc google-cloud-storage uses (which is not supported, and you would be on your own in the event of any issues), you would want to exclude all grpc related artifacts to remove the chance of duplicates

    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-storage</artifactId>
      <exclusions>
        <exclusion>
          <groupId>io.grpc</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

and then explicitly include the following grpc artifacts which are dependencies of the google-cloud-storage gRPC implementation:

  • io.grpc:grpc-alts:jar:1.58.0:compile
  • io.grpc:grpc-api:jar:1.58.0:compile
  • io.grpc:grpc-auth:jar:1.58.0:compile
  • io.grpc:grpc-context:jar:1.58.0:compile
  • io.grpc:grpc-core:jar:1.58.0:compile
  • io.grpc:grpc-googleapis:jar:1.58.0:runtime
  • io.grpc:grpc-grpclb:jar:1.58.0:compile
  • io.grpc:grpc-inprocess:jar:1.58.0:compile
  • io.grpc:grpc-netty-shaded:jar:1.58.0:runtime
  • io.grpc:grpc-protobuf:jar:1.58.0:compile
  • io.grpc:grpc-protobuf-lite:jar:1.58.0:compile
  • io.grpc:grpc-rls:jar:1.58.0:runtime
  • io.grpc:grpc-services:jar:1.58.0:runtime
  • io.grpc:grpc-stub:jar:1.58.0:compile
  • io.grpc:grpc-util:jar:1.58.0:runtime
  • io.grpc:grpc-xds:jar:1.58.0:runtime

If none of these work for you we would need more information about the build/dependency management tool you use, along with what overlapping dependencies your project declares that google-cloud-storage also declares.

patelprateek commented

Thanks for the suggestions

We use bazel for our builds : 4.2.2

Also some more infor the Poms here look problematic

https://repo1.maven.org/maven2/io/grpc/grpc-core/1.58.0/grpc-core-1.58.0.pom

https://repo1.maven.org/maven2/io/grpc/grpc-util/1.58.0/grpc-util-1.58.0.pom

patelprateek commented

update
upgrading cs storage to 2.27.0 and grpc-java to 1.56.1 Works
but 2.27.1 and 1.58.0 results in cyclic dependency

also when i take a look at the above POM it seems the cyclic dependency do exist

BenWhitehead commented

I'm personally not at all familiar with how Bazel performs dependency resolution. I'll reach out to some colleagues to see if they have any insight they can share.

suztomo commented

@patelprateek Would you share a reproducer Bazel project that shows the build failure?

As far as I know, a bare Bazel project does not use Maven's pom.xml.

patelprateek commented

Let me tryto see if i can have this reproduced because we use a custom wrapper over bazel and forked internal jvm_rule to source artifacts from gcs.

Another question I have regarding 2.27.0 (and grpc 1.56.1)
I observe storageOption have grpc and http . My legacy code was using the defaultInstance which was http. Would you guys recommend using grpc ?
Are there any performance improvements ? any benchmarks. My use case is typically downloading binary files from gcs buckets to local disk , these files can range from couple MBs to few GBs and typically can download thousands of files to SSDs which can reach terabyte scale.

patelprateek commented 
   at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:540)
    at com.snapchat.purple.indexer.IndexerJob.listIndexDirs(IndexerJob.java:126)
    at com.snapchat.purple.indexer.IndexerJob.downloadFiles(IndexerJob.java:108)
    at com.snapchat.purple.indexer.IndexerJob.main(IndexerJob.java:168)

Caused by: java.lang.ClassNotFoundException: com.google.api.gax.rpc.mtls.MtlsProvider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 19 more

frankyn commented

Hi @patelprateek, were you able to create reproducible steps to this issue or were you able to resolve it?

I observe storageOption have grpc and http . My legacy code was using the defaultInstance which was http. Would you guys recommend using grpc ?

gRPC is allowlist only and not publicly available at this time.

frankyn commented

Closing due to inactivity; please reopen if still an issue.