Guava version issue

Question

Guava version issue

SatishAthukuri opened this issue 5 years ago · 4 comments

This package uses old version[13.0.1 released in 2012] of guava creating an issue in writing an end-end integration test case for BigQueryStreamingSink [https://github.com/SatishAthukuri/change-data-capture/blob/b28fe62922cbcb1e82891ade8f0897a900a162cf/src/main/java/io/cdap/plugin/cdc/sink/BigQueryStreamingSink.java]. We're getting this exception[java.lang.NoClassDefFoundError: com/google/common/base/MoreObjects] when we try to retrive application credentials by using either service account key file or from default Instance. If we run test case with 18.0.1 or later guava version we're getting the following exception.

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2019-07-23 11:23:57,792 - INFO [main:i.c.c.t.TestBase@420] - Custom configuration set: cdap.unit.test.app.program.spark.compat = spark2_2.11
2019-07-23 11:23:57,798 - INFO [main:i.c.c.t.TestBase@420] - Custom configuration set: cdap.unit.test.explore.enabled = false
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 2.018 sec <<< FAILURE!
bqplugin.BigQueryStreamingSinkTest Time elapsed: 2.017 sec <<< ERROR!
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at io.cdap.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:44)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
at io.cdap.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:71)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getDeclaredConstructors(Class.java:2020)
at com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
at com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
at com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
at com.google.inject.internal.UntargettedBindingProcessor$1.visit(UntargettedBindingProcessor.java:51)
at com.google.inject.internal.UntargettedBindingProcessor$1.visit(UntargettedBindingProcessor.java:35)
at com.google.inject.internal.UntargettedBindingImpl.acceptTargetVisitor(UntargettedBindingImpl.java:41)
at com.google.inject.internal.UntargettedBindingProcessor.visit(UntargettedBindingProcessor.java:35)
at com.google.inject.internal.UntargettedBindingProcessor.visit(UntargettedBindingProcessor.java:27)
at com.google.inject.internal.BindingImpl.acceptVisitor(BindingImpl.java:93)
at com.google.inject.internal.AbstractProcessor.process(AbstractProcessor.java:56)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:187)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:200)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
at com.google.inject.Guice.createInjector(Guice.java:96)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at io.cdap.cdap.test.TestBase.initialize(TestBase.java:229)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at io.cdap.cdap.common.test.TestRunner.run(TestRunner.java:73)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

bqplugin.BigQueryStreamingSinkTest Time elapsed: 2.018 sec <<< ERROR!
java.lang.NullPointerException: null
at io.cdap.cdap.test.TestBase.finish(TestBase.java:485)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at io.cdap.cdap.common.test.TestRunner.run(TestRunner.java:73)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

Results :

Tests in error:
BigQueryStreamingSinkTest>TestBase.initialize:229 » IncompatibleClassChange Im...
BigQueryStreamingSinkTest>TestBase.finish:485 » NullPointer

Tests run: 2, Failures: 0, Errors: 2, Skipped: 0

To reproduce the issue with current[13.0.1] guava version available in pom run the following test case:

clone the repo.

Add BigQuery dependecy in pom:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-bigquery</artifactId>
  <version>1.71.0</version>
</dependency>
<dependency>
  <groupId>com.google.cloud.bigdataoss</groupId>
  <artifactId>bigquery-connector</artifactId>
  <version>0.13.1-hadoop2</version>
</dependency>

Add the following test case in package io.cdap.plugin.cdc.integration.sink

 import com.google.cloud.bigquery.BigQuery;
 import com.google.cloud.bigquery.BigQueryOptions;
 import io.cdap.plugin.cdc.util.BigQueryUtil;
 import org.junit.Test;

 import java.io.IOException;

 public class BigQueryAPITest {
     @Test
     public void testBigQueryAPI() throws IOException {
         //Using instance Credentials
         BigQuery bigQuery = BigQueryOptions.getDefaultInstance().getService();
         //Using service account key file path.
         BigQuery bigqueryFileCredentials = 
                 BigQueryUtil.getBigQuery("Service account file path",  "projectid");
     }
 }

Answer 1 · 2019-08-01T17:08:42.000Z

You should not be using this plugin to unit test an unrelated plugin. A custom BigQuery plugin should not be adding this as a dependency.

Answer 2 · 2019-08-16T15:17:33.000Z

The intention behind this issue is to develop a plugin that streams data directly into BQ instead of using batch GCS load. A similar streaming plugin (but for BigTable) can be found in this package: https://github.com/data-integrations/change-data-capture/blob/develop/src/main/java/io/cdap/plugin/cdc/sink/CDCBigTable.java
Before creating a pull request, we want to add an integration test. The snippet above is a minimum reproducible sample that reproduces the issue we encountered while writing the integration test.

Answer 3 · 2019-09-09T22:03:49.000Z

Hey @albertshau, could you elaborate your point given the previous comment?

Answer 4 · 2019-09-10T16:36:06.000Z

I see, you're trying to add a CDC BigQuery sink to this project. I thought you were trying to use the existing BigQuery sink in a test. If you're adding a plugin and the dependencies clash, you have to update the dependencies to get to a set that doesn't clash.

If you're talking about testing pipelines through the cdap-unit-test framework, there's a bug open in CDAP around guava clashes, since the unit test framework uses guava 13, which is incompatible with later versions. Until guava is removed (not a small task), the unit test framework is unusable. You will have to structure your code in such a way that you can test the appropriate code without actually running an in-memory pipeline.