Exception in thread "main" java.lang.NoSuchMethodError:
Closed this issue · 14 comments
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68)
at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:40)
Can you please tell me from which library you are getting 'checkArgument'?
I am facing the same problem. I was using the library without problems in google Dataproc environtment and from 22 of febrary is failing on the last version of dataproc.
Sorry folks I am away on Holiday till Friday
I believe that error is due to a guava version mismatch, if you google it you will see where the problem is
Same issue appears here #72
I think if you shade your libraries and create a fat jar it should work as long as guava is over version 14, i use 18 in my project and it seems to work but I have not tested with the latest dataproc
I shall be looking into this over the weekend, but if any of you want to do a headstart and test it out I would really appreciate it
Hi all,
I have try, WITHOUT success, packing guava library on the jar adding dependency for guava v18, in the
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
Any other think I could try?
Thanks.
Felix
This issue is now fixed, please try the latest release and reopen if not the case
Thanks for raising it
Thanks for your time! but it seems not work for me.
I have update pom file to get new version 0.2.6 how ever the error persist.
BigQueryClient:141 - Executing query SELECT * FROM [business-intelligence-noticias:DS_DEV_minsait_dev.Rel_CanalCaptacion]
2019-03-27 15:12:22 INFO BigQueryClient:164 - Staging dataset datanoticiasdesarrollo:spark_bigquery_staging_eu already exists
2019-03-27 15:12:22 INFO BigQueryClient:144 - Destination table: {datasetId=spark_bigquery_staging_eu, projectId=datanoticiasdesarrollo, tableId=spark_bigquery_20190327141222_1158602716}
2019-03-27 15:12:22 INFO BigQueryClient:200 - Using legacy Sql: true
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68)
.
I include the maven tree output if it helps.
mvnTres.txt
Thanks in advance
Felix
yes It still persists, I am also getting this error
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68)
at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:40)
at com.samelamin.spark.bigquery.streaming.BigQuerySink.addBatch(BigQuerySink.scala:29)
......
....
...
Using guava v19, spark 2.3 on dataproc.
I also face the same issue with latest version.
Similar issue on my side, didnt found work arround yet
I created a workflow with this dependencies provided by the cluster :
And it solves differents versions issues
jobs:
- sparkJob:
jarFileUris:
- gs://bucket/xxxx-0.1.0-SNAPSHOT.jar
mainClass: jobs.BQ2GCS
stepId: step1
placement:
managedCluster:
clusterName: xxx
config:
gceClusterConfig:
metadata:
bigquery-connector-version: 0.13.16
gcs-connector-version: 1.9.16
zoneUri: europe-west1-b
initializationActions:
- executableFile: gs://dataproc-initialization-actions/connectors/connectors.sh
executionTimeout: 600s
softwareConfig:
imageVersion: '1.4'
workerConfig:
numInstances: 2
@dgouyette Thank you it is working.
I could also make it works by forcing "com.google.guava:guava:27.0.1-jre"
to be use
I also face the same issue with latest version.
were you able to fix this? If yes , can you please share the soluion. Thanks
@dgouyette Thank you it is working.
I could also make it works by forcing"com.google.guava:guava:27.0.1-jre"
to be use
This still doesnt resolve the issue for me
I came across this error when running on a newer version of Dataproc cluster.
To resolve it do the following in your build (mine assembly) with shade rules to get rid of old references to guava
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.common.**" -> "my_conf.@1").inAll
)
Also see this dated email of mine
https://groups.google.com/forum/#!msg/cloud-dataproc-discuss/OICSgtN2oe8/bhOswni8EAAJ
currently I am using this dependency in my assembly
libraryDependencies += "com.google.guava" % "guava" % "27.0.1-jre"
HTH,
Mich
Thanks Mich, guava is such a pain. Essentially this is fixed by shading the google common libraries
If the above doesnt work I am happy reopening this ticket