Getting Error java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset
erarjit opened this issue · 0 comments
erarjit commented
I am using example provided on in the doc.
I started spark shell using below
bin/spark-shell --packages com.stratio.datasource:spark-mongodb_2.11:0.12.0
Then executed below code
import org.apache.spark.sql._
import com.mongodb.casbah.{WriteConcern => MongodbWriteConcern}
import com.stratio.datasource.mongodb._
import com.stratio.datasource.mongodb.config._
import com.stratio.datasource.mongodb.config.MongodbConfig._
val sparkSession = SparkSession.builder().getOrCreate()
case class Student(name: String, age: Int)
val dataFrame: DataFrame = sparkSession.createDataFrame(sc.parallelize(List(Student("Torcuato", 27), Student("Rosalinda", 34))))
val saveConfig = MongodbConfigBuilder(Map(Host -> List("localhost:27017"), Database -> "highschool", Collection ->"students", SamplingRatio -> 1.0, WriteConcern -> "normal", SplitSize -> 8, SplitKey -> "_id"))
dataFrame.saveToMongodb(saveConfig.build)
But at the last line I am getting below error.
scala> dataFrame.saveToMongodb(saveConfig.build)
java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset
at com.stratio.datasource.mongodb.MongodbDataFrame.saveToMongodb(MongodbDataFrame.scala:36)
... 56 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 57 more