Importing projects into IntelliJ
paul-reiners opened this issue · 2 comments
paul-reiners commented
Can anyone give me a step-by-step guide for importing this project and subprojects into IntelliJ IDEA and running a main method in a class?
This is the error I'm currently getting:
/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/bin/java "-javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=51401:/Applications/IntelliJ IDEA CE.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/lib/tools.jar:/Users/paulreiners/Dropbox/books/aas/ch07-graph/target/classes:/Users/paulreiners/.m2/repository/edu/umd/cloud9/1.5.0/cloud9-1.5.0.jar:/Users/paulreiners/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/Users/paulreiners/.m2/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/Users/paulreiners/.m2/repository/org/apache/spark/spark-sql_2.10/1.0.2/spark-sql_2.10-1.0.2.jar:/Users/paulreiners/.m2/repository/org/apache/spark/spark-core_2.10/1.0.2/spark-core_2.10-1.0.2.jar:/Users/paulreiners/.m2/repository/org/apache/hadoop/hadoop-client/1.0.4/hadoop-client-1.0.4.jar:/Users/paulreiners/.m2/repository/org/apache/hadoop/hadoop-core/1.0.4/hadoop-core-1.0.4.jar:/Users/paulreiners/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/paulreiners/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/paulreiners/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/paulreiners/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/paulreiners/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:/Users/paulreiners/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/paulreiners/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/paulreiners/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/paulreiners/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/paulreiners/.m2/repository/hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar:/Users/paulreiners/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/paulreiners/.m2/repository/org/apache/curator/curator-recipes/2.4.0/curator-recipes-2.4.0.jar:/Users/paulreiners/.m2/repository/org/apache/curator/curator-framework/2.4.0/curator-framework-2.4.0.jar:/Users/paulreiners/.m2/repository/org/apache/curator/curator-client/2.4.0/curator-client-2.4.0.jar:/Users/paulreiners/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/Users/paulreiners/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-plus/8.1.14.v20131031/jetty-plus-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/orbit/javax.transaction/1.1.1.v201105210645/javax.transaction-1.1.1.v201105210645.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-webapp/8.1.14.v20131031/jetty-webapp-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-xml/8.1.14.v20131031/jetty-xml-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-servlet/8.1.14.v20131031/jetty-servlet-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-jndi/8.1.14.v20131031/jetty-jndi-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/orbit/javax.mail.glassfish/1.4.1.v201005082020/javax.mail.glassfish-1.4.1.v201005082020.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/orbit/javax.activation/1.1.0.v201105071233/javax.activation-1.1.0.v201105071233.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-security/8.1.14.v20131031/jetty-security-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-util/8.1.14.v20131031/jetty-util-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-server/8.1.14.v20131031/jetty-server-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/3.0.0.v201112011016/javax.servlet-3.0.0.v201112011016.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-continuation/8.1.14.v20131031/jetty-continuation-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-http/8.1.14.v20131031/jetty-http-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/org/eclipse/jetty/jetty-io/8.1.14.v20131031/jetty-io-8.1.14.v20131031.jar:/Users/paulreiners/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.jar:/Users/paulreiners/.m2/repository/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar:/Users/paulreiners/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/paulreiners/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/Users/paulreiners/.m2/repository/org/slf4j/jul-to-slf4j/1.7.5/jul-to-slf4j-1.7.5.jar:/Users/paulreiners/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.5/jcl-over-slf4j-1.7.5.jar:/Users/paulreiners/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/paulreiners/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar:/Users/paulreiners/.m2/repository/com/ning/compress-lzf/1.0.0/compress-lzf-1.0.0.jar:/Users/paulreiners/.m2/repository/org/xerial/snappy/snappy-java/1.0.5/snappy-java-1.0.5.jar:/Users/paulreiners/.m2/repository/com/twitter/chill_2.10/0.3.6/chill_2.10-0.3.6.jar:/Users/paulreiners/.m2/repository/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar:/Users/paulreiners/.m2/repository/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07.jar:/Users/paulreiners/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:/Users/paulreiners/.m2/repository/org/objenesis/objenesis/1.2/objenesis-1.2.jar:/Users/paulreiners/.m2/repository/com/twitter/chill-java/0.3.6/chill-java-0.3.6.jar:/Users/paulreiners/.m2/repository/commons-net/commons-net/2.2/commons-net-2.2.jar:/Users/paulreiners/.m2/repository/org/spark-project/akka/akka-remote_2.10/2.2.3-shaded-protobuf/akka-remote_2.10-2.2.3-shaded-protobuf.jar:/Users/paulreiners/.m2/repository/org/spark-project/akka/akka-actor_2.10/2.2.3-shaded-protobuf/akka-actor_2.10-2.2.3-shaded-protobuf.jar:/Users/paulreiners/.m2/repository/com/typesafe/config/1.0.2/config-1.0.2.jar:/Users/paulreiners/.m2/repository/io/netty/netty/3.6.6.Final/netty-3.6.6.Final.jar:/Users/paulreiners/.m2/repository/org/spark-project/protobuf/protobuf-java/2.4.1-shaded/protobuf-java-2.4.1-shaded.jar:/Users/paulreiners/.m2/repository/org/uncommons/maths/uncommons-maths/1.2.2a/uncommons-maths-1.2.2a.jar:/Users/paulreiners/.m2/repository/org/spark-project/akka/akka-slf4j_2.10/2.2.3-shaded-protobuf/akka-slf4j_2.10-2.2.3-shaded-protobuf.jar:/Users/paulreiners/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar:/Users/paulreiners/.m2/repository/org/json4s/json4s-jackson_2.10/3.2.6/json4s-jackson_2.10-3.2.6.jar:/Users/paulreiners/.m2/repository/org/json4s/json4s-core_2.10/3.2.6/json4s-core_2.10-3.2.6.jar:/Users/paulreiners/.m2/repository/org/json4s/json4s-ast_2.10/3.2.6/json4s-ast_2.10-3.2.6.jar:/Users/paulreiners/.m2/repository/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar:/Users/paulreiners/.m2/repository/org/scala-lang/scalap/2.10.0/scalap-2.10.0.jar:/Users/paulreiners/.m2/repository/org/scala-lang/scala-compiler/2.10.0/scala-compiler-2.10.0.jar:/Users/paulreiners/.m2/repository/colt/colt/1.2.0/colt-1.2.0.jar:/Users/paulreiners/.m2/repository/concurrent/concurrent/1.3.4/concurrent-1.3.4.jar:/Users/paulreiners/.m2/repository/org/apache/mesos/mesos/0.18.1/mesos-0.18.1.jar:/Users/paulreiners/.m2/repository/io/netty/netty-all/4.0.17.Final/netty-all-4.0.17.Final.jar:/Users/paulreiners/.m2/repository/com/clearspring/analytics/stream/2.5.1/stream-2.5.1.jar:/Users/paulreiners/.m2/repository/com/codahale/metrics/metrics-core/3.0.0/metrics-core-3.0.0.jar:/Users/paulreiners/.m2/repository/com/codahale/metrics/metrics-jvm/3.0.0/metrics-jvm-3.0.0.jar:/Users/paulreiners/.m2/repository/com/codahale/metrics/metrics-json/3.0.0/metrics-json-3.0.0.jar:/Users/paulreiners/.m2/repository/com/codahale/metrics/metrics-graphite/3.0.0/metrics-graphite-3.0.0.jar:/Users/paulreiners/.m2/repository/org/tachyonproject/tachyon/0.4.1-thrift/tachyon-0.4.1-thrift.jar:/Users/paulreiners/.m2/repository/org/apache/ant/ant/1.9.0/ant-1.9.0.jar:/Users/paulreiners/.m2/repository/org/apache/ant/ant-launcher/1.9.0/ant-launcher-1.9.0.jar:/Users/paulreiners/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/paulreiners/.m2/repository/org/spark-project/pyrolite/2.0.1/pyrolite-2.0.1.jar:/Users/paulreiners/.m2/repository/net/sf/py4j/py4j/0.8.1/py4j-0.8.1.jar:/Users/paulreiners/.m2/repository/org/apache/spark/spark-catalyst_2.10/1.0.2/spark-catalyst_2.10-1.0.2.jar:/Users/paulreiners/.m2/repository/org/scala-lang/scala-reflect/2.10.4/scala-reflect-2.10.4.jar:/Users/paulreiners/.m2/repository/com/typesafe/scalalogging-slf4j_2.10/1.0.1/scalalogging-slf4j_2.10-1.0.1.jar:/Users/paulreiners/.m2/repository/com/twitter/parquet-column/1.4.3/parquet-column-1.4.3.jar:/Users/paulreiners/.m2/repository/com/twitter/parquet-common/1.4.3/parquet-common-1.4.3.jar:/Users/paulreiners/.m2/repository/com/twitter/parquet-encoding/1.4.3/parquet-encoding-1.4.3.jar:/Users/paulreiners/.m2/repository/com/twitter/parquet-generator/1.4.3/parquet-generator-1.4.3.jar:/Users/paulreiners/.m2/repository/commons-codec/commons-codec/1.5/commons-codec-1.5.jar:/Users/paulreiners/.m2/repository/com/twitter/parquet-hadoop/1.4.3/parquet-hadoop-1.4.3.jar:/Users/paulreiners/.m2/repository/com/twitter/parquet-format/2.0.0/parquet-format-2.0.0.jar:/Users/paulreiners/.m2/repository/com/twitter/parquet-jackson/1.4.3/parquet-jackson-1.4.3.jar:/Users/paulreiners/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.11/jackson-mapper-asl-1.9.11.jar:/Users/paulreiners/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.11/jackson-core-asl-1.9.11.jar:/Users/paulreiners/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.3.0/jackson-databind-2.3.0.jar:/Users/paulreiners/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.3.0/jackson-annotations-2.3.0.jar:/Users/paulreiners/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.3.0/jackson-core-2.3.0.jar com.cloudera.datascience.graph.RunGraph
objc[1137]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/bin/java (0x1050644c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x1050f04e0). One of the two will be used. Which one is undefined.
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
Process finished with exit code 1
paul-reiners commented
Never mind. I figured it out. You have to check the 'Include dependencies with "Provided" scope' checkbox on the Run Configuration.
srowen commented
They are really meant to be run by spark-submit or in the spark-shell. You can probably also manage to get it running the way you describe too.