Generating Denormalized TPCH Dataset
sophieyoung717 opened this issue · 1 comments
This is the command I used learning from here: https://github.com/SparklineData/spark-druid-olap/wiki/Generating-Denormalized-TPCH-Dataset
spark yingyang$ bin/spark-submit --packages com.databricks:spark-csv_2.10:1.1.0,SparklineData:spark-datetime:0.0.2,SparklineData:spark-druid-olap:0.0.2 --class org.sparklinedata.tpch.TpchGenMain /Users/yingyang/Downloads/tpch-spark-druid-master/tpchData/target/scala-2.10/tpchdata_2.10-0.0.1.jar /Users/yingyang/Downloads/data_dbgen --scale 1
I got an error:
Ivy Default Cache set to: /Users/yingyang/.ivy2/cache
The jars for the packages stored in: /Users/yingyang/.ivy2/jars
:: loading settings :: url = jar:file:/Users/yingyang/Downloads/spark/lib/spark-assembly-1.6.2-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.databricks#spark-csv_2.10 added as a dependency
SparklineData#spark-datetime added as a dependency
SparklineData#spark-druid-olap added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.databricks#spark-csv_2.10;1.1.0 in list
found org.apache.commons#commons-csv;1.1 in list
found com.univocity#univocity-parsers;1.5.1 in list
found SparklineData#spark-datetime;0.0.2 in spark-packages
found com.github.nscala-time#nscala-time_2.10;1.6.0 in list
found joda-time#joda-time;2.5 in list
found org.joda#joda-convert;1.2 in list
found SparklineData#spark-druid-olap;0.0.2 in spark-packages
found org.apache.httpcomponents#httpclient;4.5 in central
found org.apache.httpcomponents#httpcore;4.4.1 in central
found commons-logging#commons-logging;1.2 in central
found commons-codec#commons-codec;1.9 in central
found org.json4s#json4s-ext_2.10;3.2.10 in central
found org.joda#joda-convert;1.6 in central
found com.github.scopt#scopt_2.10;3.3.0 in list
downloading http://dl.bintray.com/spark-packages/maven/SparklineData/spark-datetime/0.0.2/spark-datetime-0.0.2.jar ...
[SUCCESSFUL ] SparklineData#spark-datetime;0.0.2!spark-datetime.jar (426ms)
downloading http://dl.bintray.com/spark-packages/maven/SparklineData/spark-druid-olap/0.0.2/spark-druid-olap-0.0.2.jar ...
[SUCCESSFUL ] SparklineData#spark-druid-olap;0.0.2!spark-druid-olap.jar (501ms)
downloading https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar ...
[SUCCESSFUL ] org.apache.httpcomponents#httpclient;4.5!httpclient.jar (99ms)
downloading https://repo1.maven.org/maven2/org/json4s/json4s-ext_2.10/3.2.10/json4s-ext_2.10-3.2.10.jar ...
[SUCCESSFUL ] org.json4s#json4s-ext_2.10;3.2.10!json4s-ext_2.10.jar (19ms)
downloading https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar ...
[SUCCESSFUL ] org.apache.httpcomponents#httpcore;4.4.1!httpcore.jar (75ms)
downloading https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar ...
[SUCCESSFUL ] commons-logging#commons-logging;1.2!commons-logging.jar (18ms)
downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar ...
[SUCCESSFUL ] commons-codec#commons-codec;1.9!commons-codec.jar (69ms)
downloading https://repo1.maven.org/maven2/org/joda/joda-convert/1.6/joda-convert-1.6.jar ...
[SUCCESSFUL ] org.joda#joda-convert;1.6!joda-convert.jar (21ms)
:: resolution report :: resolve 4244ms :: artifacts dl 1239ms
:: modules in use:
SparklineData#spark-datetime;0.0.2 from spark-packages in [default]
SparklineData#spark-druid-olap;0.0.2 from spark-packages in [default]
com.databricks#spark-csv_2.10;1.1.0 from list in [default]
com.github.nscala-time#nscala-time_2.10;1.6.0 from list in [default]
com.github.scopt#scopt_2.10;3.3.0 from list in [default]
com.univocity#univocity-parsers;1.5.1 from list in [default]
commons-codec#commons-codec;1.9 from central in [default]
commons-logging#commons-logging;1.2 from central in [default]
joda-time#joda-time;2.5 from list in [default]
org.apache.commons#commons-csv;1.1 from list in [default]
org.apache.httpcomponents#httpclient;4.5 from central in [default]
org.apache.httpcomponents#httpcore;4.4.1 from central in [default]
org.joda#joda-convert;1.6 from central in [default]
org.json4s#json4s-ext_2.10;3.2.10 from central in [default]
:: evicted modules:
org.joda#joda-convert;1.2 by [org.joda#joda-convert;1.6] in [default]
joda-time#joda-time;2.3 by [joda-time#joda-time;2.5] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 17 | 8 | 8 | 2 || 14 | 8 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
module not found: com.github.SparklineData#spark-datetime;bf5693a575a1dea5b663e4e8b30a0ba94c21d62d
==== local-m2-cache: tried
file:/Users/yingyang/.m2/repository/com/github/SparklineData/spark-datetime/bf5693a575a1dea5b663e4e8b30a0ba94c21d62d/spark-datetime-bf5693a575a1dea5b663e4e8b30a0ba94c21d62d.pom
-- artifact com.github.SparklineData#spark-datetime;bf5693a575a1dea5b663e4e8b30a0ba94c21d62d!spark-datetime.jar:
file:/Users/yingyang/.m2/repository/com/github/SparklineData/spark-datetime/bf5693a575a1dea5b663e4e8b30a0ba94c21d62d/spark-datetime-bf5693a575a1dea5b663e4e8b30a0ba94c21d62d.jar
==== local-ivy-cache: tried
/Users/yingyang/.ivy2/local/com.github.SparklineData/spark-datetime/bf5693a575a1dea5b663e4e8b30a0ba94c21d62d/ivys/ivy.xml
==== central: tried
https://repo1.maven.org/maven2/com/github/SparklineData/spark-datetime/bf5693a575a1dea5b663e4e8b30a0ba94c21d62d/spark-datetime-bf5693a575a1dea5b663e4e8b30a0ba94c21d62d.pom
-- artifact com.github.SparklineData#spark-datetime;bf5693a575a1dea5b663e4e8b30a0ba94c21d62d!spark-datetime.jar:
https://repo1.maven.org/maven2/com/github/SparklineData/spark-datetime/bf5693a575a1dea5b663e4e8b30a0ba94c21d62d/spark-datetime-bf5693a575a1dea5b663e4e8b30a0ba94c21d62d.jar
==== spark-packages: tried
http://dl.bintray.com/spark-packages/maven/com/github/SparklineData/spark-datetime/bf5693a575a1dea5b663e4e8b30a0ba94c21d62d/spark-datetime-bf5693a575a1dea5b663e4e8b30a0ba94c21d62d.pom
-- artifact com.github.SparklineData#spark-datetime;bf5693a575a1dea5b663e4e8b30a0ba94c21d62d!spark-datetime.jar:
http://dl.bintray.com/spark-packages/maven/com/github/SparklineData/spark-datetime/bf5693a575a1dea5b663e4e8b30a0ba94c21d62d/spark-datetime-bf5693a575a1dea5b663e4e8b30a0ba94c21d62d.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: com.github.SparklineData#spark-datetime;bf5693a575a1dea5b663e4e8b30a0ba94c21d62d: not found
::::::::::::::::::::::::::::::::::::::::::::::
:::: ERRORS
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.github.SparklineData#spark-datetime;bf5693a575a1dea5b663e4e8b30a0ba94c21d62d: not found]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1068)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Can you run build/sbt clean compile on this repo before running TpchGenMain.
The JitPack.IO resolver in this build.sbt will take care of downloading the datetime package to your local ivyCache.