This is a simple example of a Spark 1.6.x application that reads data from a Hive table using Spark SQL does some grouping and aggregation and saves the output to a file on HDFS
Hive --> Spark --> HDFS
- JDK 1.8
- Spark 1.6.x
- Scala 2.10
If you are using IntelliJ, import the project as Maven project using JDK 1.8
Use maven on the command line to create a jar.
mvn package
Find the jars, one with the other without dependencies in the target directory.
In order to submit your application to an existing YARN cluster with Spark 1.6.x libraries installed you need to perform the following commands. The kinit command is only required on a secure cluster.
kinit username [-kt /path/to/keytab]
spark-submit --class com.hortonworks.SparkHiveExample --master yarn spark-example-1.0-SNAPSHOT.jar MyApplicationName /path/to/hdfs-dir/with/write/access/