/sparktraining

Examples for Spark Training in chinahadoop.cn

Primary LanguageScala

sparktraining

Examples for Spark Training in chinahadoop.cn

本地运行Spark方法

  • 下载spark安装包
  • 解压spark安装包
  • 进入spark解压目录下，运行：
$ bin/spark-shell
  • 在命令行提示符下拷贝以下代码并查看执行结果
import scala.math.random

val tasks = 10
val n = tasks * 100000

val count = sc.parallelize(1 until n, tasks).map { i =>
  val x = random * 2 - 1
  val y = random * 2 - 1
  if (x*x + y*y <= 1) 1 else 0
}.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / n )

分布式运行Spark方法

搭建hadoop集群

Hadoop YARN/HDFS配置文件参考：conf/hadoop目录

配置Spark客户端，并启动spark history server

  • Spark客户端配置文件参考：conf/spark目录
  • 启动spark history server: sbin/start-history-server.sh

将spark-shell运行在yarn client或cluster模式

  • yarn client模式：bin/spark-shell --master yarn --deploy-mode client
  • yarn cluster：bin/spark-shell --master yarn --deploy-mode cluster