AuraSparkTraining
所属分类:云计算
开发工具:Scala
文件大小:8466KB
下载次数:0
上传日期:2018-10-15 14:22:10
上 传 者:
sh-1993
说明: Aura.cn的火花训练示例
(Spark Training Example For Aura.cn)
文件列表:
AuraSparkTraining.iml (19925, 2018-10-15)
bin (0, 2018-10-15)
bin\run-pi-local.sh (372, 2018-10-15)
bin\run-pi-yarn.sh (613, 2018-10-15)
bin\run.sh (147, 2018-10-15)
conf (0, 2018-10-15)
conf\hadoop (0, 2018-10-15)
conf\hadoop\core-site.xml (335, 2018-10-15)
conf\hadoop\hadoop-env.sh (4230, 2018-10-15)
conf\hadoop\hdfs-site.xml (545, 2018-10-15)
conf\hadoop\mapred-site.xml (433, 2018-10-15)
conf\hadoop\slaves (42, 2018-10-15)
conf\hadoop\yarn-site.xml (1009, 2018-10-15)
conf\spark (0, 2018-10-15)
conf\spark\spark-defaults.conf (301, 2018-10-15)
conf\spark\spark-env.sh (74, 2018-10-15)
data (0, 2018-10-15)
data\access.log (13888, 2018-10-15)
data\basketball (0, 2018-10-15)
data\basketball\leagues_NBA_1970_per_game_per_game.csv (18863, 2018-10-15)
data\basketball\leagues_NBA_1971_per_game_per_game.csv (23859, 2018-10-15)
data\basketball\leagues_NBA_1972_per_game_per_game.csv (23809, 2018-10-15)
data\basketball\leagues_NBA_1973_per_game_per_game.csv (23655, 2018-10-15)
data\basketball\leagues_NBA_1974_per_game_per_game.csv (27138, 2018-10-15)
data\basketball\leagues_NBA_1975_per_game_per_game.csv (28657, 2018-10-15)
data\basketball\leagues_NBA_1976_per_game_per_game.csv (29040, 2018-10-15)
data\basketball\leagues_NBA_1977_per_game_per_game.csv (36003, 2018-10-15)
data\basketball\leagues_NBA_1978_per_game_per_game.csv (35722, 2018-10-15)
data\basketball\leagues_NBA_1979_per_game_per_game.csv (35055, 2018-10-15)
data\basketball\leagues_NBA_1980_per_game_per_game.csv (38681, 2018-10-15)
data\basketball\leagues_NBA_1981_per_game_per_game.csv (40957, 2018-10-15)
data\basketball\leagues_NBA_1982_per_game_per_game.csv (42993, 2018-10-15)
data\basketball\leagues_NBA_1983_per_game_per_game.csv (43051, 2018-10-15)
data\basketball\leagues_NBA_1984_per_game_per_game.csv (42232, 2018-10-15)
data\basketball\leagues_NBA_1985_per_game_per_game.csv (43594, 2018-10-15)
data\basketball\leagues_NBA_1986_per_game_per_game.csv (44332, 2018-10-15)
data\basketball\leagues_NBA_1987_per_game_per_game.csv (45693, 2018-10-15)
... ...
## sparktraining
Examples for Spark Training in aura.cn
## 本地运行Spark方法
- 下载spark安装包
- 解压spark安装包
- 进入spark解压目录下,运行:
```bash
$ bin/spark-shell
```
- 在命令行提示符下拷贝以下代码并查看执行结果
```scala
import scala.math.random
val tasks = 10
val n = tasks * 100000
val count = sc.parallelize(1 until n, tasks).map { i =>
val x = random * 2 - 1
val y = random * 2 - 1
if (x*x + y*y <= 1) 1 else 0
}.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / n )
```
## 分布式运行Spark方法
### 搭建hadoop集群
Hadoop YARN/HDFS配置文件参考:conf/hadoop目录
### 配置Spark客户端,并启动spark history server
- Spark客户端配置文件参考:conf/spark目录
- 启动spark history server: sbin/start-history-server.sh
### 将spark-shell运行在yarn client或cluster模式
- yarn client模式:bin/spark-shell --master yarn --deploy-mode client
- yarn cluster:bin/spark-shell --master yarn --deploy-mode cluster
近期下载者:
相关文件:
收藏者: