sparktraining
所属分类:云计算
开发工具:Scala
文件大小:7042KB
下载次数:0
上传日期:2018-02-18 13:25:01
上 传 者:
sh-1993
说明: chinahadoop.cn星火训练示例
(Examples for Spark Training in chinahadoop.cn)
文件列表:
.idea (0, 2017-05-05)
.idea\.name (13, 2017-05-05)
.idea\compiler.xml (1178, 2017-05-05)
.idea\copyright (0, 2017-05-05)
.idea\copyright\profiles_settings.xml (74, 2017-05-05)
.idea\dictionaries (0, 2017-05-05)
.idea\dictionaries\xicheng_dong.xml (91, 2017-05-05)
.idea\libraries (0, 2017-05-05)
.idea\libraries\Maven__aopalliance_aopalliance_1_0.xml (502, 2017-05-05)
.idea\libraries\Maven__com_101tec_zkclient_0_3.xml (477, 2017-05-05)
.idea\libraries\Maven__com_alibaba_fastjson_1_2_14.xml (502, 2017-05-05)
.idea\libraries\Maven__com_chuusai_shapeless_2_11_2_0_0.xml (537, 2017-05-05)
.idea\libraries\Maven__com_clearspring_analytics_stream_2_7_0.xml (537, 2017-05-05)
.idea\libraries\Maven__com_esotericsoftware_kryo_shaded_3_0_3.xml (552, 2017-05-05)
.idea\libraries\Maven__com_esotericsoftware_minlog_1_3_0.xml (517, 2017-05-05)
.idea\libraries\Maven__com_fasterxml_jackson_core_jackson_annotations_2_6_0.xml (632, 2017-05-05)
.idea\libraries\Maven__com_fasterxml_jackson_core_jackson_core_2_6_5.xml (583, 2017-05-05)
.idea\libraries\Maven__com_fasterxml_jackson_core_jackson_databind_2_6_5.xml (611, 2017-05-05)
.idea\libraries\Maven__com_fasterxml_jackson_module_jackson_module_paranamer_2_6_5.xml (675, 2017-05-05)
.idea\libraries\Maven__com_fasterxml_jackson_module_jackson_module_scala_2_11_2_6_5.xml (682, 2017-05-05)
.idea\libraries\Maven__com_github_fommil_netlib_core_1_1_2.xml (519, 2017-05-05)
.idea\libraries\Maven__com_github_rwl_jtransforms_2_4_0.xml (528, 2017-05-05)
.idea\libraries\Maven__com_google_code_findbugs_jsr305_1_3_9.xml (533, 2017-05-05)
.idea\libraries\Maven__com_google_guava_guava_14_0_1.xml (501, 2017-05-05)
.idea\libraries\Maven__com_google_inject_guice_3_0.xml (484, 2017-05-05)
.idea\libraries\Maven__com_google_protobuf_protobuf_java_2_5_0.xml (562, 2017-05-05)
.idea\libraries\Maven__com_ning_compress_lzf_1_0_3.xml (511, 2017-05-05)
.idea\libraries\Maven__com_thoughtworks_paranamer_paranamer_2_6.xml (548, 2017-05-05)
.idea\libraries\Maven__com_twitter_chill_2_11_0_8_0.xml (509, 2017-05-05)
.idea\libraries\Maven__com_twitter_chill_java_0_8_0.xml (509, 2017-05-05)
.idea\libraries\Maven__com_univocity_univocity_parsers_2_2_1.xml (566, 2017-05-05)
.idea\libraries\Maven__com_yammer_metrics_metrics_core_2_2_0.xml (551, 2017-05-05)
.idea\libraries\Maven__commons_beanutils_commons_beanutils_1_7_0.xml (582, 2017-05-05)
.idea\libraries\Maven__commons_beanutils_commons_beanutils_core_1_8_0.xml (617, 2017-05-05)
.idea\libraries\Maven__commons_cli_commons_cli_1_2.xml (502, 2017-05-05)
.idea\libraries\Maven__commons_codec_commons_codec_1_10.xml (531, 2017-05-05)
.idea\libraries\Maven__commons_collections_commons_collections_3_2_1.xml (604, 2017-05-05)
.idea\libraries\Maven__commons_configuration_commons_configuration_1_6.xml (612, 2017-05-05)
.idea\libraries\Maven__commons_digester_commons_digester_1_8.xml (557, 2017-05-05)
... ...
## sparktraining
Examples for Spark Training in chinahadoop.cn
## 本地运行Spark方法
- 下载spark安装包
- 解压spark安装包
- 进入spark解压目录下,运行:
```bash
$ bin/spark-shell
```
- 在命令行提示符下拷贝以下代码并查看执行结果
```scala
import scala.math.random
val tasks = 10
val n = tasks * 100000
val count = sc.parallelize(1 until n, tasks).map { i =>
val x = random * 2 - 1
val y = random * 2 - 1
if (x*x + y*y <= 1) 1 else 0
}.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / n )
```
## 分布式运行Spark方法
### 搭建hadoop集群
Hadoop YARN/HDFS配置文件参考:conf/hadoop目录
### 配置Spark客户端,并启动spark history server
- Spark客户端配置文件参考:conf/spark目录
- 启动spark history server: sbin/start-history-server.sh
### 将spark-shell运行在yarn client或cluster模式
- yarn client模式:bin/spark-shell --master yarn --deploy-mode client
- yarn cluster:bin/spark-shell --master yarn --deploy-mode cluster
近期下载者:
相关文件:
收藏者: