首页 > 代码库 > eclipse下进行spark开发(已实践)

eclipse下进行spark开发(已实践)

开发准备:

  jdk1.8.45

  spark-2.0.0-bin-hadoop2.7(windows下和linux个留一份)

  Linux系统(centos或其它)

  spark安装环境

  hadoop-2.7.2(linux一份)

  Hadoop安装环境

 

开发环境搭建步骤如下:

1. 下载scala-SDK-4.4.1-vfinal-2.11-win32.win32.x86_64.tgz

2. 解压压缩包,直接运行里面的eclipse

3. 创建scala project,并创建scala类WordCount

技术分享

4. 右键工程属性,添加spark-2.0.0-bin-hadoop2.7下面所有的库,可自定义库放进来:

技术分享

5. 编辑代码如下:

import org.apache.spark._import SparkContext._object WordCount {   def main(args: Array[String]) {    if (args.length != 3 ){      println("usage is org.test.WordCount <master> <input> <output>")      return    }    val sc = new SparkContext(args(0), "WordCount",    System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_TEST_JAR")))    val textFile = sc.textFile(args(1))    val result = textFile.flatMap(line => line.split("\\s+"))        .map(word => (word, 1)).reduceByKey(_ + _)    result.saveAsTextFile(args(2))  }}

6. 右键类,导出jar文件:

技术分享

 

7. 在spark部署路径执行(可以通过spark的日志找到spark的master地址):

  ./spark-submit  --num-executors 1 --executor-memory 1g --class WordCount --master spark://10.130.41.59:7077 spark-wordcount-in-scala.jar spark://10.130.41.59:7077 hdfs://hadoop:9000/user/hadoop/input hdfs://hadoop:9000/user/hadoop/outspark

8. 参数解析:

  可以执行./spark-submit --help获得帮助

 

eclipse下进行spark开发(已实践)