首页 > 代码库 > ubuntu下Sprak(IDE)wordcount例子

ubuntu下Sprak(IDE)wordcount例子

一、进入IDE界面

cd ~/Downloads/idea/bin

idea.sh

 

二、建立scala项目

Step 1:导入Spark-hadoop对应的包,次选择“File”–> “Project Structure” –> “Libraries”,选择“+”,将spark-hadoop 对应的包导入:

 

点击“OK”确认:

点击“OK”:

IDEA工作完成后会发现Spark的jar包导入到了我们的工程中:

第二步、写scala代码实现wordcount

import SparkContext

import SparkContext._
object Wordcount {
def main(args: Array[String]) {
val sc = new SparkContext("local", "Simple App", "home\\yuanqin\\Downloads\\spark-0.9.0-incubating-bin-hadoop1",null)
//List("lib/spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar")
val logFile = sc.textFile("hdfs://Master:9000/user/root/input/file.txt") // Should be some file on your system
val count=logFile.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
println(count)
count.saveAsTextFile("home\\yuanqin\\output\\2014")

}
}

第三步、右击run as....

 

 

ubuntu下Sprak(IDE)wordcount例子