首页 > 代码库 > Spark
Spark
Spark简介
官网地址：http://spark.apache.org/
Apache Spark is a fast and general engine for large-scale data processing.
Speed
Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.
Apache Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing.
Ease of Use
Write applications quickly in Java, Scala, Python, R.
Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python and R shells.
Generality
Combine SQL, streaming, and complex analytics.
Spark powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application.
Runs Everywhere
Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3.
You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN, or on Apache Mesos. Access data in HDFS, Cassandra, HBase, Hive, Tachyon, and any Hadoop data source.
官方文档地址：http://spark.apache.org/docs/1.6.0/
安装
下载：
Spark runs on Java 7+, Python 2.6+ and R 3.1+. For the Scala API, Spark 1.6.0 uses Scala 2.10. You will need to use a compatible Scala version (2.10.x).
spark版本：spark-1.6.0-bin-hadoop2.6.tgz
scala版本：scala-2.10.4.tgz
spark监控端口 :4040
安装集群：
tar -xzvf spark-1.6.0-bin-hadoop2.6.tgz -C /usr/
修改解压后的配置文件 /conf/spark-env.sh
export SPARK_MASTER_IP=node1export SPARK_MASTER_PORT=7077export SPARK_WORKER_CORES=1export SPARK_WORKER_INSTANCES=1export SPARK_WORKER_MEMORY=1024mexport SPARK_LOCAL_DIRS=/data/spark/dataDirexport HADOOP_CONF_DIR=/usr/hadoop-2.5.1/etc/hadoop
修改slaves文件
node2node3
集群管理器类型有
Standalone
Apache Mesos
Hadoop Yarn
这里介绍两种方式 Standalone 和 Yarn
配置文件中的 HADOOP_CONF_DIR就是为了使用Yarn配置的
Standalone
启动spark自身的管理器
spark/sbin/start-all.sh
运行测试脚本
standalone client模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://node1:7077 --executor-memory 512m --total-executor-cores 1 ./lib/spark-examples-1.6.0-hadoop2.6.0.jar 100
standalone cluster模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://node1:7077 --deploy-mode cluster --supervise --executor-memory 512M --total-executor-cores 1 ./lib/spark-examples-1.6.0-hadoop2.6.0.jar 100
查询结果http://node1:4040 client模式通过4040查询不到结果，在脚本执行结束后在终端就能看到结果。
Yarn
启动hadoop集群, 由于使用Yarn不需要启动spark
start-all.sh
运行测试脚本：
Yarn client模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --executor-memory 512M --num-executors 1 ./lib/spark-examples-1.6.0-hadoop2.6.0.jar 100
Yarn cluster模式
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --executor-memory 512m --num-executors 1 ./lib/spark-examples-1.6.0-hadoop2.6.0.jar 100
查询执行结果 http://node1:8088 client模式通过8088查询不到结果，在脚本执行结束后在终端就能看到结果。
集群模式
The following table summarizes terms you’ll see used to refer to cluster concepts:
| Term | Meaning |
|---|---|
| Application | User program built on Spark. Consists of a driver program and executors on the cluster. |
| Application jar | A jar containing the user‘s Spark application. In some cases users will want to create an "uber jar" containing their application along with its dependencies. The user‘s jar should never include Hadoop or Spark libraries, however, these will be added at runtime. |
| Driver program | The process running the main() function of the application and creating the SparkContext |
| Cluster manager | An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN) |
| Deploy mode | Distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster. |
| Worker node | Any node that can run application code in the cluster |
| Executor | A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors. |
| Task | A unit of work that will be sent to one executor |
| Job | A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save, collect); you‘ll see this term used in the driver‘s logs. |
| Stage | Each job gets divided into smaller sets of tasks called stages that depend on each other (similar to the map and reduce stages in MapReduce); you‘ll see this term used in the driver‘s logs. |
Spark