首页 > 代码库 > 64位Linux编译hadoop-2.5.1

64位Linux编译hadoop-2.5.1

2024-08-01 01:51:47   223人阅读

Apache Hadoop生态系统安装包下载地址：http://archive.apache.org/dist/

软件安装目录：~/app

jdk: jdk-7u45-linux-x64.rpmhadoop: hadoop-2.5.1-src.tar.gzmaven: apache-maven-3.0.5-bin.zipprotobuf: protobuf-2.5.0.tar.gz

 

1、下载hadoop

wget http://archive.apache.org/dist/hadoop/core/stable/hadoop-2.5.1-src.tar.gz tar -zxvf hadoop-2.5.1-src.tar.gz

在解压后的hadoop根目录下有个BUILDING.txt文件，可以看到编译hadoop的环境要求

Requirements:* Unix System* JDK 1.6+* Maven 3.0 or later* Findbugs 1.3.9 (if running findbugs)* ProtocolBuffer 2.5.0* CMake 2.6 or newer (if compiling native code)* Zlib devel (if compiling native code)* openssl devel ( if compiling native hadoop-pipes )* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

2、安装jdk

sudo yum install jdk-7u45-linux-x64.rpm

查看jdk安装位置：

which java/usr/java/jdk1.7.0_45/bin/java

添加jdk到环境变量(~/.bash_profile)：

export JAVA_HOME=/usr/java/jdk1.7.0_45export PATH=.:$JAVA_HOME/bin:$PATH

验证：

java -version
java version "1.7.0_45"Java(TM) SE Runtime Environment (build 1.7.0_45-b18)Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

3、安装maven

wget http://apache.fayea.com/apache-mirror/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.zipunzip apache-maven-3.0.5-bin.zip

添加maven到环境变量(~/.bash_profile):

export MAVEN_HOME=/home/hadoop/app/apache-maven-3.0.5export PATH=.:$MAVEN_HOME/bin:$PATH

验证：

mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 05:51:28-0800)Maven home: /home/hadoop/app/apache-maven-3.0.5Java version: 1.7.0_45, vendor: Oracle CorporationJava home: /usr/java/jdk1.7.0_45/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

4、安装protobuf

protobuf的官方地址貌似上不了，自行下载protobuf安装包；为了编译安装protobuf，需要先gcc/gcc-c++/make

sudo yum install gccsudo yum install gcc-c++sudo yum install make
tar -zvxf protobuf-2.5.0.tar.gz cd protobuf-2.5.0./configure --prefix=/usr/local/protoc/ sudo makesudo make install

添加protobuf到环境变量(~/.bash_profile):

export PATH=.:/usr/local/protoc/bin:$PATH

验证：

protoc --versionlibprotoc 2.5.0

5、安装其他依赖

sudo yum install cmakesudo yum install openssl-develsudo yum install ncurses-devel

6、编译hadoop源代码

cd ~/app/hadoop-2.5.1-src mvn package -DskipTests -Pdist,native
[INFO] Reactor Summary:[INFO] [INFO] Apache Hadoop Main ................................ SUCCESS [1.980s][INFO] Apache Hadoop Project POM ......................... SUCCESS [1.575s][INFO] Apache Hadoop Annotations ......................... SUCCESS [3.324s][INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.318s][INFO] Apache Hadoop Project Dist POM .................... SUCCESS [1.550s][INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [4.548s][INFO] Apache Hadoop MiniKDC ............................. SUCCESS [3.410s][INFO] Apache Hadoop Auth ................................ SUCCESS [4.503s][INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.915s][INFO] Apache Hadoop Common .............................. SUCCESS [1:25.913s][INFO] Apache Hadoop NFS ................................. SUCCESS [8.324s][INFO] Apache Hadoop Common Project ...................... SUCCESS [0.064s][INFO] Apache Hadoop HDFS ................................ SUCCESS [2:31.023s][INFO] Apache Hadoop HttpFS .............................. SUCCESS [16.389s][INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [8.235s][INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [4.493s][INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.041s][INFO] hadoop-yarn ....................................... SUCCESS [0.031s][INFO] hadoop-yarn-api ................................... SUCCESS [1:11.828s][INFO] hadoop-yarn-common ................................ SUCCESS [30.542s][INFO] hadoop-yarn-server ................................ SUCCESS [0.047s][INFO] hadoop-yarn-server-common ......................... SUCCESS [13.953s][INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [16.537s][INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [3.270s][INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [7.840s][INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [15.877s][INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.421s][INFO] hadoop-yarn-client ................................ SUCCESS [6.406s][INFO] hadoop-yarn-applications .......................... SUCCESS [0.025s][INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [3.208s][INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [1.885s][INFO] hadoop-yarn-site .................................. SUCCESS [0.058s][INFO] hadoop-yarn-project ............................... SUCCESS [2.870s][INFO] hadoop-mapreduce-client ........................... SUCCESS [0.065s][INFO] hadoop-mapreduce-client-core ...................... SUCCESS [26.292s][INFO] hadoop-mapreduce-client-common .................... SUCCESS [19.197s][INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [5.229s][INFO] hadoop-mapreduce-client-app ....................... SUCCESS [12.322s][INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [10.640s][INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [5.154s][INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [1.939s][INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [8.088s][INFO] hadoop-mapreduce .................................. SUCCESS [2.979s][INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [5.615s][INFO] Apache Hadoop Distributed Copy .................... SUCCESS [7.668s][INFO] Apache Hadoop Archives ............................ SUCCESS [2.014s][INFO] Apache Hadoop Rumen ............................... SUCCESS [6.567s][INFO] Apache Hadoop Gridmix ............................. SUCCESS [4.398s][INFO] Apache Hadoop Data Join ........................... SUCCESS [3.151s][INFO] Apache Hadoop Extras .............................. SUCCESS [3.251s][INFO] Apache Hadoop Pipes ............................... SUCCESS [1.901s][INFO] Apache Hadoop OpenStack support ................... SUCCESS [5.722s][INFO] Apache Hadoop Client .............................. SUCCESS [4.021s][INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.095s][INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [5.776s][INFO] Apache Hadoop Tools Dist .......................... SUCCESS [2.768s][INFO] Apache Hadoop Tools ............................... SUCCESS [0.035s][INFO] Apache Hadoop Distribution ........................ SUCCESS [8.571s][INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 10:30.071s[INFO] Finished at: Sat Nov 08 04:42:01 PST 2014[INFO] Final Memory: 91M/324M[INFO] ------------------------------------------------------------------------

编译后的代码在hadoop-2.5.1-src/hadoop-dist/target/hadoop-2.5.1下，以后要搭建hadoop环境直接使用hadoop-2.5.1文件夹部署即可。

 

64位Linux编译hadoop-2.5.1


联系
我们