首页 > 代码库 > hadoop入门-在windows上编译x64位hadoop
hadoop入门-在windows上编译x64位hadoop
环境
Windows 7 x64位,visual studio 2010 professional
hadoop 源码版本2.2.0
步骤(来源于书《Pro Apache Hadoop, Second Edition》略作修改。
- Ensure that JDK 1.6 is or higher is installed. We assume that it is installed in thec:/MyApps/jdkl6/ folder, which should have a bin subfolder.
Download the hadoop-2.2.x-src.tar.gz files (2.2.0 at the time of this writing) from the download section of the Apache web site for Hadoop. The download link is as follows:http://www.apache.org/dist/hadoop/core/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz.
Explode the tar file into a directory. For the purpose of this section, we assume that the directory is calledc:/myapps/hadoop/.
If using Visual Studio, use Visual Studio 2010 Professional (not 2012). Do not use Visual Studio Express because it does not support compiling for 64-bit, which presents problems if running on a 64-bit system. Alternatively, download the Microsoft Windows SDK v7.1. At the time of writing, the link is the following:
http://www.microsoft.com/en-us/download/confirmation.aspx?id=8279.
Place the SDK in the folder with full privileges. For this section, we assume that the folder isc:/myapps/winsdk71/.
Download and install the Linux-like environment for Windows named Cygwin. We assume that Cygwin home folder isc:/myapps/cygwin/. Ensure that the following UNIX command-line tools are installed:sh, mkdir, rm, cp, tar, and gzip. The download location for Cygwin ishttp://cygwin.com/install.html.
Download and install Maven 3.1.1 or a compatible version. Installation consists of simply exploding thetar bundle into a Windows folder. We assume that the home folder for Maven download isc:/MyApps/apache-maven-3.1.1/. Verify that this folder has the bin subfolder.
Download and install Protocol Buffers 2.5.0. At the time of writing, the link is http://protobuf.googlecode.com/files/protoc-2.5.0-win32.zip.
Simply unzip the zip file into a folder (assumed to be c:/MyApps/protobuf/). Verify that the installation is correct by ensuring that the folder contains theprotoc.exe file.
Install ZLIB from http://zlib.net/zlib128-dll.zip and explode the zip file in the c:/zlib-1.2.8 folder. At runtime, this folder must be accessible from thePATH variable.
Add the environment variables shown in Table A-1.
Environment Variable | Value |
---|---|
JAVA_HOME | c:/MyApps/jdk16/ |
M2_HOME | c:/MyApps/apache-maven-3.1.1/ |
Platform | x64 (or Win32 when building on the 32-bit system) |
CYGWIN_HOME | c:/myapps/cygwin/ |
PROTOBUF_HOME | c:/myapps/protobuf/ |
ZLIB_HOME | c:/zlib-1.2.8 12 .Add the following to the Windows PATH variable:
|
开始编译,执行下面的步骤:
hadoop 2.2.0在编译的时候有一个bug,参考https://issues.apache.org/jira/browse/HADOOP-10110描述,然后下载https://issues.apache.org/jira/secure/attachment/12614482/HADOOP-10110.patch这个patch,按照这个patch修改hadoop-common-project/hadoop-auth/pom.xml文件,然后再进行编译,否则会出错。
Go to the Windows Start All Programs Microsoft Windows SDK v7.1 and start the Windows SDK 7.1 command prompt in Run As Administrator mode.
-
Change the folder to c:/MyApps/hadoop. Recall that this is the folder in which the Hadoop source was placed.
Execute the following command mvn package with the -Pdist,native-win -DskipTests -Dtar options. The exact command is this:
mvn package -Pdist,native-win -DskipTests –Dtar
If there are no errors, the hadoop-2.2.0.tar.gz file is created in the c:/myapps/hadoop/ hadoop-dist/hadoop-2.2.0/ folder.
编译完后,如果没有任何错误,在hadoop-2.2.0-src\hadoop-dist\target目录下面会生成hadoop-2.2.0.tar.gz文件。
第一步大功告成!
hadoop入门-在windows上编译x64位hadoop