首页 > 代码库 > hadoop入门-在windows上编译x64位hadoop

hadoop入门-在windows上编译x64位hadoop

环境

Windows 7 x64位,visual studio 2010 professional

hadoop 源码版本2.2.0

步骤(来源于书《Pro Apache Hadoop, Second Edition》略作修改。

  1. Ensure that JDK 1.6 is or higher is installed. We assume that it is installed in thec:/MyApps/jdkl6/ folder, which should have a bin subfolder.
  2. Add a note hereDownload the hadoop-2.2.x-src.tar.gz files (2.2.0 at the time of this writing) from the download section of the Apache web site for Hadoop. The download link is as follows:http://www.apache.org/dist/hadoop/core/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz.

  3. Add a note hereExplode the tar file into a directory. For the purpose of this section, we assume that the directory is calledc:/myapps/hadoop/.

  4. Add a note hereIf using Visual Studio, use Visual Studio 2010 Professional (not 2012). Do not use Visual Studio Express because it does not support compiling for 64-bit, which presents problems if running on a 64-bit system. Alternatively, download the Microsoft Windows SDK v7.1. At the time of writing, the link is the following:

    Add a note herehttp://www.microsoft.com/en-us/download/confirmation.aspx?id=8279.

  5. Add a note herePlace the SDK in the folder with full privileges. For this section, we assume that the folder isc:/myapps/winsdk71/.

  6. Add a note hereDownload and install the Linux-like environment for Windows named Cygwin. We assume that Cygwin home folder isc:/myapps/cygwin/. Ensure that the following UNIX command-line tools are installed:sh, mkdir, rm, cp, tar, and gzip. The download location for Cygwin ishttp://cygwin.com/install.html.

  7. Add a note hereDownload and install Maven 3.1.1 or a compatible version. Installation consists of simply exploding thetar bundle into a Windows folder. We assume that the home folder for Maven download isc:/MyApps/apache-maven-3.1.1/. Verify that this folder has the bin subfolder.

  8. Add a note hereDownload and install Protocol Buffers 2.5.0. At the time of writing, the link is http://protobuf.googlecode.com/files/protoc-2.5.0-win32.zip.

  9. Add a note hereSimply unzip the zip file into a folder (assumed to be c:/MyApps/protobuf/). Verify that the installation is correct by ensuring that the folder contains theprotoc.exe file.

  10. Add a note hereInstall ZLIB from http://zlib.net/zlib128-dll.zip and explode the zip file in the c:/zlib-1.2.8 folder. At runtime, this folder must be accessible from thePATH variable.

  11. Add a note hereAdd the environment variables shown in Table A-1.



Add a note hereTable A-1: Environment Variables for Windows

Add a note hereEnvironment Variable

Add a note hereValue

Add a note hereJAVA_HOME

Add a note herec:/MyApps/jdk16/

Add a note hereM2_HOME

Add a note herec:/MyApps/apache-maven-3.1.1/

Add a note herePlatform

Add a note herex64 (or Win32 when building on the 32-bit system)

Add a note hereCYGWIN_HOME

Add a note herec:/myapps/cygwin/

Add a note herePROTOBUF_HOME

Add a note herec:/myapps/protobuf/

Add a note hereZLIB_HOME

Add a note herec:/zlib-1.2.8


12 .Add the following to the Windows PATH variable:

  • Add a note here%JAVA_HOME%/bin/

  • Add a note here%CYGWIN_HOME%/bin/

  • Add a note here%M2_HOME%/bin/

  • Add a note here%PROTOBUF_HOME%

  • Add a note here%ZLIB_HOME

开始编译,执行下面的步骤:

hadoop 2.2.0在编译的时候有一个bug,参考https://issues.apache.org/jira/browse/HADOOP-10110描述,然后下载https://issues.apache.org/jira/secure/attachment/12614482/HADOOP-10110.patch这个patch,按照这个patch修改hadoop-common-project/hadoop-auth/pom.xml文件,然后再进行编译,否则会出错。

  1. Go to the Windows Start Image from book All Programs Image from book Microsoft Windows SDK v7.1 and start the Windows 
    SDK 7.1 command prompt in Run As Administrator mode.
    
  2. Add a note hereChange the folder to c:/MyApps/hadoop. Recall that this is the folder in which the Hadoop source was placed.

  3. Add a note hereExecute the following command mvn package with the -Pdist,native-win -DskipTests -Dtar options. The exact command is this:

    Add a note here
    mvn package -Pdist,native-win -DskipTests –Dtar
    
  4. Add a note hereIf there are no errors, the hadoop-2.2.0.tar.gz file is created in the c:/myapps/hadoop/ hadoop-dist/hadoop-2.2.0/ folder.

编译完后,如果没有任何错误,在hadoop-2.2.0-src\hadoop-dist\target目录下面会生成hadoop-2.2.0.tar.gz文件。

第一步大功告成!




hadoop入门-在windows上编译x64位hadoop