首页 > 代码库 > Azkaban 2.5.0 job type 插件安装

Azkaban 2.5.0 job type 插件安装

一、环境及软件

安装环境:

  • 安装目录: /usr/local/ae/ankaban
  • Hadoop 安装目录 export HADOOP_HOME=/usr/local/ae/hadoop-1.2.1
  • azkaban-executor-2.5.0安装目录:/usr/local/ae/azkaban/azkaban-executor-2.5.0

安装软件:

  • azkaban-jobtype-2.5.0.tar.gz

Azkaban jobtype plugin source:github.com/azkaban/azkaban-plugins/tree/master/plugins/jobtype

二、配置jobtype

  1. 解压 azkaban-jobtype-2.5.0.tar.gz 到 $AZKABAN-EXECUTOR_HOME/plugins/jobtypes 目录下
    user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0/plugins/jobtypes$ tar -zxvx azkaban--jobtype-2.5.0.tar.gz
  2. jobtype properties 配置
    common.properties
    ## everything that the user job can knowhadoop.home=/usr/local/ae/hadoop-1.2.1#hive.home=#pig.home=azkaban.should.proxy=falsejobtype.global.classpath=${hadoop.home}/hadoop-core-1.2.1.jar,${hadoop.home}/*,${hadoop.home}/conf/,${hadoop.home}/lib/*

    commonprivate.properties

    ## hadoop security manager setting common to all hadoop jobshadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_1_0## hadoop security related settings# proxy.keytab.location=YOUR_AZKABAN_KERBEROS_PRINCIPAL#  proxy.user=KEYTAB_LOCATIONazkaban.should.proxy=falseobtain.binary.token=false# obtain.namenode.token=true# obtain.jobtracker.token=true# global classpath items for all jobs. e.g. hadoop-core jar, hadoop conf#jobtype.global.classpath=${hadoop.home}/*,${hadoop.home}/conf# global jvm args for all jobs. e.g. java.io.temp.dir, java.library.path#jobtype.global.jvm.args=# hadoophadoop.home=/usr/local/ae/hadoop-1.2.1#pig.home=#hive.home=

三、配置 jobtype 插件安装目录到 Azkaban-executor

  1. 配置 azkaban.properties
    在 azkaban.properties 文件中添加以下配置
    # Azkaban JobTypes Pluginsazkaban.jobtype.plugin.dir=../plugins/jobtypes

四、重启 Azkaban-executor

user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0/bin$ sh azkaban-executor-shutdown.shuser@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0/bin$ sh azkaban-executor-start.sh

五、HadoopJava jobtype profile

type=hadoopJava#jvm.argsjob.class=×××××××××××××××××××××****main.args=×××××××××method.run=run#method.cancelclasspath=./lib/*,/usr/local/ae/hadoop-1.2.1/*,/usr/local/ae/hadoop-1.2.1/lib/*,/usr/local/ae/hbase-0.94.8/*,/usr/local/ae/hbase-0.94.8/lib/*#dependencies#getJobGeneratedProperties

Note:若 ${job.class} 继承自 AbstractHadoopJob 则需要设置 FileIutPutFormat 与 FileOutPutFormat