首页 > 代码库 > Sqoop2远程调试

Sqoop2远程调试

Sqoop2是一个CS的架构,客户端包括sqoop-shell和sqoop-client,服务器端包括sqoop-server,sqoop-server就是一个部署在Tomcat下的web应用,由几个servlet组成。

调试sqoop-shell

调试sqoop-shell就是在启动sqoop client main方法时,加上调试参数。

sqoop-shell的启动方式为:sqoop.shclient,找到解决问题的起点,查找sqoop.sh脚本,在其中找到这样一段:

client)
    #Build class path with full path to each library
   for f in $CLIENT_LIB/*.jar; do
     CLASSPATH="${CLASSPATH}:${BASEDIR}/$f"
   done
    #We need to change current directory back to original as optional user sidescript
    #might be specified with relative path.
   cd ${OLD_DIR}
   EXEC_JAVA='java'
   if [ -n "${JAVA_HOME}" ] ; then
       EXEC_JAVA="${JAVA_HOME}/bin/java"
    fi
   ${EXEC_JAVA} -classpath ${CLASSPATH} org.apache.sqoop.shell.SqoopShell $2
   ;;
  *)
   echo "Command is not recognized."
;;

可以看到,${EXEC_JAVA} -classpath ${CLASSPATH} org.apache.sqoop.shell.SqoopShell $2,这行shell脚本就是启动sqoop客户端的main方法的地方了,在其中加入:

-Xdebug -Xnoagent-Xrunjdwp:transport=dt_socket,address=8199,server=y,suspend=n

这一行代码成了下面这样:

${EXEC_JAVA} -classpath ${CLASSPATH} -Xdebug-Xnoagent -Xrunjdwp:transport=dt_socket,address=8199,server=y,suspend=norg.apache.sqoop.shell.SqoopShell $2

注意,放到一行里,不要换行。Server不是指的IP。

参数具体的意义可以查看相关文档,IBM上有个系列的文章,是专门讲JAVA的调试体系的,叫做《深入 Java 调试体系深入 Java 调试体系深入JAVA调试体系》,讲的很细很好,相信对于很多人来说,会填补一块JAVA知识的空白。

这样就设置好了,当运行sqoop.sh client 启动sqoop shell的时候,会看到输出中包含下面的内容:

Listening for transport dt_socket ataddress: 8199

在IDE,比如Eclipse里,选择远程调试,在Eclipse中是Remote Java Application。主要是填程序所在机器的网络地址和端口号,在这个例子中,端口号就是8199。

Eclipse这端开始Debug之后,设置好断点,然后再Sqoop所在的机器启动的sqoopshell中进行操作,运行到断点,就会停住,在Eclipse这端跟调试本地程序一样。

调试sqoop-server

因为sqoop-server就是一个JavaWeb应用,所以设置sqoop-server远程调试,就是设置Tomcat为远程调试。运行Tomcat的catalina.sh命令可以看到,Tomcat已经为我们提供了jpda选项:

Usage: catalina.sh ( commands ... )
commands:
 debug             Start Catalinain a debugger
 debug -security   Debug Catalinawith a security manager
 jpda start        Start Catalinaunder JPDA debugger
 run               Start Catalinain the current window
  run-security     Start in the current windowwith security manager
  start             Start Catalina in a separatewindow
 start -security   Start in aseparate window with security manager
 stop              Stop Catalina,waiting up to 5 seconds for the process to end
 stop n            Stop Catalina,waiting up to n seconds for the process to end
 stop -force       Stop Catalina,wait up to 5 seconds and then use kill -KILL if still running
 stop n -force     Stop Catalina,wait up to n seconds and then use kill -KILL if still running
 version           What version oftomcat are you running?

由于这个内置的Tomcat的启动是由sqoop.sh脚本控制的,启动命令如下:

sqoop.sh server start
于是去查看sqoop.sh脚本的代码,找到如下部分:

   actionCmd=$2
 
   source ${BASEDIR}/bin/sqoop-sys.sh
   setup_catalina_opts
 
    #There seems to be a bug in catalina.sh whereby catalina.sh doesn't respect
    #CATALINA_OPTS when stopping the tomcat server. Consequently, we have to hackaround
    #by specifying the CATALINA_OPTS properties in JAVA_OPTS variable
   if [ "$actionCmd" == "stop" ]; then
     export JAVA_OPTS="$JAVA_OPTS $CATALINA_OPTS"
   fi
 
    #Remove the first 2 command line arguments (server and action command(start/stop)) so we can pass
    #the rest to catalina.sh script
   shift
   shift
 
   $CATALINA_BIN/catalina.sh $actionCmd "$@"
;;

从sqoop.sh server start命令来看,actionCmd就是start了,也就是说,sqoop.sh server start命令最终执行catalina.sh命令时是传入的start,即catalina.sh start。我们想运行:

catalina.sh jpda start

于是把原来$CATALINA_BIN/catalina.sh $actionCmd "$@"这行脚本的中直接加入jpda,最终的脚本为:

$CATALINA_BIN/catalina.sh jpda $actionCmd"$@"

上边我们设置sqoop shell 的jpda的时候,是在JVM启动时加入参数,但是从catalina.sh打印出的帮助信息来看,没有传入参数的地方,想到可能是在配置文件或者脚本中有环境变量:

最后在catalina.sh中找到下面的配置:

if [ "$1" = "jpda" ] ;then
  if[ -z "$JPDA_TRANSPORT" ]; then
   JPDA_TRANSPORT="dt_socket"
  fi
  if[ -z "$JPDA_ADDRESS" ]; then
   JPDA_ADDRESS="8000"
  fi
  if[ -z "$JPDA_SUSPEND" ]; then
   JPDA_SUSPEND="n"
  fi
  if[ -z "$JPDA_OPTS" ]; then
   JPDA_OPTS="-agentlib:jdwp=transport=$JPDA_TRANSPORT,address=$JPDA_ADDRESS,server=y,suspend=$JPDA_SUSPEND"
  fi
 CATALINA_OPTS="$CATALINA_OPTS $JPDA_OPTS"
 shift
fi

这是默认的配置,也可以自己更改。

 

完成上边的配置后,在Eclipse这端,调试sqoop-server项目,Remote Java Application,填上远程JVM地址,端口号,这个例子中是8000,点击debug就可以了。

 

Sqoop2远程调试