首页 > 代码库 > Job Submission failed with exception 'java.io.IOException

Job Submission failed with exception 'java.io.IOException

2024-07-24 09:01:22   218人阅读

在hive上执行如下sql：

select a.product_id, a.merchant_id, b.category_id 
from pm_info a join product b on (a.product_id=b.id) 
where a.is_deleted=0 and a.product_is_gift=0 and a.can_sale=1 and a.can_show=1 and b.is_deleted=0 and b.PRODUCT_IS_GIFT=0;

出现错误，错误如下：

java.io.IOException: Failed to run job : org.apache.hadoop.security.AccessControlException: User pms cannot submit applications to queue root.default
	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
	at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:448)
	at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.io.IOException(Failed to run job : org.apache.hadoop.security.AccessControlException: User pms cannot submit applications to queue root.default)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

原因分析：执行的SQL语句含有where条件，join连接，就会执行MapReduce任务去执行，1号店新集群的MapReduce含有两个队列，一个是root.default队列，一个是pms队列；如果不指定提交到哪个队列，hadoop就会将此Job提交到默认的root.default队列中，但是集群设定了job由MapReduce的pms队列受理，所以提交到root.default队列的任务通通被打回


解决方案（指定Job提交到哪个MapReduce的队列，在shell中执行）：

hive -e "set mapred.job.queue.name=pms;select a.product_id, a.merchant_id, b.category_id 
from pm_info a join product b on (a.product_id=b.id) 
where a.is_deleted=0 and a.product_is_gift=0 and a.can_sale=1 and a.can_show=1 and b.is_deleted=0 and b.PRODUCT_IS_GIFT=0;"

Job Submission failed with exception 'java.io.IOException


联系
我们