首页 > 代码库 > 运行时候报异常could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation.

运行时候报异常could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation.

运行时候报异常could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and no node(s) are excluded in this operation.

解决方法:

1首先看一下dfs.replication的数目是否超过了datanode的数目,应该要小于或者等于datanode的数目。

2更改mapreduce。map.memory.mb的大小,把每个map任务的物理限制提高,代码如下,reduce同理。

<property> 
<name>mapreduce.map.memory.mb</name> 
<value>1024</value> 
<description>每个Map任务的物理内存限制</description> 
</property> 

yarn.scheduler.minimum-allocation-mb/yarn.scheduler.maximum-allocation-mb

参数解释:单个可申请的最小/最大内存资源量。比如设置为1024和3072,则运行MapRedce作业时,每个Task最少可申请1024MB内存,最多可申请3072MB内存。

默认值:1024/8192

yarn.scheduler.minimum-allocation-vcores/yarn.scheduler.maximum-allocation-vcores

参数解释:单个可申请的最小/最大虚拟CPU个数。比如设置为1和4,则运行MapRedce作业时,每个Task最少可申请1个虚拟CPU,最多可申请4个虚拟CPU。什么是虚拟CPU,可阅读我的这篇文章:“YARN 资源调度器剖析”。

默认值:1/32

运行时候报异常could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation.