首页 > 代码库 > 在YARN中,如何控制和监控map/reduce的并发数
在YARN中,如何控制和监控map/reduce的并发数
配置建议:
1.
In MR1, the mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum properties dictated how many map and reduce slots each TaskTracker had.
These properties no longer exist in YARN. Instead, YARN uses yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores, which control the amount of memory and CPU on each node, both available to both maps and reduces
Essentially:
YARN has no TaskTrackers, but just generic NodeManagers. Hence, there‘s no more Map slots and Reduce slots separation. Everything depends on the amount of memory in use/demanded
2.
Using the web UI you can get lot of monitoring/admin kind of info:
NameNode - http://:50070/
Resource Manager - http://:8088/
其他配置参考:
- There is a good guide on YARN configuration from Hortonworks
- You may analyze your job in Job History server. It usually may be found on port 19888. Ambari andGanglia are also very good for cluster utilization measurement.