首页 > 代码库 > <Dr.Elephant>
<Dr.Elephant>
Why Dr.Elephant?
- Most of Hadoop optimization tools out there, but they are focused on simplifying the deploy and managment of Hadoop clusters.
- Very few tools are designed to help Hadoop users optimize their flows.
- Dr.Elephant supports Hadoop with a variety of frameworks and can be easily extended to newer frameworks.
- You can plugin and configure as many custom heuristics as you like.
- It is designed to help the users of Hadoop and Spark understand the internals of their flow and to help them tune their jobs easily.
Key Features
- Pluggable and configurable rule-based heuristics that diagnose a job;
- Out-of-the-box integration with Azkaban scheduler and support for adding any other Hadoop scheduler, such as Oozie;
- Representation of historic performance of jobs and flows;
- Job-level comparison of flows;
- Diagnostic heuristics for MapReduce and Spark;
- Easily extensible to newer job types, applications, and schedulers;
- REST API to fetch all the information.
How does it work?
- Dr. Elephant gets a list of all recent succeeded and failed applications, at regular intervals, from the YARN resource manager.
- The metadata for each application—namely, the job counters, configurations, and the task data—are fetched from the Job History server.
- Dr. Elephant runs a set of heuristics on them and generates a diagnostic report on how the individual heuristics and the job as a whole performed.
- These are then tagged with one of five severity levels, to indicate potential performance problems.
Sample Usage
- Once a job completes, it can be found in the Dashboard.
- The color Red means the job is in critical state and requires tuning while Green means the job is running efficiently. As follow
- And u can click into the app to get the complete report, including details on each of the individual heuristics and a link, [Explain], which provides suggestions on how to tune the job to improve that heuristic.
Getting Started
User Guide
- TBD...
FYI
- Linkedin Dr.Elephant
<Dr.Elephant>
声明:以上内容来自用户投稿及互联网公开渠道收集整理发布,本网站不拥有所有权,未作人工编辑处理,也不承担相关法律责任,若内容有误或涉及侵权可进行投诉: 投诉/举报 工作人员会在5个工作日内联系你,一经查实,本站将立刻删除涉嫌侵权内容。