首页 > 代码库 > Spark分析之Master
Spark分析之Master
override def preStart() { logInfo("Starting Spark master at " + masterUrl) webUi.bind() //绑定WEBUI masterWebUiUrl = "http://" + masterPublicAddress + ":" + webUi.boundPort context.system.scheduler.schedule(0 millis, WORKER_TIMEOUT millis, self, CheckForWorkerTimeOut) //定时检查超时未发送心跳的Worker masterMetricsSystem.registerSource(masterSource) masterMetricsSystem.start() applicationMetricsSystem.start()
//用作master recover persistenceEngine = RECOVERY_MODE match { case "ZOOKEEPER" => logInfo("Persisting recovery state to ZooKeeper") new ZooKeeperPersistenceEngine(SerializationExtension(context.system), conf) case "FILESYSTEM" => logInfo("Persisting recovery state to directory: " + RECOVERY_DIR) new FileSystemPersistenceEngine(RECOVERY_DIR, SerializationExtension(context.system)) case _ => new BlackHolePersistenceEngine() }
//用作master的leader选举 leaderElectionAgent = RECOVERY_MODE match { case "ZOOKEEPER" => context.actorOf(Props(classOf[ZooKeeperLeaderElectionAgent], self, masterUrl, conf)) case _ => context.actorOf(Props(classOf[MonarchyLeaderAgent], self)) }}
override def receive = {
case RegisterWorker
case RequestSubmitDriver
case RegisterApplication
case Heartbeat
case ExecutorStateChanged
......
}
Master的主要功能总结:
1、Master的Leader选举;
2、对Driver、Worker、Application的管理
1)Driver:RequestSubmitDriver/RequestKillDriver/RequestDriverStatus/DriverStateChanged
2)Worker:RegisterWorker/Heartbeat/WorkerSchedulerStateResponse/CheckForWorkerTimeOut/ExecutorStateChanged
3)Application:RegisterApplication
声明:以上内容来自用户投稿及互联网公开渠道收集整理发布,本网站不拥有所有权,未作人工编辑处理,也不承担相关法律责任,若内容有误或涉及侵权可进行投诉: 投诉/举报 工作人员会在5个工作日内联系你,一经查实,本站将立刻删除涉嫌侵权内容。