企业搜索引擎开发之连接器connector（二十一）

首页 > 代码库 > 企业搜索引擎开发之连接器connector（二十一）

企业搜索引擎开发之连接器connector（二十一）

2024-07-07 13:11:00 223人阅读

从上文中的QueryTraverser对象的BatchResult runBatch(BatchSize batchSize)方法上溯到CancelableBatch类，该类实现了TimedCancelable接口，后者又extends了TimedCancelable接口，后者又extends了Cancelable接口，后者又extends了Runnable接口

Cancelable接口源码

/**
 * A {@link Runnable} that supports cancellation.
 */
public interface Cancelable extends Runnable {
  /**
   * Cancel the operation performed by this {@link Runnable}.
   * While this {@link Runnable#run} method is running in one thread this
   * may be called in another so implementors must provide any needed
   * synchronization.
   */
  public void cancel();
}

TimedCancelable接口源码

/**
 * A {@link Runnable} that supports cancellation and timeout.
 */
public interface TimedCancelable extends Cancelable {
  /**
   * Complete the operations performed by this {@link Runnable} due to the
   * expiration of its time interval. While this {@link Runnable#run} method is
   * running in one thread this may be called in another so implementors must
   * provide any needed synchronization.
   */
  public void timeout(TaskHandle taskHandle);
}

下面才是CancelableBatch类，实现了TimedCancelable接口

/**
 * A {@link TimedCancelable} for running a {@link Connector} batch using
 * a {@link Traverser}
 */
class CancelableBatch implements TimedCancelable {
  private static final Logger LOGGER =
    Logger.getLogger(CancelableBatch.class.getName());

  final Traverser traverser;
  final String traverserName;
  final BatchResultRecorder batchResultRecorder;
  final BatchTimeout batchTimeout;
  final BatchSize batchSize;

  /**
   * Construct a {@link CancelableBatch}.
   *
   * @param traverser {@link Traverser} for running the batch.
   * @param traverserName traverser name for logging purposes.
   * @param batchResultRecorder {@link BatchResultRecorder} for recording
   *        the result of running the batch.
   * @param batchSize hint and constraints as to the number of documents
   *        to process in the batch.
   */
  public CancelableBatch(Traverser traverser, String traverserName,
      BatchResultRecorder batchResultRecorder, BatchTimeout batchTimeout,
      BatchSize batchSize) {
    this.traverser = traverser;
    this.traverserName = traverserName;
    this.batchResultRecorder = batchResultRecorder;
    this.batchSize = batchSize;
    this.batchTimeout = batchTimeout;
  }
  
  /**
   * 取消执行
   */
  public void cancel() {
   traverser.cancelBatch();
  }
  
  /**
   * 运行超时
   */
  public void timeout(TaskHandle taskHandle) {
     batchTimeout.timeout();
  }

  public void run() {
    NDC.push("Traverse " + traverserName);
    try {
      LOGGER.fine("Begin runBatch; traverserName = " + traverserName
          + "  " + batchSize);
      BatchResult batchResult = traverser.runBatch(batchSize);
      LOGGER.fine("Traverser " + traverserName + " batchDone with result = "
          + batchResult);
      batchResultRecorder.recordResult(batchResult);
    } finally {
      NDC.remove();
    }
  }

  @Override
  public String toString() {
    return "CancelableBatch: traverser = " + traverser
           + ", batchSize = " + batchSize;
  }
}

在上面的run方法里面，调用了BatchResult batchResult = traverser.runBatch(batchSize);方法，用于获取数据源数据并向服务器推送数据

另外两方法注意下，后面会用到

/**
   * 取消执行
   */
  public void cancel() {
   traverser.cancelBatch();
  }
  
  /**
   * 运行超时
   */
  public void timeout(TaskHandle taskHandle) {
     batchTimeout.timeout();
  }

综上所述，CancelableBatch是一个实现了Runnable接口的线程类，姑且如是说

继续上溯到ConnectorCoordinatorImpl类，该类实现了ConnectorCoordinator接口，该接口声明了一个startBatch()方法

/**
   * Starts running a batch for this {@link ConnectorCoordinator} if a batch is
   * not already running.
   *
   * @return true if this call started a batch
   * @throws ConnectorNotFoundException if this {@link ConnectorCoordinator}
   *         does not exist.
   */
  public boolean startBatch() throws ConnectorNotFoundException;

首先需要明白的是，一个连接器实例对应一个ConnectorCoordinatorImpl实例对象，ConnectorCoordinatorImpl类实在庞大，我们先分析startBatch()方法源码

/**
   * 开始采集
   * Starts running a batch for this {@link ConnectorCoordinator} if a batch is
   * not already running.
   *
   * @return true if this call started a batch
   */
  /* @Override */
  public synchronized boolean startBatch() {
    if (!shouldRun()) {
      return false;
    }

    BatchSize batchSize = loadManager.determineBatchSize();
    if (batchSize.getHint() == 0) {
      return false;
    }

    try {
      TraversalManager traversalManager = getTraversalManager();
      if (traversalManager == null) {
        return false;
      }
      //当前标识
      currentBatchKey = new Object();
      
      BatchCoordinator batchCoordinator = new BatchCoordinator(this);
      
      //batchCoordinator 作为 TraversalStateStore stateStore角色
      Traverser traverser = new QueryTraverser(pusherFactory,
          traversalManager, batchCoordinator, name,
          Context.getInstance().getTraversalContext(), clock);
      
      //batchCoordinator 作为 BatchResultRecorder batchResultRecorder, BatchTimeout batchTimeout角色
      
      //调用Traverser traverser的取消方法   
      //BatchResultRecorder batchResultRecorder记录运行结果；[不能由外部调用]
      //BatchTimeout batchTimeout的超时方法
      TimedCancelable batch =  new CancelableBatch(traverser, name,
          batchCoordinator, batchCoordinator, batchSize);
      taskHandle = threadPool.submit(batch);
      //threadPool.shutdown(interrupt, waitMillis)
      //taskHandle.cancel();
      
      return true;
    } catch (ConnectorNotFoundException cnfe) {
      LOGGER.log(Level.WARNING, "Connector not found - this is normal if you "
          + " recently reconfigured your connector instance: " + cnfe);
    } catch (InstantiatorException ie) {
      LOGGER.log(Level.WARNING,
          "Failed to perform connector content traversal.", ie);
      delayTraversal(TraversalDelayPolicy.ERROR);
    }
    return false;
  }

我们可以看到，在该方法里面首先构造QueryTraverser对象(需要构造引用PusherFactory pusherFactory、TraversalManager traversalManager、TraversalStateStore stateStore实例)，然后构造CancelableBatch对象（构造函数传入QueryTraverser对象和BatchSize batchSize对象*批次尺寸），最后线程池对象提交CancelableBatch对象执行（到现在我们知道，一次线程执行只执行批次尺寸的数据采集，而并不一定是全部数据）

这里的BatchCoordinator batchCoordinator = new BatchCoordinator(this)对象在上面方法中充当不同的角色，即该对象实现了不同的接口

其构造函数传入了当前对象，即ConnectorCoordinatorImpl connectorCoordinator实例对象

在BatchCoordinator batchCoordinator对象实现不同接口的实现方法里面，基本上都是回调ConnectorCoordinatorImpl connectorCoordinator实例对象的方法，采用这种迂回的回马枪策略，大概是出于职责分明考虑吧，另外可能需要用到ConnectorCoordinatorImpl connectorCoordinator实例对象的状态

基本上BatchCoordinator batchCoordinator对象实现的方法在ConnectorCoordinatorImpl connectorCoordinator实例对象里面都要实现，这里设计方法采用的是一种包装器模式、或者是代理模式

可以猜想到，BatchCoordinator batchCoordinator对象实现了的接口实际上ConnectorCoordinatorImpl connectorCoordinator实例对象名义上甚至实际上也同样实现了（而类ConnectorCoordinatorImpl 实现的接口BatchCoordinator并不一定要实现）

我们先睹为快，不先做瞎猜了

BatchCoordinator类实现

class BatchCoordinator implements TraversalStateStore,
BatchResultRecorder, BatchTimeout

ConnectorCoordinatorImpl类实现

class ConnectorCoordinatorImpl implements
     ConnectorCoordinator, ChangeHandler, BatchResultRecorder

貌似BatchCoordinator类实现的接口之中，只有BatchResultRecorder接口ConnectorCoordinatorImpl类名义上也实现了

下面逐一分析

BatchCoordinator batchCoordinator = new BatchCoordinator(this);
      
      //batchCoordinator 作为 TraversalStateStore stateStore角色
      Traverser traverser = new QueryTraverser(pusherFactory,
          traversalManager, batchCoordinator, name,
          Context.getInstance().getTraversalContext(), clock);

这里batchCoordinator 作为 TraversalStateStore stateStore角色，实现raversalStateStore接口方法为：

 public String getTraversalState() {
    synchronized (connectorCoordinator) {
      if (connectorCoordinator.currentBatchKey == requiredBatchKey) {
        return cachedState;
      } else {
        throw new BatchCompletedException();
      }
    }
  }

  public void storeTraversalState(String state) {
    synchronized (connectorCoordinator) {
      // Make sure our batch is still valid and that nobody has modified
      // the checkpoint while we were away.
      try {
        if ((connectorCoordinator.currentBatchKey == requiredBatchKey) &&
            isCheckpointUnmodified()) {
          connectorCoordinator.setConnectorState(state);
          cachedState = state;
        } else {
          throw new BatchCompletedException();
        }
      } catch (ConnectorNotFoundException cnfe) {
        // Connector disappeared while we were away.
        // Don‘t try to store results.
        throw new BatchCompletedException();
      }
    }
  }

上面方法分别为获取断点状态和更新断点状态，需要考虑同步问题，两者都回调了ConnectorCoordinatorImpl connectorCoordinator对象的方法

获取状态

/**
   * Returns the Connector‘s traversal state.
   *
   * @return String representation of the stored state, or
   *         null if no state is stored.
   * @throws ConnectorNotFoundException if this {@link ConnectorCoordinator}
   *         does not exist.
   */
  /* @Override */
  public synchronized String getConnectorState()
      throws ConnectorNotFoundException {
    return getInstanceInfo().getConnectorState();
  }

更新状态

/**
   * Set the Connector‘s traversal state.
   *
   * @param state a String representation of the state to store.
   *        If null, any previous stored state is discarded.
   * @throws ConnectorNotFoundException if this {@link ConnectorCoordinator}
   *         does not exist.
   */
  /* @Override */
  public synchronized void setConnectorState(String state)
      throws ConnectorNotFoundException {
    getInstanceInfo().setConnectorState(state);
    // Must not call ChangeDetector, as this is called from a synchronized
    // block in BatchCoordinator.
  }

接下来分析BatchCoordinator batchCoordinator对象充当的其他角色

 //batchCoordinator 作为 BatchResultRecorder batchResultRecorder, BatchTimeout batchTimeout角色
      
      //调用Traverser traverser的取消方法   
      //BatchResultRecorder batchResultRecorder记录运行结果；[不能由外部调用]
      //BatchTimeout batchTimeout的超时方法
      TimedCancelable batch =  new CancelableBatch(traverser, name,
          batchCoordinator, batchCoordinator, batchSize);

前者作为BatchResultRecorder batchResultRecorder角色，后者作为BatchTimeout batchTimeout角色

实现BatchResultRecorder接口的方法为

public void recordResult(BatchResult result) {
    synchronized (connectorCoordinator) {
      if (connectorCoordinator.currentBatchKey == requiredBatchKey) {
        connectorCoordinator.recordResult(result);
      } else {
        LOGGER.fine("Ignoring a BatchResult returned from a "
            + "prevously canceled traversal batch.  Connector = "
            + connectorCoordinator.getConnectorName()
            + "  result = " + result + "  batchKey = " + requiredBatchKey);
      }
    }
  }

里面进一步回调了ConnectorCoordinatorImpl connectorCoordinator对象如下方法

/**
   * Records the supplied traversal batch results.  Updates the
   * {@link LoadManager} with number of documents traversed,
   * and implements the requested {@link TraversalDelayPolicy}.
   *
   * @param result a BatchResult
   */
  /* @Override */
  public synchronized void recordResult(BatchResult result) {
    loadManager.recordResult(result);
    delayTraversal(result.getDelayPolicy());
  }

记录执行结果及决定延迟策略

这里调用的方法名一致，我们再前面已经看到，两者都实现了BatchResultRecorder接口

实现BatchTimeout接口方法为

public void timeout() {
    synchronized (connectorCoordinator) {
      if (connectorCoordinator.currentBatchKey == requiredBatchKey) {
        connectorCoordinator.resetBatch();
      } else {
        LOGGER.warning("Ignoring Timeout for previously prevously canceled"
            + " or completed traversal batch.  Connector = "
            + connectorCoordinator.getConnectorName()
            + "  batchKey = "+ requiredBatchKey);
      }
    }
  }

回调ConnectorCoordinatorImpl connectorCoordinator对象如下方法

/**
   * 取消采集
   * Halts any in-progess traversals for this {@link Connector} instance.
   * Some or all of the information collected during the current traversal
   * may be discarded.
   */
  synchronized void resetBatch() {
    if (taskHandle != null) {
      taskHandle.cancel();
    }
    taskHandle = null;
    currentBatchKey = null;
    interfaces = null;

    // Discard cached interface instances.
    traversalManager = null;
    retriever = null;
    traversalSchedule = null;
  }

前两者分别用于状态管理和记录执行结果及延迟策略，第三者用于取消采集（里面调用了taskHandle.cancel()方法）

执行序列为

TimedCancelable类型对象（CancelableBatch对象）的timeout(TaskHandle taskHandle)方法-->

BatchTimeout类型对象(即BatchCoordinator batchCoordinator)的batchTimeout.timeout()方法-->

ConnectorCoordinatorImpl connectorCoordinator对象的resetBatch()方法-->

TaskHandle taskHandle的taskHandle.cancel()方法-->Cancelable类型对象（CancelableBatch对象）的cancel()方法-->Traverser traverser的 cancelBatch()方法

即CancelableBatch对象的timeout(TaskHandle taskHandle)方法绕来绕去最终接上了自己的cancel()方法

从下文中我们可以看到，这样处理的目的在于当一个线程超时时，由另一个监控超时的线程执行取消操作；在正常情况下，该执行序列将不会发生

TaskHandle taskHandle是一个任务执行句柄，用于对执行过程进行操控

/**
 * Handle for the management of a {@link Cancelable} primary task.
 */
public class TaskHandle {
  /**
   * The primary {@link Cancelable} that is run by this task to
   * perform some useful work.
   */
  final Cancelable cancelable;

  /*
   * The {@link future} for the primary task.
   */
  final Future<?> taskFuture;

  /*
   * The time the task starts.
   */
  final long startTime;

  /**
   * Create a TaskHandle.
   *
   * @param cancelable {@link Cancelable} for the primary task.
   * @param taskFuture {@link Future} for the primary task.
   * @param startTime startTime for the primary task.
   */
  TaskHandle(Cancelable cancelable, Future<?> taskFuture, long startTime) {
    this.cancelable = cancelable;
    this.taskFuture = taskFuture;
    this.startTime = startTime;
  }

  /**
   * Cancel the primary task and the time out task.
   */
  public void cancel() {
    cancelable.cancel();
    taskFuture.cancel(true);
  }

  /**
   * Return true if the primary task has completed.
   */
  public boolean isDone() {
    return taskFuture.isDone();
  }
}

---------------------------------------------------------------------------

本系列企业搜索引擎开发之连接器connector系本人原创

转载请注明出处博客园刺猬的温驯

本人邮箱： chenying998179@163#com （#改为.）

本文链接 http://www.cnblogs.com/chenying99/p/3775591.html

声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉：投诉/举报工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。

联系
我们

首页 > 代码库 > 企业搜索引擎开发之连接器connector（二十一）

企业搜索引擎开发之连接器connector（二十一）

看完仍有疑问？有类似问题直接问程序猿