SMILA/Documentation/CrawlerController

Overview

The CrawlerController is a component that manages and monitors Crawlers.

API

/**
 * Management interface for the CrawlerController.
 */
public interface CrawlerController {
 
  /**
   * Starts a crawl of the given dataSourceId. This method creates a new Thread. If it is called for a dataSourceId that
   * is currently crawled a ConnectivityException is thrown. Returns the hashCode of the crawler instance used for
   * performance counter.
   * 
   * @param dataSourceId
   *          the ID of the data source to crawl
   * @return - the hashcode of the crawler instance as int value
   * @throws ConnectivityException
   *           if any error occurs
   */
  int startCrawl(String dataSourceId) throws ConnectivityException;
 
  /**
   * Stops an active crawl of the given dataSourceId.
   * 
   * @param dataSourceId
   *          the ID of the data source to stop the crawl
   * @throws ConnectivityException
   *           if any error occurs
   */
  void stopCrawl(String dataSourceId) throws ConnectivityException;
 
  /**
   * Checks if there are any active crawls.
   * 
   * @return true if there are active crawls, false otherwise
   * @throws ConnectivityException
   *           if any error occurs
   */
  boolean hasActiveCrawls() throws ConnectivityException;
 
  /**
   * Returns a Collection of Strings containing the dataSourceIds of the currently active crawls.
   * 
   * @return a Collection of Strings containing the dataSourceIds
   * @throws ConnectivityException
   *           if any error occurs
   */
  Collection<String> getActiveCrawls() throws ConnectivityException;
 
  /**
   * Returns the CrawlState of the crawl with the given dataSourceId.
   * 
   * @param dataSourceId
   *          the ID of the data source to get the state
   * @return the CrawlState
   * @throws ConnectivityException
   *           if any error occurs
   */
  CrawlState getState(String dataSourceId) throws ConnectivityException;
}

Implementations

It is possible to provide different implementations for the CrawlerController interface. At the moment there is one implementation available.

org.eclipse.smila.connectivity.framework.impl

This bundle contains the default implementation of the CrawlerController interface.

The CrawlerController implements the general processing logic common for all types of Crawlers. Its interface is a pure management interface that can be accessed by its Java interface or its wrapping JMX interface. It has references to the following OSGi services:

• Crawler ComponentFactory
• ConnectivityManager
• DeltaIndexingManager (optional)
• CompoundManager
• ConfigurationManagement (t.b.d.)

Crawler Factories register themselves at the CrawlerController. Each time a crawl for a certain type of crawler is initiated, a new instance of that Crawler type is created via the Crawler ComponentFactory. This allows parallel crawling of datasources with the same type (e.g. several websites). Note that it is not possible to crawl the same data source concurrently!

This chart shows the current CrawlerController processing logic for one crawl run:

• the CrawlerController checks if the Crawler has more data available
• true: the CrawlerController checks each received DataReference if it needs to be updated by creating a Record object containing only DeltaIndexing information and executing ConnectivityManager::checkForUpdate(Record[])
  • true: the CrawlerController request the complete record from the Crawler and executes ConnectivityManager::add(Record[])
  • false: the DataReference is skipped
• false: if no error occured so far it executes ConnectivityManager::finishDeltaIndexingByDataSourceId(String)

The processing logic will be enhanced as soon as CompoundManagement is available.

Configuration

There are no configuration options available for this bundle.

JMX interface

/**
 * The Interface CrawlerControllerAgent.
 */
public interface CrawlerControllerAgent {
 
  /**
   * Start crawl.
   * 
   * @param dataSourceId
   *          the data source id
   * 
   * @return the string
   */
  String startCrawl(final String dataSourceId);
 
  /**
   * Stop crawl.
   * 
   * @param dataSourceId
   *          the data source id
   * 
   * @return the string
   */
  String stopCrawl(final String dataSourceId);
 
  /**
   * Gets the status.
   * 
   * @param dataSourceId
   *          the data source id
   * 
   * @return the status
   */
  String getStatus(final String dataSourceId);
 
  /**
   * Gets the active crawls status.
   * 
   * @return the active crawls status
   */
  String getActiveCrawlsStatus();
 
  /**
   * Gets the active crawls.
   * 
   * @return the active crawls
   */
  String[] getActiveCrawls();
}

Here is a screenshot of the CrawlerController in the JMX Console: