Difference between revisions of "SMILA/Documentation/Crawler"

From Eclipsepedia

Jump to: navigation, search
m (Configuration)
 
(9 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
{{note|This is deprecated for SMILA 1.0, the connectivity framework is still functional but will aimed to be replaced by scalable import based on SMILAs job management.}}
 +
 
[[Image:CrawlerWorkflow.png|thumb|Crawler Wokflow]]
 
[[Image:CrawlerWorkflow.png|thumb|Crawler Wokflow]]
  
 
== Overview ==
 
== Overview ==
  
A Crawler gathers information about resources, both content and metadata of interest like size or mime type. SMILA currently comes with three types of crawlers, each for a different data source type, namely WebCrawler, JDBC DatabaseCrawler, and FileSystemCrawler to facilitate gathering information from the internet, databases, or files from a hard disk. Furthermore, the Connectivity Framework provides an API for developers to create own crawlers.
+
A crawler gathers information about resources, both content and metadata of interest like size or MIME type. SMILA currently comes with three types of crawlers, each adequate for a different datasource type, namely Web crawler, JDBC Database crawler, and File System crawler, to allow gathering information from the internet, databases, or files from a hard disk. Furthermore, the Connectivity Framework provides an API for developers which allows them to create their own crawlers.
 
+
  
 
== API ==
 
== API ==
  
A Crawler has to implement two interfaces: <tt>Crawler</tt> and <tt>CrawlerCallback</tt>. The easiest way to achieve this is to extend the abstract base class <tt>AbstractCrawler</tt> located in bundle <tt>org.eclipse.smila.connectivity.framework</tt>. This class already contains handling for the crawlers Id and an OSGI service activate method. The Crawler method <tt>getNext()</tt> is designed to support an array of Datareference objects, as this reduces the number of method calls. In general there are no restrictions on the size of the array, in fact the size could vary on multiple method calls. This allows a Crawler to internally implement a Producer/Consumer pattern. A Crawler implementation that is restricted to work as an iterator only can also enforce this by always returning an array of size one.
+
A crawler has to implement two interfaces: <tt>Crawler</tt> and <tt>CrawlerCallback</tt>. The easiest way to achieve this is to extend the abstract base class <tt>AbstractCrawler</tt> located in bundle <tt>org.eclipse.smila.connectivity.framework</tt>. This class already contains handling for the crawlers Id and an OSGI service activate method. The crawler  method <tt>getNext()</tt> is designed to support an array of Datareference objects, as this reduces the number of method calls. In general there are no restrictions on the size of the array, in fact the size could vary on multiple method calls. This allows a crawler to internally implement a Producer/Consumer pattern. A <tt>Crawler</tt> implementation that is restricted to work as an iterator only can also enforce this by always returning an array of size one.
 
+
<source lang="java">
+
public interface Crawler {
+
 
+
  /**
+
  * Returns the ID of this Crawler.
+
  *
+
  * @return a String containing the ID of this Crawler
+
  *
+
  * @throws CrawlerException
+
  *          if any error occurs
+
  */
+
  String getCrawlerId() throws CrawlerException;
+
 
+
  /**
+
  * Returns an array of DataReference objects. The size of the returned array may vary from call to call. The maximum
+
  * size of the array is determined by configuration or by the implementation class.
+
  *
+
  * @return an array of DataReference objects or null, if no more DataReference exist
+
  *
+
  * @throws CrawlerException
+
  *          if any error occurs
+
  * @throws CrawlerCriticalException
+
  *          the crawler critical exception
+
  */
+
  DataReference[] getNext() throws CrawlerException, CrawlerCriticalException;
+
 
+
  /**
+
  * Initialize.
+
  *
+
  * @param config
+
  *          the DataSourceConnectionConfig
+
  *
+
  * @throws CrawlerException
+
  *          the crawler exception
+
  * @throws CrawlerCriticalException
+
  *          the crawler critical exception
+
  */
+
  void initialize(DataSourceConnectionConfig config) throws CrawlerException, CrawlerCriticalException;
+
 
+
  /**
+
  * Ends crawl, allowing the Crawler implementation to close any open resources.
+
  *
+
  * @throws CrawlerException
+
  *          if any error occurs
+
  */
+
  void close() throws CrawlerException;
+
 
+
}
+
</source>
+
 
+
 
+
<source lang="java">
+
/**
+
* A callback interface to access metadata and attachments of crawled data.
+
*/
+
public interface CrawlerCallback {
+
  /**
+
  * Returns the MObject for the given id.
+
  *
+
  * @param id
+
  *          the record id
+
  * @return the MObject
+
  * @throws CrawlerException
+
  *          if any non critical error occurs
+
  * @throws CrawlerCriticalException
+
  *          if any critical error occurs         
+
  */
+
  MObject getMObject(Id id) throws CrawlerException, CrawlerCriticalException;
+
 
+
  /**
+
  * Returns an array of String[] containing the names of the available attachments for the given id.
+
  *
+
  * @param id
+
  *          the record id
+
  * @return an array of String[] containing the names of the available attachments
+
  * @throws CrawlerException
+
  *          if any non critical error occurs
+
  * @throws CrawlerCriticalException
+
  *          if any critical error occurs 
+
  */
+
  String[] getAttachmentNames(Id id) throws CrawlerException, CrawlerCriticalException;
+
 
+
  /**
+
  * Returns the attachment for the given Id and name pair.
+
  *
+
  * @param id
+
  *          the record id
+
  * @param name
+
  *          the name of the attachment
+
  * @return a byte[] containing the attachment
+
  * @throws CrawlerException
+
  *          if any non critical error occurs
+
  * @throws CrawlerCriticalException
+
  *          if any critical error occurs 
+
  */
+
  byte[] getAttachment(Id id, String name) throws CrawlerException, CrawlerCriticalException;
+
 
+
  /**
+
  * Disposes the record with the given Id.
+
  *
+
  * @param id
+
  *          the record id
+
  */
+
  void dispose(Id id);
+
}
+
</source>
+
 
+
For completeness, here is the interface DataReference:
+
<source lang="java">
+
/**
+
* A proxy object interface to a record provided by a Crawler. The object contains the Id and the hash value of the
+
* record but no additional data. The complete record can be loaded via the CrawlerCallback.
+
*/
+
public interface DataReference {
+
  /**
+
  * Returns the Id of the referenced record.
+
  *
+
  * @return the Id of the referenced record
+
  */
+
  Id getId();
+
 
+
  /**
+
  * Returns the hash of the referenced record as a String.
+
  *
+
  * @return the hash of the referenced record as a String
+
  */
+
  String getHash();
+
 
+
  /**
+
  * Returns the complete Record object via the CrawlerCallback.
+
  *
+
  * @return the complete record
+
  * @throws CrawlerException
+
  *          if any non critical error occurs
+
  * @throws CrawlerCriticalException
+
  *          if any critical error occurs
+
  * @throws InvalidTypeException
+
  *          if the hash attribute cannot be set
+
  */
+
  Record getRecord() throws CrawlerException, CrawlerCriticalException, InvalidTypeException;
+
 
+
  /**
+
  * Disposes the referenced record object.
+
  */
+
  void dispose();
+
}
+
</source>
+
  
 +
Javadoc: [http://build.eclipse.org/rt/smila/javadoc/current/org/eclipse/smila/connectivity/framework/Crawler.html org.eclipse.smila.connectivity.framework.Crawler]
  
 
== Architecture ==
 
== Architecture ==
  
Crawlers are managed and instantiated by the CrawlerController. The CrawlerController communicates with the Crawler via interface <tt>Crawler</tt>, only. The Crawler's <tt>getNext()</tt> method returns <tt>DataReference</tt> objects to the CrawlerController. <tt>DataReference</tt> is also an interface implemented by class <tt>org.eclipse.smila.connectivity.framework.util.internal.DataReferenceImpl</tt>. A DataReference, as the name suggests, is only a reference to data provided by the Crawler. This is mainly a performance issue, as due to the use of DeltaIndexing it may not be neccessary to transfer all the data from the Crawler to the CrawlerController and to ConnectivityManager. Therefore a DataReference contains only the minumum data needed to perform DeltaIndexing: an Id and a hash token. To access the whole object it provideds method <tt>getRecord()</tt> that returns a complete Record object containing Id, attributes, annotations and attachments. To create the Record object, the DataReference communicates with the Crawler via interface <tt>CrawlerCallback</tt>, as each DataReference has a reference to the Crawler that created it.
+
Crawlers are managed and instantiated by the CrawlerController. The CrawlerController communicates with the crawler via interface <tt>Crawler</tt>, only. The crawler's <tt>getNext()</tt> method returns <tt>DataReference</tt> objects to the CrawlerController. <tt>DataReference</tt> is also an interface implemented by class <tt>org.eclipse.smila.connectivity.framework.util.internal.DataReferenceImpl</tt>. A DataReference, as the name suggests, is only a reference to data provided by the crawler. This is mainly a performance issue, as due to the use of DeltaIndexing it may not be neccessary to transfer all the data from the crawler to the CrawlerController and to ConnectivityManager. Therefore a DataReference contains only the minumum data needed to perform DeltaIndexing: an Id and a hash token. To access the whole object it provideds method <tt>getRecord()</tt> that returns a complete Record object containing Id, attributes, annotations and attachments. To create the Record object, the DataReference communicates with the crawler via interface <tt>CrawlerCallback</tt>, as each DataReference has a reference to the crawler that created it.
  
The following chart shows the Crawler Architecture and how data is shared with the CrawlerController:
+
The following chart shows the crawler architecture and how data is shared with the CrawlerController:
 
[[Image:Crawler Architecture.png|Crawler Architecture]]
 
[[Image:Crawler Architecture.png|Crawler Architecture]]
  
Package <tt>org.eclipse.smila.connectivity.framework.util</tt> provides some factory classes for crawlers to create Ids, hashes and DataReference objects. More utility classes are planned to be implemented, that allow easy realisation of Crawlers using an Iterator or Producer/Consumer pattern.
+
Package <tt>org.eclipse.smila.connectivity.framework.util</tt> provides some factory classes for crawlers to create Ids, hashes and DataReference objects. More utility classes are planned to be implemented, that allow easy realization of crawlers using an iterator or producer/consumer pattern.
  
 
== Configuration ==
 
== Configuration ==
  
A Crawler is started with a specific, named configuration, that defines what information is to be crawled (e.g. content, kinds of metadata) and where to find that data (e.g. file system path, JDBC Connection String). See each Crawler documentation for details on configuration options.
+
A crawler is started with a specific, named configuration, that defines what information is to be crawled (e.g. content, kinds of metadata) and where to find that data (e.g. file system path, JDBC Connection String). See each crawler documentation for details on configuration options.
  
  
Each crawler can define its own configuration because crawlers need different information to execute specifc crawl jobs. As example a JDBC-Crawler need information about which database and which table should be crawled and which columns should be returned.
+
Each crawler can define its own configuration because crawlers need different information to execute specifc crawl jobs. As example a JDBC crawler needs information about which database and which table should be crawled and which columns should be returned.
  
 
Therefore the crawler developer defines a schema that contains all interesting information. This schema is based on a root schema that is supported by the SMILA framework. It declares the generic framework/frame which has to be used to send DataSourceConnectionConfigs (a crawl task) to the SMILA framework.
 
Therefore the crawler developer defines a schema that contains all interesting information. This schema is based on a root schema that is supported by the SMILA framework. It declares the generic framework/frame which has to be used to send DataSourceConnectionConfigs (a crawl task) to the SMILA framework.
Line 184: Line 38:
  
 
;DataSourceID
 
;DataSourceID
:a description string that is used in the whole framework to separate and address information that apply to the same crawl job
+
:A description string that is used in the whole framework to separate and address information that apply to the same crawl job
 +
 
 
;SchemaID
 
;SchemaID
:The SchemaID contains the whole bundle name of the crawler (e.g. FileSystem-Crawler: org.eclipse.smila.connectivity.framework.crawler.filesystem).<br /> The SMILA Framework uses this information to gather the schema for the validation of the  DataSourceConnectionConfig that should be executed.
+
:The SchemaID contains the whole bundle name of the crawler (e.g. File System crawler: org.eclipse.smila.connectivity.framework.crawler.filesystem).<br /> The SMILA Framework uses this information to gather the schema for the validation of the  DataSourceConnectionConfig that should be executed.
 +
 
 
;DataConnectionID
 
;DataConnectionID
;:Crawler
+
:This tag describes if an agent or crawler should be used. It contains either of the following tags:
: This tags describes which crawler should be used for the crawl job. The name that is used in this tag is the Service name of the crawler.
+
:*<b>Agent</b>
;:Agent
+
:*<b>Crawler</b>
:placeholder for upcoming implementations
+
:The name that is used in these tags is the Service name of the agent/crawler.  
 +
 
 +
;RecordBuffer
 +
:Here you can specify settings to optimize record transfer to ConnectivityManager
 +
:*Size - the number of records to be send to ConnectivityyManager in one block. Default is 1.
 +
:*FlushInterval - a time interval in milliseconds after which to send the current elements of the RecordBuffer to ConnectivityManager. Default is 1000.
 +
 
 +
;DeltaIndexing:
 +
:Configuration options for delta indexing that are to be interpreted by the CrawlerController. The following values are supported:
 +
:*<tt>full</tt> - delta indexing is fully activated. Records are checked if they need to be updated, entries for new/updated records are added to the deltaIndexingManager, delta-delete is executed if no error occured
 +
:*<tt>additive</tt> - as <tt>full</tt> but delta-delete is not executed
 +
:*<tt>initial</tt> - For an initial import in an empty index or a new source in an existing index performance can be optimized by NOT checking if a record needs to be updated (we know that all records are new) but adding an entry in the DeltaIndexingManager for each Record. This allows later runs using <tt>full</tt> or <tt>additive</tt> to make use of DeltaIndexing infformation.
 +
:*<tt>disabled</tt> - delta indexing is fully disabled. No checks are done, no entries are created/updated, no Delta-Delete is executed. Later runs cannot benefit from DeltaIndexing
 +
 
 
;CompoundHandling:
 
;CompoundHandling:
:placeholder for upcoming implementations
+
:Configuration options for CompoundHandling. See [[SMILA/Documentation/CompoundManagement#Configuration|CompoundManagement]] for details.
  
 
;Attributes
 
;Attributes
:placeholder for each Crawler. <br />Each Crawler can define here which Attributes it can return. An attribute is a specific information of an entry in the data source that is crawled by the crawler (E.g. In a filesystem an entry is a file, and attributes of an file are Size, Content, etc.)
+
:Placeholder for each crawler's attribute definition. <br />Each crawler can define here which attributes it can return. An attribute is a specific information of an entry in the datasource that is crawled by the crawler (E.g. In a filesystem an entry is a file, and attributes of an file are Size, Content, etc.)
  
 
;Process
 
;Process
:placeholder for Tags that the Crawler developer can define. <br /> In this Tag all information can be transferred for a crawl task that are necessary to start a crawling process. These information are maybe: starting urls/folder, and which entries should be crawled ( e.g. queries/wildcards/include/excludes).
+
:Placeholder for Tags that the crawler developer can define. <br /> In this Tag all information can be transferred for a crawl task that are necessary to start a crawling process. These information are maybe: starting urls/folder, and which entries should be crawled ( e.g. queries/wildcards/include/excludes).
  
  
 
=== Further Information: ===
 
=== Further Information: ===
# see for each Crawler Attributes and Process Tags
+
# See for each crawler attributes and process tags
# [[SMILA/Development Guidelines/How to implement a crawler|How to implement a Crawler]]
+
# [[SMILA/Development Guidelines/How to implement a crawler|How to implement a crawler]]
  
 
== Crawler lifecycle ==
 
== Crawler lifecycle ==
  
The CrawlerController manages the life cycle of the crawler (e.g. start, stop, abort) and may instantiate multiple Crawlers concurrently, even of the same type. This is realised by using OSGi ComponentFactories. Each Crawler doea not automatically start an OSGi service, but registers only a Crawler ComponentFactory with the CrawlerController. Via the ComponmentFactory the CrawlerController can instantiate Crawlers on demand.
+
The CrawlerController manages the life cycle of the crawler (e.g. start, stop, abort) and may instantiate multiple crawlers concurrently, even of the same type. This is realised by using OSGi ComponentFactories. Each crawler does not automatically start an OSGi service, but registers only a crawler ComponentFactory with the CrawlerController. Via the ComponentFactory the CrawlerController can instantiate crawlers on demand.
  
Here is a template for a Crawler OSGi component definition
+
Here is a template for a crawler OSGi component definition:
 
<source lang="xml">
 
<source lang="xml">
 
<component name="%CRAWLER_TYPE%" immediate="false" factory="CrawlerFactory">
 
<component name="%CRAWLER_TYPE%" immediate="false" factory="CrawlerFactory">
Line 222: Line 91:
 
== See also ==
 
== See also ==
  
More information about the different Crawlers can be found here:
+
More information about the different crawlers can be found here:
* [[SMILA/Documentation/Filesystem Crawler|Filesystem Crawler]]
+
* [[SMILA/Documentation/Filesystem Crawler|File System crawler]]
* [[SMILA/Documentation/Web Crawler|Web Crawler]]
+
* [[SMILA/Documentation/Web Crawler|Web crawler]]
* [[SMILA/Documentation/JDBC Crawler|JDBC Crawler]]
+
* [[SMILA/Documentation/JDBC Crawler|JDBC crawler]]
  
 
[[Category:SMILA]]
 
[[Category:SMILA]]

Latest revision as of 05:38, 24 January 2012

Note.png
This is deprecated for SMILA 1.0, the connectivity framework is still functional but will aimed to be replaced by scalable import based on SMILAs job management.


Crawler Wokflow

Contents

[edit] Overview

A crawler gathers information about resources, both content and metadata of interest like size or MIME type. SMILA currently comes with three types of crawlers, each adequate for a different datasource type, namely Web crawler, JDBC Database crawler, and File System crawler, to allow gathering information from the internet, databases, or files from a hard disk. Furthermore, the Connectivity Framework provides an API for developers which allows them to create their own crawlers.

[edit] API

A crawler has to implement two interfaces: Crawler and CrawlerCallback. The easiest way to achieve this is to extend the abstract base class AbstractCrawler located in bundle org.eclipse.smila.connectivity.framework. This class already contains handling for the crawlers Id and an OSGI service activate method. The crawler method getNext() is designed to support an array of Datareference objects, as this reduces the number of method calls. In general there are no restrictions on the size of the array, in fact the size could vary on multiple method calls. This allows a crawler to internally implement a Producer/Consumer pattern. A Crawler implementation that is restricted to work as an iterator only can also enforce this by always returning an array of size one.

Javadoc: org.eclipse.smila.connectivity.framework.Crawler

[edit] Architecture

Crawlers are managed and instantiated by the CrawlerController. The CrawlerController communicates with the crawler via interface Crawler, only. The crawler's getNext() method returns DataReference objects to the CrawlerController. DataReference is also an interface implemented by class org.eclipse.smila.connectivity.framework.util.internal.DataReferenceImpl. A DataReference, as the name suggests, is only a reference to data provided by the crawler. This is mainly a performance issue, as due to the use of DeltaIndexing it may not be neccessary to transfer all the data from the crawler to the CrawlerController and to ConnectivityManager. Therefore a DataReference contains only the minumum data needed to perform DeltaIndexing: an Id and a hash token. To access the whole object it provideds method getRecord() that returns a complete Record object containing Id, attributes, annotations and attachments. To create the Record object, the DataReference communicates with the crawler via interface CrawlerCallback, as each DataReference has a reference to the crawler that created it.

The following chart shows the crawler architecture and how data is shared with the CrawlerController: Crawler Architecture

Package org.eclipse.smila.connectivity.framework.util provides some factory classes for crawlers to create Ids, hashes and DataReference objects. More utility classes are planned to be implemented, that allow easy realization of crawlers using an iterator or producer/consumer pattern.

[edit] Configuration

A crawler is started with a specific, named configuration, that defines what information is to be crawled (e.g. content, kinds of metadata) and where to find that data (e.g. file system path, JDBC Connection String). See each crawler documentation for details on configuration options.


Each crawler can define its own configuration because crawlers need different information to execute specifc crawl jobs. As example a JDBC crawler needs information about which database and which table should be crawled and which columns should be returned.

Therefore the crawler developer defines a schema that contains all interesting information. This schema is based on a root schema that is supported by the SMILA framework. It declares the generic framework/frame which has to be used to send DataSourceConnectionConfigs (a crawl task) to the SMILA framework. The root-schema can be found in: configuration\org.eclipse.smila.connectivity.framework.schema/schemas/RootDataSourceConnectionConfigSchema.xsd.

The root schema looks like as follows:

RootdatasourceConnectionConfig.png

DataSourceID
A description string that is used in the whole framework to separate and address information that apply to the same crawl job
SchemaID
The SchemaID contains the whole bundle name of the crawler (e.g. File System crawler: org.eclipse.smila.connectivity.framework.crawler.filesystem).
The SMILA Framework uses this information to gather the schema for the validation of the DataSourceConnectionConfig that should be executed.
DataConnectionID
This tag describes if an agent or crawler should be used. It contains either of the following tags:
  • Agent
  • Crawler
The name that is used in these tags is the Service name of the agent/crawler.
RecordBuffer
Here you can specify settings to optimize record transfer to ConnectivityManager
  • Size - the number of records to be send to ConnectivityyManager in one block. Default is 1.
  • FlushInterval - a time interval in milliseconds after which to send the current elements of the RecordBuffer to ConnectivityManager. Default is 1000.
DeltaIndexing
Configuration options for delta indexing that are to be interpreted by the CrawlerController. The following values are supported:
  • full - delta indexing is fully activated. Records are checked if they need to be updated, entries for new/updated records are added to the deltaIndexingManager, delta-delete is executed if no error occured
  • additive - as full but delta-delete is not executed
  • initial - For an initial import in an empty index or a new source in an existing index performance can be optimized by NOT checking if a record needs to be updated (we know that all records are new) but adding an entry in the DeltaIndexingManager for each Record. This allows later runs using full or additive to make use of DeltaIndexing infformation.
  • disabled - delta indexing is fully disabled. No checks are done, no entries are created/updated, no Delta-Delete is executed. Later runs cannot benefit from DeltaIndexing
CompoundHandling
Configuration options for CompoundHandling. See CompoundManagement for details.
Attributes
Placeholder for each crawler's attribute definition.
Each crawler can define here which attributes it can return. An attribute is a specific information of an entry in the datasource that is crawled by the crawler (E.g. In a filesystem an entry is a file, and attributes of an file are Size, Content, etc.)
Process
Placeholder for Tags that the crawler developer can define.
In this Tag all information can be transferred for a crawl task that are necessary to start a crawling process. These information are maybe: starting urls/folder, and which entries should be crawled ( e.g. queries/wildcards/include/excludes).


[edit] Further Information:

  1. See for each crawler attributes and process tags
  2. How to implement a crawler

[edit] Crawler lifecycle

The CrawlerController manages the life cycle of the crawler (e.g. start, stop, abort) and may instantiate multiple crawlers concurrently, even of the same type. This is realised by using OSGi ComponentFactories. Each crawler does not automatically start an OSGi service, but registers only a crawler ComponentFactory with the CrawlerController. Via the ComponentFactory the CrawlerController can instantiate crawlers on demand.

Here is a template for a crawler OSGi component definition:

<component name="%CRAWLER_TYPE%" immediate="false" factory="CrawlerFactory">
    <implementation class="%CRAWLER_IMPLEMENTATION_CLASS%" />
    <service>
         <provide interface="org.eclipse.smila.connectivity.framework.Crawler"/>
    </service>    
</component>

[edit] See also

More information about the different crawlers can be found here: