Skip to main content
Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/Crawler"

Line 1: Line 1:
== What does Crawler do ==
== What does Crawler do ==

Revision as of 03:15, 19 March 2009


What does Crawler do

A Crawler gathers information about resources, both content and metadata of interest like size or mime type.

Crawler types

SMILA contains three types of crawlers, each for a different data source type, namely WebCrawler, JDBC DatabaseCrawler, and FileSystemCrawler to facilitate gathering information from the internet, databases, or files from a hard disk.

Furthermore, the Connectivity Framework provides an API for developers to create own crawlers.

Crawler configuration

A Crawler is started with a specific, named configuration, that defines what information is to be crawled (e.g. content, kinds of metadata) and where to find that data (e.g. file system path, JDBC Connection String).

Crawler lifecycle

The CrawlerController manages the life cycle of the crawler (e.g. start, stop, abort) and may instantiate multiple Crawlers concurrently, even of the same type.

See also

Back to the top