SMILA/Documentation/Importing/Crawler/Web - Eclipsepedia

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

< SMILA‎ | Documentation

Revision as of 10:47, 12 December 2011 by Unnamed Poltroon (Talk)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Currently, the web crawler workers are implemented very simplistic so that we can test the importing framework. A more sophisticated implementation will follow soon (hopefully).

Web Crawler

Worker name: webCrawler
Parameters:
- dataSource
- startUrl
Task generator: runOnceTrigger
Input slots:
- linksToCrawl

Output slots:
- linksToCrawl
- crawledLinks

Internal structure

Retrieved from "https://wiki.eclipse.org/index.php?title=SMILA/Documentation/Importing/Crawler/Web&oldid=281950"