Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Documentation/Importing/Crawler/Feed"
(→FeedCrawler) |
(→FeedCrawler) |
||
| Line 16: | Line 16: | ||
* RSS 2.0 | * RSS 2.0 | ||
* Atom 0.3 | * Atom 0.3 | ||
| − | * Atom 1.0 | + | * Atom 1.0 |
==== Configuration ==== | ==== Configuration ==== | ||
Revision as of 05:24, 27 June 2012
The FeedCrawler is used to read RSS or Atom feed in importing workflows.
Contents
FeedCrawler
The Feed crawler offers the functionality to read RSS and Atom feeds. The implementation uses ROME Fetcher to retrieve and parse the feeds. ROME supports the following feed formats:
- RSS 0.90
- RSS 0.91 Netscape
- RSS 0.91 Userland
- RSS 0.92
- RSS 0.93
- RSS 0.94
- RSS 1.0
- RSS 2.0
- Atom 0.3
- Atom 1.0
Configuration
The FeedCrawler worker is usually the first worker in a workflow and the job is started in runOnce mode.
- Worker name: feedCrawler
- Parameters:
- dataSource (req.) value for attribute _source, needed e.g. by the delta service
- feedUrls (req.) URLs (usually HTTP) of the feeds to read. Can be a single string value or a list of string values. Currently, all feeds are read in a single task.
- mapping (req.) Mapping of feed and feed item properties to record attribute names. See below for the available property names.
- deltaProperties (opt.) a list of feed or feed item property names (see below) used to generate the value for attribute _deltaHash. If not set, a unique _deltaHash value is generated for each record so that it will be updated in any case, if delta checking is enabled.
- maxRecordsPerBulk (opt.) maximum number of item records in one bulk in the output bucket. (default: 1000)
- Output slots:
- crawledRecords: One record per item read from the feeds.
Feed properties
These are properties of the feed that can be mapped to record attributes. The values will be identical for all records created from entries of a single feed. Some are not only simple values, but structured, i.e. (mostly list of) maps. The attributes of these map objects are described in further tables below, they cannot be changed via the mapping. Attributes associated to structured properties are not set to empty objects, e.g. a list attribute is either not set at all or the list does indeed have elements.
| Property | Type | Description |
|---|---|---|
| feedAuthors | List<Person> | Returns the feed authors |
| feedCategories | List<Category> | Returns the feed categories |
| feedContributors | List<Person> | Returns the feed contributors |
| feedCopyright | String | Returns the feed copyright information |
| feedDescription | String | Returns the feed description |
| feedEncoding | String | Returns the charset encoding of the feed |
| feedType | String | Returns the feed type |
| feedImage | Image | Returns the feed image |
| feedLanguage | String | Returns the feed language |
| feedLinks | List<Link> | Returns the feed links |
| feedPublishDate | Date | Returns the feed published date |
| feedTitle | String | Returns the feed title |
| feedUri | String | Returns the feed uri |
Feed Item properties
And these are properties extracted from the single feed items:
| Attribute | Type | Description |
|---|---|---|
| itemAuthors | List<Person> | Returns a feed entry authors |
| itemCategories | List<Category> | Returns a feed entry categories |
| itemContents | List<Content> | Returns a feed entry contents |
| itemContributors | List<Person> | Returns a feed entry contributors |
| itemDescription | Content | Returns a feed entry description |
| itemEnclosures | List<Enclosure> | Returns a feed entry enclosures |
| itemLinks | List<Link> | Returns a feed entry links |
| itemPublishDate | Date | Returns a feed entry publish date |
| itemTitle | String | Returns a feed entry title |
| itemUpdateDate | Date | Returns a feed entry update date. |
| itemUri | String | Returns a feed entry uri. |
Properties of structured feed/item properties
Content maps can contain these properties:
- Mode: String
- Value: String
- Type: String
Person maps can contain these properties:
- Email: String
- Name: String
- Uri: String
Image maps can contain these properties:
- Link: String
- Title: String
- Url: String
- Description: String
Property maps can contain these properties:
- Name: String
- TaxanomyUri: String
Property maps can contain these properties:
- Type: String
- Url: String
- Length: Integer
Link maps can contain these properties:
- Href: String
- Hreflang: String
- Rel: Integer
- Title: String
- Type: String
- Length: Integer