Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Documentation/Importing/UpdatePusher"
< SMILA | Documentation
(→Worker description) |
(→Worker description) |
||
| Line 3: | Line 3: | ||
* Name: <tt>updatePusher</tt> | * Name: <tt>updatePusher</tt> | ||
* Parameters: | * Parameters: | ||
| − | ** <tt>jobToPushTo</tt>: The job to push the crawled records to. | + | ** <tt>jobToPushTo</tt>: The job to push the crawled records to. A job with this worker will cannot be started, if the <tt>jobToPushTo</tt> is not running. |
* Input Slots: | * Input Slots: | ||
** <tt>recordToPush</tt>: a bucket of type <tt>recordBulks</tt> containing the records produced by the crawl workflow. | ** <tt>recordToPush</tt>: a bucket of type <tt>recordBulks</tt> containing the records produced by the crawl workflow. | ||
Revision as of 09:32, 30 January 2012
Worker description
- Name: updatePusher
- Parameters:
- jobToPushTo: The job to push the crawled records to. A job with this worker will cannot be started, if the jobToPushTo is not running.
- Input Slots:
- recordToPush: a bucket of type recordBulks containing the records produced by the crawl workflow.
- Output Slots:
- pushedRecords: (optional) the records that could be successfully submitted to the destination job. Usually not set, but may be used to trigger further actions on submitted records.
The UpdatePusher takes each record from the input, sends it to a bulkbuilder service. If an output bucket is connected the record is written to it. If the record contains a _deltaHash attribute value, the worker checks with DeltaService if the record has not been pushed yet to prevent duplicates, and marks it updated afterwards. If the _deltaHash attribute is empty, the record is pushed always and not marked as updated in DeltaService.
Exception handling of bulkbuilder errors:
- If an InvalidRecordException is thrown by Bulkbuilder it is logged and the record is skipped (and is also not added to the output bulk, if set).
- Other BulkbuilderExceptions are not catched. If they are marked as recoverable they should lead to an retry of the task, else the task will fail fatal.