Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Documentation/Importing/UpdatePusher"
(→Worker description) |
(→Worker description) |
||
| Line 19: | Line 19: | ||
* Other BulkbuilderExceptions are not catched. If they are marked as recoverable they should lead to an retry of the task, else the task will fail fatal. | * Other BulkbuilderExceptions are not catched. If they are marked as recoverable they should lead to an retry of the task, else the task will fail fatal. | ||
| − | If enabled (parameter <tt> | + | If enabled (parameter <tt>deltaImportStrategy="full"</tt> or not set), the worker scans the DeltaService in the completion phase of the job run for records that must be sent to the BulkBuilder as "deleted records" and removes these entries from the DeltaServer afterwards. See [[SMILA/Documentation/Importing/Concept#Delta_Delete|DeltaDelete]] for details. |
[[Category:SMILA]] | [[Category:SMILA]] | ||
Revision as of 10:10, 20 February 2012
Worker description
- Name: updatePusher
- Parameters:
- jobToPushTo: The job to push the crawled records to. A job with this worker will cannot be started, if the jobToPushTo is not running.
- deltaImportStrategy: Configure usage of the DeltaService by this worker. There are four possible values, two of them have the same effect on this worker (see DeltaDelete for an overview):
- none: do not record delta information, do not perform delta-delete in the completion phase of the job.
- initial or additive: record delta information, but do not perform delta-delete in the completion phase of the job.
- full: default mode, record delta information and perform delta-delete in the completion phase of the job.
- Input Slots:
- recordToPush: a bucket of type recordBulks containing the records produced by the crawl workflow.
- Output Slots:
- pushedRecords: (optional) the records that could be successfully submitted to the destination job. Usually not set, but may be used to trigger further actions on submitted records.
The UpdatePusher takes each record from the input, sends it to a bulkbuilder service. If an output bucket is connected the record is written to it. If the record contains a _deltaHash attribute value, the worker checks with DeltaService if the record has not been pushed yet to prevent duplicates, and marks it updated afterwards (if enabled, see above). If the _deltaHash attribute is empty, the record is pushed always and not marked as updated in DeltaService.
Exception handling of bulkbuilder errors:
- If an InvalidRecordException is thrown by Bulkbuilder it is logged and the record is skipped (and is also not added to the output bulk, if set).
- Other BulkbuilderExceptions are not catched. If they are marked as recoverable they should lead to an retry of the task, else the task will fail fatal.
If enabled (parameter deltaImportStrategy="full" or not set), the worker scans the DeltaService in the completion phase of the job run for records that must be sent to the BulkBuilder as "deleted records" and removes these entries from the DeltaServer afterwards. See DeltaDelete for details.