Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Documentation/Default configuration workflow overview"
< SMILA | Documentation
| Line 18: | Line 18: | ||
* 12. Predefined asynchronous workflow ''indexUpdate'' contains [[SMILA/Documentation/Worker/PipelineProcessorWorker|BPEL worker]] for embedding (resp. executing) synchronous BPEL pipelines in the asynchronous workflow. Added records are passed to the predefined BPEL pipeline ''AddPipeline'', deleted records to the ''DeletePipeline''. | * 12. Predefined asynchronous workflow ''indexUpdate'' contains [[SMILA/Documentation/Worker/PipelineProcessorWorker|BPEL worker]] for embedding (resp. executing) synchronous BPEL pipelines in the asynchronous workflow. Added records are passed to the predefined BPEL pipeline ''AddPipeline'', deleted records to the ''DeletePipeline''. | ||
* 13. A BPEL pipeline uses a set of ''Pipelets'' to process a record's data (e.g. extracting text from various document or image file types). After processing the records the pipelets can store the gathered additional data via the [[SMILA/Documentation/Usage_of_Blackboard_Service|Blackboard]] service. | * 13. A BPEL pipeline uses a set of ''Pipelets'' to process a record's data (e.g. extracting text from various document or image file types). After processing the records the pipelets can store the gathered additional data via the [[SMILA/Documentation/Usage_of_Blackboard_Service|Blackboard]] service. | ||
| − | * 14. The Add- and DeletePipeline contain a [[SMILA/Documentation/ | + | * 14. The Add- and DeletePipeline contain a [[SMILA/Documentation/Solr#How_to_use_Solr_with_SMILA|SolrIndexPipelet]] which is finally invoked to update the '''Solr/Lucene Index'''. |
Revision as of 05:18, 16 January 2012
(original slides can be found here: DefaultConfigurationWorkflow.zip)
The diagram description
- 1. Data is imported via Crawler (or Agent) by configuring a data source and a job name via the Crawler Controller (resp. Agent Controller) JMX API.
- 2. The Crawler Controller initializes the Crawler by assigning a data source and starting the import
- 3. The Crawler retrieves data references from the Data Source and returns them to the Crawler Controller.
- 4. The Crawler Controller determines whether this particular data is new/modified or was already indexed by querying the Delta Indexing Service.
- 5. If the data was not previously indexed, the Crawler Controller instructs the Crawler to retrieve the full data plus content as Record (metadata + attachment).
- 6. The Crawler fetches the complete record from the Data Source. Each record has an ID and can contain metadata and attachments (binary content).
- 7. The Crawler Controller sends the complete retrieved records to the Connectivity Manager.
- 8. The Connectivity Manager routes the records to the configured job by pushing them to the Bulkbuilder.
- 9. The Bulkbuilder persists the record's attachment content via the Blackboard in the Binary Storage. Only attachment references remanin in the records. Should any subsequent processes require the record’s full content, they can access it via the Blackboard.
- 10. Records are cumulated in bulks for asynchronous workflow processing. Record bulks are stored in ObjectStore.
- 11. An asynchronous workflows is executed triggered by the Bulkbuilder generated record bulk. This is managed by the Jobmanager and Taskmanager components. Runtime/Synchronization data is stored in Zookeeper, persistent data is stored in ObjectStore.
- 12. Predefined asynchronous workflow indexUpdate contains BPEL worker for embedding (resp. executing) synchronous BPEL pipelines in the asynchronous workflow. Added records are passed to the predefined BPEL pipeline AddPipeline, deleted records to the DeletePipeline.
- 13. A BPEL pipeline uses a set of Pipelets to process a record's data (e.g. extracting text from various document or image file types). After processing the records the pipelets can store the gathered additional data via the Blackboard service.
- 14. The Add- and DeletePipeline contain a SolrIndexPipelet which is finally invoked to update the Solr/Lucene Index.