Skip to main content
Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/Default configuration workflow overview"

m
m (The diagramme description)
Line 3: Line 3:
 
(original slides can be found here: [[Media:DefaultConfigurationWorkflow.zip|DefaultConfigurationWorkflow.zip]])
 
(original slides can be found here: [[Media:DefaultConfigurationWorkflow.zip|DefaultConfigurationWorkflow.zip]])
  
== The diagramme description==
+
== The diagram description==
  
 
* 1. Data is imported via [[SMILA/Documentation/Crawler|Crawler]] (or [[SMILA/Documentation/Agent|Agent]]) by configuring a data source and a [[SMILA/Glossary#J|job]] name via the [[SMILA/Documentation/CrawlerController|Crawler Controller]] (resp. [[SMILA/Documentation/AgentController|Agent Controller]]) JMX API.  
 
* 1. Data is imported via [[SMILA/Documentation/Crawler|Crawler]] (or [[SMILA/Documentation/Agent|Agent]]) by configuring a data source and a [[SMILA/Glossary#J|job]] name via the [[SMILA/Documentation/CrawlerController|Crawler Controller]] (resp. [[SMILA/Documentation/AgentController|Agent Controller]]) JMX API.  
Line 16: Line 16:
 
* 10. Records are cumulated in [[SMILA/Glossary#B|bulks]] for asynchronous workflow processing. [[SMILA/Glossary#R|Record bulks]] are stored in '''ObjectStore'''.
 
* 10. Records are cumulated in [[SMILA/Glossary#B|bulks]] for asynchronous workflow processing. [[SMILA/Glossary#R|Record bulks]] are stored in '''ObjectStore'''.
 
* 11. An [[SMILA/Glossary#W|asynchronous workflows]] is executed triggered by the [[SMILA/Documentation/Bulkbuilder|Bulkbuilder]] generated record bulk. This is managed by the [[SMILA/Documentation/JobManager|Jobmanager]] and [[SMILA/Documentation/TaskManager|Taskmanager]] components. Runtime/Synchronization data is stored in '''Zookeeper''', persistent data is stored in '''ObjectStore'''.
 
* 11. An [[SMILA/Glossary#W|asynchronous workflows]] is executed triggered by the [[SMILA/Documentation/Bulkbuilder|Bulkbuilder]] generated record bulk. This is managed by the [[SMILA/Documentation/JobManager|Jobmanager]] and [[SMILA/Documentation/TaskManager|Taskmanager]] components. Runtime/Synchronization data is stored in '''Zookeeper''', persistent data is stored in '''ObjectStore'''.
* 12. Predefined asynchronous workflow ''indexUpdate'' contains [[SMILA/Documentation/Worker/PipelineProcessingWorker|BPEL worker]] for embedding (resp. executing) synchronous BPEL pipelines in the asynchronous workflow. Added records are passed to the predefined BPEL pipeline ''AddPipeline'', deleted records to the ''DeletePipeline''. A BPEL pipeline is a process using a set of ''Pipelets'' to process a record's data (e.g. extracting text from various document or image file types).
+
* 12. Predefined asynchronous workflow ''indexUpdate'' contains [[SMILA/Documentation/Worker/PipelineProcessingWorker|BPEL worker]] for embedding (resp. executing) synchronous BPEL pipelines in the asynchronous workflow. Added records are passed to the predefined BPEL pipeline ''AddPipeline'', deleted records to the ''DeletePipeline''.
* 13. After processing the records the pipelets store the gathered additional data via the [[SMILA/Documentation/Usage_of_Blackboard_Service|Blackboard]] service.  
+
* 13. A BPEL pipeline uses a set of ''Pipelets'' to process a record's data (e.g. extracting text from various document or image file types). After processing the records the pipelets can store the gathered additional data via the [[SMILA/Documentation/Usage_of_Blackboard_Service|Blackboard]] service.  
* 14. The Add- and DeletePipeline finally invoke the [[SMILA/Documentation/LuceneIndexPipelet|LuceneIndexPipelet]] to update the '''Lucene Index'''.
+
* 14. The Add- and DeletePipeline contain a [[SMILA/Documentation/LuceneIndexPipelet|LuceneIndexPipelet]] which is finally invoked to update the '''Lucene Index'''.

Revision as of 09:34, 5 September 2011

DefaultConfigurationWorkflow.jpg

(original slides can be found here: DefaultConfigurationWorkflow.zip)

The diagram description

  • 1. Data is imported via Crawler (or Agent) by configuring a data source and a job name via the Crawler Controller (resp. Agent Controller) JMX API.
  • 2. The Crawler Controller initializes the Crawler by assigning a data source and starting the import
  • 3. The Crawler retrieves data references from the Data Source and returns them to the Crawler Controller.
  • 4. The Crawler Controller determines whether this particular data is new/modified or was already indexed by querying the Delta Indexing Service.
  • 5. If the data was not previously indexed, the Crawler Controller instructs the Crawler to retrieve the full data plus content as Record (metadata + attachment).
  • 6. The Crawler fetches the complete record from the Data Source. Each record has an ID and can contain metadata and attachments (binary content).
  • 7. The Crawler Controller sends the complete retrieved records to the Connectivity Manager.
  • 8. The Connectivity Manager routes the records to the configured job by pushing them to the Bulkbuilder.
  • 9. The Bulkbuilder persists the record's attachment content via the Blackboard in the Binary Storage. Only attachment references remanin in the records. Should any subsequent processes require the record’s full content, they can access it via the Blackboard.
  • 10. Records are cumulated in bulks for asynchronous workflow processing. Record bulks are stored in ObjectStore.
  • 11. An asynchronous workflows is executed triggered by the Bulkbuilder generated record bulk. This is managed by the Jobmanager and Taskmanager components. Runtime/Synchronization data is stored in Zookeeper, persistent data is stored in ObjectStore.
  • 12. Predefined asynchronous workflow indexUpdate contains BPEL worker for embedding (resp. executing) synchronous BPEL pipelines in the asynchronous workflow. Added records are passed to the predefined BPEL pipeline AddPipeline, deleted records to the DeletePipeline.
  • 13. A BPEL pipeline uses a set of Pipelets to process a record's data (e.g. extracting text from various document or image file types). After processing the records the pipelets can store the gathered additional data via the Blackboard service.
  • 14. The Add- and DeletePipeline contain a LuceneIndexPipelet which is finally invoked to update the Lucene Index.

Back to the top