SMILA/Documentation/Default configuration workflow overview

The diagramme description

• 1. Data is imported via Crawler (or Agent) by configuring a data source and a job name via the Crawler Controller (resp. Agent Controller) JMX API.
• 2. The Crawler Controller initializes the Crawler by assigning a data source and starting the import
• 3. The Crawler retrieves data references from the Data Source and returns them to the Crawler Controller.
• 4. The Crawler Controller determines whether this particular data is new/modified or was already indexed by querying the Delta Indexing Service.
• 5. If the data was not previously indexed, the Crawler Controller instructs the Crawler to retrieve the full data plus content as Record (metadata + attachment).
• 6. The Crawler fetches the complete record from the Data Source. Each record has an ID and can contain metadata and attachments (binary content).
• 7. The Crawler Controller sends the complete retrieved records to the Connectivity Manager.
• 8. The Connectivity Manager routes the records to the configured job by pushing them to the Bulkbuilder.
• 9. The Bulkbuilder persists the record's attachment content via the Blackboard in the Binary Storage. Only attachment references remanin in the records. Should any subsequent processes require the record’s full content, they can access it via the Blackboard.
• 10. Records are cumulated in bulks for asynchronous workflow processing. Record bulks are stored in ObjectStore.
• 11. An asynchronous workflows is executed triggered by the Bulkbuilder generated record bulk. This is managed by the Jobmanager and Taskmanager components. Runtime/Synchronization data is stored in Zookeeper, persistent data is stored in ObjectStore.
• 12. Predefined asynchronous workflow indexUpdate contains BPEL worker for embedding (resp. executing) synchronous BPEL pipelines in the asynchronous workflow. Added records are passed to the predefined BPEL pipeline AddPipeline, deleted records to the DeletePipeline. A BPEL pipeline is a process using a set of Pipelets to process a record's data (e.g. extracting text from various document or image file types).
• 13. After processing the records the pipelets store the gathered additional data via the Blackboard service.
• 14. The Add- and DeletePipeline finally invoke the LuceneIndexPipelet to update the Lucene Index.