Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/HowTo/Howto integrate a component in SMILA"

(General review on content and style)
Line 1: Line 1:
This page summarizes the different types and complexity levels of integration of components in SMILA.
+
This page summarizes the different types and complexity levels for the integration of components in SMILA.
 
+
  
 
== Introduction ==
 
== Introduction ==
  
Due to its architecture SMILA allows for the easy integration of third-party components into its framework. Actually there are three different possible integration scenarios available:
+
Due to its architecture SMILA allows for the easy integration of third-party components into its framework. Actually there are three different possible integration scenarios available that are depicted in the following table.
  
 
{| class="wikitable" border="1" cellpadding="5"
 
{| class="wikitable" border="1" cellpadding="5"
 
|-
 
|-
! Integrating services in BPEL
+
! width="33%" | [[#Integrating services in BPEL|Integrating services in BPEL]]
! Integrating agents or crawlers
+
! width="33%" | [[#Integrating agents and crawlers|Integrating agents and crawlers]]
! Integrating alternative implementations for SMILA core components
+
! width="33%" | [[#Integrating alternative implementations for SMILA core components|Integrating alternative implementations for SMILA core components]]
 
|-valign="top"
 
|-valign="top"
| The most frequently used integration scenario. It allows integration or exchange of functionality (services, 3rd party software, etc.) used to process records in the workflow engine.
+
| This is propably the most frequently used integration scenario. It allows for the integration or exchange of functionality (services, 3rd party software, etc.) used to process records in the workflow engine.
 
| Agents and crawlers unlock new data sources, providing additional input to SMILA
 
| Agents and crawlers unlock new data sources, providing additional input to SMILA
 
| Offers experienced developers the potential to exchange existing implementations of SMILA core components with their own implementations.
 
| Offers experienced developers the potential to exchange existing implementations of SMILA core components with their own implementations.
Line 19: Line 18:
 
| [[image:Integrate-Crawler.png]]  
 
| [[image:Integrate-Crawler.png]]  
 
| [[image:Provide-Alternative-To-Core-Component.png]]  
 
| [[image:Provide-Alternative-To-Core-Component.png]]  
 +
|-
 +
| colspan="3" | The above figures exemplary demonstrate at which levels in the [http://www.eclipse.org/smila SMILA architecture] an integration of new components is applicable. However, for simplicity reason, we restricted the above figures to the index processing chain while completely ignoring the search processing chain that also offers similar integration options but is currently not in the focus of this page.
 
|}
 
|}
  
 +
== Integrating services in BPEL ==
  
 
+
There are several options on how to integrate new functionality in SMILA BPEL workflows. In SMILA we call these workflows [[SMILA/Glossary#P|pipelines]].
== Integration of Services in BPEL ==
+
 
+
There are several options on how to integrate new functionality in SMILA BPEL workflows (called [[SMILA/Glossary#P|Pipelines]]).
+
 
* [[#Simple:_webservices|Simple]]: as the standard BPEL functionality to invoke webservices. Not all data of SMILA [[SMILA/Glossary#R|Records]] is usable in this option.
 
* [[#Simple:_webservices|Simple]]: as the standard BPEL functionality to invoke webservices. Not all data of SMILA [[SMILA/Glossary#R|Records]] is usable in this option.
 
* [[#Default:_local SMILA Pipelet or ProcessingService|Default]]: the recommended way to integrate additional functionality in SMILA. Two interfaces allow for easy Java implementations of so called [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]].
 
* [[#Default:_local SMILA Pipelet or ProcessingService|Default]]: the recommended way to integrate additional functionality in SMILA. Two interfaces allow for easy Java implementations of so called [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]].
 
* [[#Advanced:_remote SMILA ProcessingService|Advanced]]: an advanced alternative for integrating ProcessingServices that do not run in the same OSGi runtime as the BPEL workflow but in another OSGI runtime that may even run on a remote machine.
 
* [[#Advanced:_remote SMILA ProcessingService|Advanced]]: an advanced alternative for integrating ProcessingServices that do not run in the same OSGi runtime as the BPEL workflow but in another OSGI runtime that may even run on a remote machine.
  
==== Simple: webservices ====
+
==== Simple: Integrating web services ====
 
The simplest way of integrating additional functionality in SMILA is to call a webservice. This is a standard BPEL workflow engine functionality independent of SMILA. However, there are some limitations concerning the input and result data to/from webservices: In SMILA the workflow object (a DOM object) that enters the BPEL workflow contains by default only the [[SMILA/Glossary#I|Record IDs]]. [[SMILA/Glossary#R|Records]] and the data contained therin are NOT accessible from a BPEL workflow! The BPEL workflow can only access and use the values contained in the BPEL workflow object. It is possible to change this behaviour and add additional data to the workflow object by configuring filters in the configuration file <tt>org.eclipse.smila.blackboard/RecordFilters.xml</tt>. There you can select certain [[SMILA/Glossary#A|Attributes]] and [[SMILA/Glossary#A|Annotations]] that will be copied to the workflow object and so will be accessible by the BPEL workflow. [[SMILA/Glossary#A|Attachments]] are currently NOT supported, as binary data is not reasonable in DOM! Note that you also need to include all [[SMILA/Glossary#A|Attributes]] and [[SMILA/Glossary#A|Annotations]] in the <tt>RecordFilters.xml</tt> you want to write data to.
 
The simplest way of integrating additional functionality in SMILA is to call a webservice. This is a standard BPEL workflow engine functionality independent of SMILA. However, there are some limitations concerning the input and result data to/from webservices: In SMILA the workflow object (a DOM object) that enters the BPEL workflow contains by default only the [[SMILA/Glossary#I|Record IDs]]. [[SMILA/Glossary#R|Records]] and the data contained therin are NOT accessible from a BPEL workflow! The BPEL workflow can only access and use the values contained in the BPEL workflow object. It is possible to change this behaviour and add additional data to the workflow object by configuring filters in the configuration file <tt>org.eclipse.smila.blackboard/RecordFilters.xml</tt>. There you can select certain [[SMILA/Glossary#A|Attributes]] and [[SMILA/Glossary#A|Annotations]] that will be copied to the workflow object and so will be accessible by the BPEL workflow. [[SMILA/Glossary#A|Attachments]] are currently NOT supported, as binary data is not reasonable in DOM! Note that you also need to include all [[SMILA/Glossary#A|Attributes]] and [[SMILA/Glossary#A|Annotations]] in the <tt>RecordFilters.xml</tt> you want to write data to.
  
Line 41: Line 40:
  
  
==== Default: local SMILA Pipelet or ProcessingService ====
+
==== Default: Integrating local SMILA pipelets or processing services ====
 
The default technique to integrate functionality or software in SMILA is to write a [[SMILA/Glossary#P|Pipelet]] or [[SMILA/Glossary#P|ProcessingService]] that runs in the same OSGi runtime as the BPEL workflow engine. [[SMILA/Glossary#P|Pipelets]] are easier to implement than [[SMILA/Glossary#P|ProcessingServices]], as they require only standard Java knowledge. For more information about [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] see [[SMILA/Documentation/Pipelets_and_ProcessingServices|Pipelets and ProcessingServices]].
 
The default technique to integrate functionality or software in SMILA is to write a [[SMILA/Glossary#P|Pipelet]] or [[SMILA/Glossary#P|ProcessingService]] that runs in the same OSGi runtime as the BPEL workflow engine. [[SMILA/Glossary#P|Pipelets]] are easier to implement than [[SMILA/Glossary#P|ProcessingServices]], as they require only standard Java knowledge. For more information about [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] see [[SMILA/Documentation/Pipelets_and_ProcessingServices|Pipelets and ProcessingServices]].
 
Both [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] have full access to [[SMILA/Glossary#R|Records]] in SMILA via the [[SMILA/Glossary#B|BlackboardService]]. So it's easily possible to read, modify and store [[SMILA/Glossary#R|Records]]. In general [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:
 
Both [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] have full access to [[SMILA/Glossary#R|Records]] in SMILA via the [[SMILA/Glossary#B|BlackboardService]]. So it's easily possible to read, modify and store [[SMILA/Glossary#R|Records]]. In general [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:
Line 65: Line 64:
 
* [[SMILA/Development_Guidelines/How to integrate the HelloWorld webservice as a Pipelet|How to integrate the HelloWorld webservice as a Pipelet]]
 
* [[SMILA/Development_Guidelines/How to integrate the HelloWorld webservice as a Pipelet|How to integrate the HelloWorld webservice as a Pipelet]]
  
==== Advanced: remote SMILA ProcessingService ====
+
==== Advanced: Integrating remote SMILA processing services ====
 
tbd.
 
tbd.
  
 +
== Integrating agents and crawlers ==
  
 +
SMILA's connectivity framework allows easy integration of additional datasources by providing implementations of [[SMILA/Glossary#A|Agents]] and/or [[SMILA/Glossary#C|Crawlers]].
  
== Integration of Agents and Crawlers ==
+
* [[SMILA/Development_Guidelines/How to implement an agent|How to implement an agent]]
 +
* [[SMILA/Development_Guidelines/How to implement a crawler|How to implement a crawler]]
  
SMILA's Connectivity Framework allows easy integration of additional datasources by providing implementations of [[SMILA/Glossary#A|Agents]] and/or [[SMILA/Glossary#C|Crawlers]].
+
== Integrating alternative implementations for SMILA core components ==
 
+
* [[SMILA/Development_Guidelines/How to implement a Agent|How to implement a Agent]]
+
* [[SMILA/Development_Guidelines/How to implement a Crawler|How to implement a Crawler]]
+
 
+
 
+
== Providing alternative implementations for SMILA Core-Components ==
+
  
 
SMILA's component based architecture even allows you to provide your own implementations of SMILA core components.  
 
SMILA's component based architecture even allows you to provide your own implementations of SMILA core components.  
Line 85: Line 81:
 
'''Examples''':  
 
'''Examples''':  
 
* a typical example is an alternative implementation of the DeltaIndexingManager that does not store it's state in memory but in the filesystem or in a database
 
* a typical example is an alternative implementation of the DeltaIndexingManager that does not store it's state in memory but in the filesystem or in a database
 
  
 
[[Category:SMILA]]
 
[[Category:SMILA]]

Revision as of 03:55, 30 September 2008

This page summarizes the different types and complexity levels for the integration of components in SMILA.

Introduction

Due to its architecture SMILA allows for the easy integration of third-party components into its framework. Actually there are three different possible integration scenarios available that are depicted in the following table.

Integrating services in BPEL Integrating agents and crawlers Integrating alternative implementations for SMILA core components
This is propably the most frequently used integration scenario. It allows for the integration or exchange of functionality (services, 3rd party software, etc.) used to process records in the workflow engine. Agents and crawlers unlock new data sources, providing additional input to SMILA Offers experienced developers the potential to exchange existing implementations of SMILA core components with their own implementations.
Integrate-Service.png Integrate-Crawler.png Provide-Alternative-To-Core-Component.png
The above figures exemplary demonstrate at which levels in the SMILA architecture an integration of new components is applicable. However, for simplicity reason, we restricted the above figures to the index processing chain while completely ignoring the search processing chain that also offers similar integration options but is currently not in the focus of this page.

Integrating services in BPEL

There are several options on how to integrate new functionality in SMILA BPEL workflows. In SMILA we call these workflows pipelines.

  • Simple: as the standard BPEL functionality to invoke webservices. Not all data of SMILA Records is usable in this option.
  • Default: the recommended way to integrate additional functionality in SMILA. Two interfaces allow for easy Java implementations of so called Pipelets and ProcessingServices.
  • Advanced: an advanced alternative for integrating ProcessingServices that do not run in the same OSGi runtime as the BPEL workflow but in another OSGI runtime that may even run on a remote machine.

Simple: Integrating web services

The simplest way of integrating additional functionality in SMILA is to call a webservice. This is a standard BPEL workflow engine functionality independent of SMILA. However, there are some limitations concerning the input and result data to/from webservices: In SMILA the workflow object (a DOM object) that enters the BPEL workflow contains by default only the Record IDs. Records and the data contained therin are NOT accessible from a BPEL workflow! The BPEL workflow can only access and use the values contained in the BPEL workflow object. It is possible to change this behaviour and add additional data to the workflow object by configuring filters in the configuration file org.eclipse.smila.blackboard/RecordFilters.xml. There you can select certain Attributes and Annotations that will be copied to the workflow object and so will be accessible by the BPEL workflow. Attachments are currently NOT supported, as binary data is not reasonable in DOM! Note that you also need to include all Attributes and Annotations in the RecordFilters.xml you want to write data to.

Examples:

  • A good example for this use case is the integration of [Language Weaver]. The Language Weaver Translation Server provides a webservice interface that allows a text to be translated into another language. This service could be easily used within SMILA.

Here are more detailed technical descriptions:


Default: Integrating local SMILA pipelets or processing services

The default technique to integrate functionality or software in SMILA is to write a Pipelet or ProcessingService that runs in the same OSGi runtime as the BPEL workflow engine. Pipelets are easier to implement than ProcessingServices, as they require only standard Java knowledge. For more information about Pipelets and ProcessingServices see Pipelets and ProcessingServices. Both Pipelets and ProcessingServices have full access to Records in SMILA via the BlackboardService. So it's easily possible to read, modify and store Records. In general Pipelets and ProcessingServices follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:

  • read the configuration (optional)
  • read input data from Blackboard (optional)
  • execute the business logic
  • write result data to Blackboard (optional)

In the part of your Pipelet/Processing service that implements the business logic you are totally free to use any desired technology. Some of the posibilities include


Examples:

Here are more detailed technical descriptions:

Advanced: Integrating remote SMILA processing services

tbd.

Integrating agents and crawlers

SMILA's connectivity framework allows easy integration of additional datasources by providing implementations of Agents and/or Crawlers.

Integrating alternative implementations for SMILA core components

SMILA's component based architecture even allows you to provide your own implementations of SMILA core components. More info comming soon ...

Examples:

  • a typical example is an alternative implementation of the DeltaIndexingManager that does not store it's state in memory but in the filesystem or in a database