Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/HowTo/Howto integrate a component in SMILA"

(General review on content and style)
(General review on content and style)
Line 52: Line 52:
  
 
==== Default: Integrating local SMILA pipelets or processing services ====
 
==== Default: Integrating local SMILA pipelets or processing services ====
The default technique to integrate functionality or software in SMILA is to write a [[SMILA/Glossary#P|Pipelet]] or [[SMILA/Glossary#P|ProcessingService]] that runs in the same OSGi runtime as the BPEL workflow engine. [[SMILA/Glossary#P|Pipelets]] are easier to implement than [[SMILA/Glossary#P|ProcessingServices]], as they require only standard Java knowledge. For more information about [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] see [[SMILA/Documentation/Pipelets_and_ProcessingServices|Pipelets and ProcessingServices]].
+
The default and thus recommended technique to integrate functionality or software in SMILA is to provide a [[SMILA/Glossary#P|pipelet]] or a [[SMILA/Glossary#P|processing service]] that runs in the same OSGi runtime as the BPEL workflow engine. Pipelets are easier to implement as they require only standard Java knowledge but lack the functionality of processing services to be shared between multiple pipelines and to be accessed by multiple threads. Another difference is that the lifecycle and configuration of pipelets is managed by the workflow engine whereas processing services are handled by OSGi runtime. For further information on pipelets and processing services refer to [[SMILA/Documentation/Pipelets_and_ProcessingServices|Pipelets and ProcessingServices]].
Both [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] have full access to [[SMILA/Glossary#R|Records]] in SMILA via the [[SMILA/Glossary#B|BlackboardService]]. So it's easily possible to read, modify and store [[SMILA/Glossary#R|Records]]. In general [[SMILA/Glossary#P|Pipelets]] and [[SMILA/Glossary#P|ProcessingServices]] follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:
+
* read the configuration (optional)
+
* read input data from [[SMILA/Glossary#B|Blackboard]] (optional)
+
* execute the business logic
+
* write result data to [[SMILA/Glossary#B|Blackboard]] (optional)
+
  
In the part of your [[SMILA/Glossary#P|Pipelet]]/Processing service that implements the business logic you are totally free to use any desired technology. Some of the posibilities include
+
The above mentioned restriction of integrated web services using the BPEL default engine functionality does not apply to pipelets and processing services. Both have full access to SMILA [[SMILA/Glossary#R|records]] by using the [[SMILA/Glossary#B|blackboard service]], which makes it easy to read, modify, and store [[SMILA/Glossary#R|records]].
* use of POJOs (e.g. see the various [[SMILA/Documentation/Bundle_org.eclipse.smila.processing.pipelets.xmlprocessing|XML Processing Pipelets]])
+
 
* use any local available OSGi service, even other [[SMILA/Glossary#P|ProcessingServices]] (e.g. see the [[SMILA/Documentation/AperturePipelet|AperturePipelet]] that uses the [[SMILA/Documentation/ApertureMimeTypeIdentifier|ApertureMimeTypeIdentifier]])
+
In general pipelets and processing services follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:
* use other technologies like JNI, RMI, Corba, etc. to integrate remote or non Java components (e.g. integration of Oracle Outside In Technology)
+
* Read the configuration (optional)
* ...
+
* Read input data from blackboard (optional)
 +
* Execute the business logic
 +
* Write result data to blackboard (optional)
 +
 
 +
In terms of the pipelet or processing service that implements the business logic you are totally free to use any desired technology. Some of the posibilities include:
 +
* Using POJOs (For examples refer to the [[SMILA/Documentation/Bundle_org.eclipse.smila.processing.pipelets.xmlprocessing|XML processing pipelets]])
 +
* Using any local available OSGi service, even other processing services (For an example refer to the [[SMILA/Documentation/AperturePipelet|Aperture pipelet]] that uses the [[SMILA/Documentation/ApertureMimeTypeIdentifier|ApertureMimeTypeIdentifier]] processing service.)
 +
* Using other technologies such as JNI, RMI, or CORBA to integrate remote or non Java components (As an example consider the integration of [http://www.oracle.com/technologies/embedded/outside-in.html Oracle Outside In Technology].)
  
 
===== Examples =====
 
===== Examples =====
Line 71: Line 73:
  
 
===== Further reading =====
 
===== Further reading =====
Here are more detailed technical descriptions:
+
Please consult the following how-to tutorials for a more detailed technical description:
* [[SMILA/Development_Guidelines/How to write a Pipelet|How to write a Pipelet]]
+
* [[SMILA/Development_Guidelines/How to write a Pipelet|How to write a pipelet]]
* [[SMILA/Development_Guidelines/How to write a ProcessingService|How to write a ProcessingService]]
+
* [[SMILA/Development_Guidelines/How to write a ProcessingService|How to write a processing service]]
* [[SMILA/Development_Guidelines/How to integrate the HelloWorld webservice as a Pipelet|How to integrate the HelloWorld webservice as a Pipelet]]
+
* [[SMILA/Development_Guidelines/How to integrate the HelloWorld webservice as a Pipelet|How to integrate the HelloWorld web service as a pipelet]]
  
 
==== Advanced: Integrating remote SMILA processing services ====
 
==== Advanced: Integrating remote SMILA processing services ====

Revision as of 07:41, 30 September 2008

This page summarizes the different types and complexity levels for the integration of components in SMILA.

Introduction

Due to its architecture SMILA allows for the easy integration of third-party components into its framework. Actually there are three different possible integration scenarios available that are depicted in the following table.

Integrating services in BPEL Integrating agents and crawlers Integrating alternative implementations for SMILA core components
This is propably the most frequently used integration scenario. It allows for the integration or exchange of functionality (services, 3rd party software, etc.) used to process records in the workflow engine. Integrating your own crawler or agent implementations is another common scenario for adding functionality to SMILA. By doing so, further data sources can be unlocked to provide additional input to SMILA. This scenario is particularly intended for the experienced (SMILA) developer and comprises the possibility to exchange existing implementations of the SMILA core components by your own implementations.
Integrate-Service.png Integrate-Crawler.png Provide-Alternative-To-Core-Component.png
The figure demonstrates how you can integrate the functionality of your service or your piece of software to SMILA by adding it to the workflow engine. The figure above exemplary shows how you can add your own crawler implementation to SMILA. Please note that though you may also add an agent implementation likewise this option is not shown in the figure. This was chosen due to simplicity. The figure above demonstrates how two of the SMILA core components -- connectivity and data store -- may be exchanged by your own implementations. These components serve as examples only, that is, you may also exchange other core components such as the blackboard service or the delta indexing manager.
The above figures exemplary demonstrate at which levels in the SMILA architecture an integration of new components is applicable. However, for simplicity reason, we restricted the above figures to the index processing chain while completely ignoring the search processing chain that also offers similar integration options but is currently not in the focus of this page.

Integrating services in BPEL

As already shown in the overview above, SMILA offers the possibility to integrate your own service or piece of software into SMILA BPEL workflows. In SMILA we simply call these workflows pipelines. A pipeline is the definition of a BPEL process (or workflow) that orchestrates pipelets, processing services, and other BPEL services (e.g. web services).

There are several options on how to achieve this:

  • Simple: The easiest method to add functionality is to invoke a web service by using the standard functionality of BPEL. However, the disadvantage is that not all data of SMILA records are accessible if you opt for this method of integration.
  • Default: The recommended way to integrate additional functionality in SMILA is to provide Java implementations of two interfaces that allow for an easy creation of the above mentioned pipelets and processing services.
  • Advanced: This method extends the default mechanism by providing an alternative procedure for integrating processing services that do not run in the same OSGi runtime as the BPEL workflow but in another OSGI runtime that may even run on a remote machine.

Simple: Integrating web services

The simplest way of integrating additional functionality in SMILA is to call a web service, which is a standard BPEL workflow engine functionality independent of SMILA. However, there are some limitations concerning the input and result data to/from web services: The workflow object (a DOM object) that enters the BPEL workflow in SMILA contains only the record IDs by default. That means records and the data contained therein - attributes, annotations, and attachments - are not accessible from a BPEL workflow because it can only access and use the values contained in the BPEL workflow object.

To overcome this restriction you can add additional data to the workflow object by adding filters in the configuration file located at org.eclipse.smila.blackboard/RecordFilters.xml. These filter rules define which attributes and annotations should be copied to the workflow object to make them accessible in the BPEL workflow. Additionally, you should not forget to include all attributes and annotations in the RecordFilters.xml file that you wish to write data to. Though filters work on attributes and annotations there is no possibility to access attachments of records because binary data is not reasonable in DOM.

Examples

A good example for this use case is the integration of the Language Weaver web service. The Language Weaver Translation Server provides a web service interface that allows a text to be translated into another language. This service could easily be used within SMILA to extends its functionality.

Further reading

Please consult the following how-to tutorials for a more detailed technical description:

Default: Integrating local SMILA pipelets or processing services

The default and thus recommended technique to integrate functionality or software in SMILA is to provide a pipelet or a processing service that runs in the same OSGi runtime as the BPEL workflow engine. Pipelets are easier to implement as they require only standard Java knowledge but lack the functionality of processing services to be shared between multiple pipelines and to be accessed by multiple threads. Another difference is that the lifecycle and configuration of pipelets is managed by the workflow engine whereas processing services are handled by OSGi runtime. For further information on pipelets and processing services refer to Pipelets and ProcessingServices.

The above mentioned restriction of integrated web services using the BPEL default engine functionality does not apply to pipelets and processing services. Both have full access to SMILA records by using the blackboard service, which makes it easy to read, modify, and store records.

In general pipelets and processing services follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:

  • Read the configuration (optional)
  • Read input data from blackboard (optional)
  • Execute the business logic
  • Write result data to blackboard (optional)

In terms of the pipelet or processing service that implements the business logic you are totally free to use any desired technology. Some of the posibilities include:

Examples
Further reading

Please consult the following how-to tutorials for a more detailed technical description:

Advanced: Integrating remote SMILA processing services

tbd.

Integrating agents and crawlers

SMILA's connectivity framework allows easy integration of additional data sources by providing implementations of Agents and/or Crawlers.

Examples

tbd.

Further reading

Integrating alternative implementations for SMILA core components

SMILA's component based architecture even allows you to provide your own implementations of SMILA core components. More info comming soon ...

Examples

A typical example is an alternative implementation of the DeltaIndexingManager that does not store it's state in memory but in the filesystem or in a database

Further reading

tbd.

Copyright © Eclipse Foundation, Inc. All Rights Reserved.