Difference between revisions of "SMILA/Documentation/HowTo/Howto integrate a component in SMILA"

From Eclipsepedia

Jump to: navigation, search
(Handling of Character Encoding)
m (Introduction)
 
(20 intermediate revisions by 6 users not shown)
Line 3: Line 3:
 
== Introduction ==
 
== Introduction ==
  
Due to its architecture SMILA allows for the easy integration of third-party components into its framework. Actually there are three different possible integration scenarios available that are depicted in the following table.  
+
Due to its architecture SMILA allows for the easy integration of third-party components into its framework. Actually there are four different possible integration scenarios available that are depicted in the following table.  
  
 
{| class="wikitable" border="1" cellpadding="5"
 
{| class="wikitable" border="1" cellpadding="5"
 
|-
 
|-
! width="33%" | [[#Integrating BPEL services|Integrating BPEL services]]
+
| [[#Integrating a SMILA pipelet|Integrating a SMILA pipelet]]
! width="33%" | [[#Integrating data sources|Integrating data sources]]
+
! width="33%" | [[#Integrating alternative implementations of SMILA core components|Integrating alternative implementations of SMILA core components]]
+
 
|-valign="top"
 
|-valign="top"
 
| This is probably the most frequently used integration scenario. It allows for the integration or exchange of functionality (services, 3rd party software, etc.) used to process records in the workflow engine.
 
| This is probably the most frequently used integration scenario. It allows for the integration or exchange of functionality (services, 3rd party software, etc.) used to process records in the workflow engine.
| Integrating your own [[SMILA/Glossary#C|crawler]] or [[SMILA/Glossary#A|agent]] implementations is another common scenario for adding functionality to SMILA. By doing so, further data sources can be unlocked to provide additional input to SMILA.
 
| This scenario is particularly intended for the experienced (SMILA) developer and comprises the possibility to exchange existing implementations of the SMILA core components by your own implementations.
 
 
|-valign="top"
 
|-valign="top"
| [[image:Integrate-Service.png]]
+
| [[image:Integrate-Service_1.1.0.png|900px]]  
| [[image:Integrate-Crawler.png]]
+
| [[image:Provide-Alternative-To-Core-Component.png]]  
+
 
|-valign="top"
 
|-valign="top"
 
| The figure demonstrates how you can integrate the functionality of your service or your piece of software to SMILA by adding it to the workflow engine.
 
| The figure demonstrates how you can integrate the functionality of your service or your piece of software to SMILA by adding it to the workflow engine.
| The figure above exemplary shows how you can add your own crawler implementation to SMILA. Please note that though you may also add an agent implementation likewise this option is not shown in the figure. This was chosen due to simplicity.
 
| The figure above demonstrates how two of the SMILA core components -- ''connectivity'' and ''data store'' -- may be exchanged by your own implementations. These components serve as examples only, that is, you may also exchange other core components such as the [[SMILA/Glossary#B|blackboard service]] or the [[SMILA/Glossary#|delta indexing manager]].
 
|-valign="top"
 
| colspan="3" | The above figures exemplary demonstrate at which levels in the [http://www.eclipse.org/smila SMILA architecture] an integration of new components is applicable. However, for simplicity reason, we restricted the above figures to the index processing chain while completely ignoring the search processing chain that offers the same integration options (except for the integration of agents and crawlers), but is currently not in the focus of this page.
 
 
|}
 
|}
  
== Conventions ==
 
  
=== Handling of Character Encoding ===
+
{| class="wikitable" border="1" cellpadding="5"
 +
|-
 +
| [[#Integrating workers|Integrating workers]]
 +
|-valign="top"
 +
| Integrating your own [[SMILA/Glossary#W|worker]] implementation is another common scenario for adding functionality or adapting workflows to SMILA.
 +
|-valign="top"
 +
| [[image:Integrate-Worker_1.1.0.png|900px]]
 +
|-valign="top"
 +
| The figure above exemplary shows how you can add your own worker implementation to SMILA. Please note that you also have to add the worker to the job manager configuration files (workers.json, workflows.json) and add your worker as an OSGi service to activate it.
 +
|}
  
To make processing of data in SMILA easier: If external data must be converted to a string (e.g. an attribute value), the crawler, agent or any other component accessing an external data source should try everything that is possible to ensure that the conversion is done using the correct encoding. For example, HTTP clients should use the encoding reported by the HTTP server. If the data source does not provide information about the character encoding, you can use the class <tt>org.eclipse.smila.utils.file.EncodingHelper</tt> that tries to convert a byte[] to a string by trying to detect the correct encoding from a <tt>byte[]</tt> by checking BOMs or checking XML and HTML content for instructions and finally by using UTF-8 or, if this fails, the default platform encoding. You may find this helpful.
 
  
On the other hand, if valid string data must be converted to a byte[] (e.g. if it is stored as a attachment after pipelet processing), the conversion must always use UTF-8 encoding.
+
{| class="wikitable" border="1" cellpadding="5"
 +
|-
 +
| [[#Integrating data sources|Integrating data sources]]
 +
|-valign="top"
 +
| Integrating a data source by implementing your own crawler is another scenario for adding functionality to SMILA. By doing so, further data sources can be unlocked to provide additional input to SMILA.
 +
|-valign="top"
 +
| [[image:Integrate-DataSources_1.1.0.png|900px]]
 +
|-valign="top"
 +
| The figure above exemplary shows how you can add your own crawler implementation to SMILA. Please note that eventually you have to add a new Fetcher or extractors or further workers as well and you also have to add a new workflow definition, which is not shown in the figure. This was chosen due to simplicity.
 +
|}
  
== Integrating BPEL services ==
 
  
As already shown in the overview above, SMILA offers the possibility to integrate your own service or piece of software into SMILA [[SMILA/Glossary#BPEL|BPEL]] workflows.
+
{| class="wikitable" border="1" cellpadding="5"
In SMILA we simply call these workflows [[SMILA/Glossary#P|pipelines]]. A pipeline is the definition of a BPEL process (or workflow) that orchestrates [[SMILA/Glossary#P|pipelets]], [[SMILA/Glossary#P|processing services]], and other BPEL services (e.g. web services).
+
|-
 +
| [[#Integrating alternative implementations of SMILA core components|Integrating alternative implementations of SMILA core components]]
 +
|-valign="top"
 +
| This scenario is particularly intended for the experienced (SMILA) developer and comprises the possibility to exchange existing implementations of the SMILA core components by your own implementations.
 +
|-valign="top"
 +
| [[image:Provide-Alternative-To-Core-Component_1.1.0.png|900px]]  
 +
|-valign="top"
 +
| The figure above demonstrates one of the SMILA core components -- ''data store'' -- may be exchanged by your own implementation. This serves as examples only, that is, you may also exchange other core components such as the [[SMILA/Glossary#B|blackboard service]].
 +
|}
  
There are several options on how to achieve this:
+
{{Note|The above figures exemplary demonstrate at which levels in the [http://www.eclipse.org/smila SMILA architecture] an integration of new components is applicable. However, for simplicity reason, we restricted the above figures to the index processing chain while completely ignoring the search processing chain that offers the same integration options (except for the integration of agents and crawlers), but is currently not in the focus of this page.
 +
}}
  
* [[#Simple: Integrating web services|Simple]]: The easiest method to add functionality is to invoke a web service by using the standard functionality of BPEL. However, the disadvantage is that not all data of SMILA [[SMILA/Glossary#R|records]] are accessible if you opt for this method of integration.
+
== Conventions ==
* [[#Default: Integrating local SMILA pipelets or processing services|Default]]: The recommended way to integrate additional functionality in SMILA is to provide Java implementations of one of two available interfaces that allow for an easy creation of the above mentioned [[SMILA/Glossary#P|pipelets]] and [[SMILA/Glossary#P|processing services]].
+
* [[#Advanced: Integrating remote SMILA processing services|Advanced]]: This method extends the default mechanism by providing an alternative procedure for integrating processing services that do not run in the same OSGi runtime as the BPEL workflow but in another OSGI runtime that may even run on a remote machine.
+
  
==== Simple: Integrating web services ====
+
=== Handling of Character Encoding ===
The simplest way of integrating additional functionality in SMILA is to call a web service, which is a standard BPEL workflow engine functionality independent of SMILA. However, there are some limitations concerning the input and result data to/from web services: The workflow object (a DOM object) that enters the BPEL workflow in SMILA contains only the record [[SMILA/Glossary#I|IDs]] by default. That means [[SMILA/Glossary#R|records]] and the data contained therein - [[SMILA/Glossary#A|attributes]], [[SMILA/Glossary#A|annotations]], and [[SMILA/Glossary#A|attachments]] - are '''not''' accessible from a BPEL workflow because it can only access and use the values contained in the BPEL workflow object.
+
  
To overcome this restriction you can add additional data to the workflow object by adding filters in the configuration file located at <tt>org.eclipse.smila.blackboard/RecordFilters.xml</tt>. These filter rules define which [[SMILA/Glossary#A|attributes]] and [[SMILA/Glossary#A|annotations]] should be copied to the workflow object to make them accessible in the BPEL workflow. Additionally, you should not forget to include all attributes and annotations in the <tt>RecordFilters.xml</tt> file that you wish to write data to. Though filters work on attributes and annotations there is no possibility to access attachments of records because binary data is not reasonable in DOM.  
+
To make processing of data in SMILA easier: If external data must be converted to a string (e.g. an attribute value), the crawler/fetcher or any other component accessing an external data source should try everything that is possible to ensure that the conversion is done using the correct encoding. For example, HTTP clients should use the encoding reported by the HTTP server. If the data source does not provide information about the character encoding, you can use the class <tt>org.eclipse.smila.utils.file.EncodingHelper</tt> that tries to convert a byte[] to a string by trying to detect the correct encoding from a <tt>byte[]</tt> by checking BOMs or checking XML and HTML content for instructions and finally by using UTF-8 or, if this fails, the default platform encoding. You may find this helpful.
  
===== Examples =====
+
On the other hand, if valid string data must be converted to a byte[] (e.g. if it is stored as a attachment after pipelet processing), the conversion must always use UTF-8 encoding.
A good example for this use case is the integration of the [http://www.languageweaver.com/home.asp Language Weaver] web service. The Language Weaver Translation Server provides a web service interface that translates a text into another language. This service could easily be used within SMILA to extend its functionality.
+
  
===== Further reading =====
+
== Integrating a SMILA pipelet ==
Please consult the following how-to tutorials for a more detailed technical description:
+
  
* [[SMILA/Development_Guidelines/How to filter and access record data in BPEL|How to filter and access record data in BPEL]]
+
As already shown in the overview above, SMILA offers the possibility to integrate your own service or piece of software into SMILA [[SMILA/Glossary#BPEL|BPEL]] workflows.
* [[SMILA/Development_Guidelines/How to integrate the HelloWorld webservice in BPEL|How to integrate the HelloWorld webservice in BPEL]]
+
In SMILA we simply call these workflows [[SMILA/Glossary#P|pipelines]]. A pipeline is the definition of a BPEL process (or workflow) that orchestrates [[SMILA/Glossary#P|pipelets]] and other BPEL services (e.g. web services).
 
+
==== Default: Integrating local SMILA pipelets or processing services ====
+
The default and thus recommended technique to integrate functionality or software in SMILA is to provide a [[SMILA/Glossary#P|pipelet]] or a [[SMILA/Glossary#P|processing service]] that runs in the same OSGi runtime as the BPEL workflow engine. Pipelets are easier to implement as they require only standard Java knowledge but lack the functionality of processing services to be shared between multiple pipelines and to be accessed by multiple threads. Another difference is that the lifecycle and configuration of pipelets is managed by the workflow engine whereas processing services are handled by OSGi runtime. For further information on pipelets and processing services refer to [[SMILA/Documentation/Pipelets_and_ProcessingServices|Pipelets and ProcessingServices]].
+
  
The above mentioned restriction of integrated web services using the BPEL default engine functionality does not apply to pipelets and processing services. Both have full access to SMILA [[SMILA/Glossary#R|records]] by using the [[SMILA/Glossary#B|blackboard service]], which makes it easy to read, modify, and store [[SMILA/Glossary#R|records]].
+
The default and thus recommended technique to integrate simple and small functionality or software in SMILA is to provide a [[SMILA/Glossary#P|pipelet]]. Pipelets are easy to implement as they require only standard Java knowledge. They are not shared between multiple pipelines, even multiple invocations of a Pipelet in the same Pipeline do not share the same instance. The lifecycle and configuration of pipelets is managed by the workflow engine, not by OSGi runtime. Pipelets have full access to SMILA [[SMILA/Glossary#R|records]] by using the [[SMILA/Glossary#B|blackboard service]], which makes it easy to read, modify, and store [[SMILA/Glossary#R|records]]. For further information on pipelets refer to the [[SMILA/Documentation/Pipelets|Pipelets documentation]].
  
In general pipelets and processing services follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:
+
In general pipelets follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:
 
* Read the configuration (optional)
 
* Read the configuration (optional)
 
* Read input data from blackboard (optional)
 
* Read input data from blackboard (optional)
Line 70: Line 76:
 
* Write result data to blackboard (optional)
 
* Write result data to blackboard (optional)
  
In terms of the pipelet or processing service that implements the business logic you are totally free to use any desired technology. Some of the posibilities include:
+
In terms of the pipelet that implements the business logic you are totally free to use any desired technology. Some of the posibilities include:
 
* Using POJOs (For examples refer to the [[SMILA/Documentation/Bundle_org.eclipse.smila.processing.pipelets.xmlprocessing|XML processing pipelets]])
 
* Using POJOs (For examples refer to the [[SMILA/Documentation/Bundle_org.eclipse.smila.processing.pipelets.xmlprocessing|XML processing pipelets]])
* Using any local available OSGi service, even other processing services (For an example refer to the [[SMILA/Documentation/AperturePipelet|Aperture pipelet]] that uses the [[SMILA/Documentation/ApertureMimeTypeIdentifier|ApertureMimeTypeIdentifier]] processing service.)
+
* Using any local available OSGi service (For an example refer to the [[SMILA/Documentation/Bundle org.eclipse.smila.processing.pipelets#org.eclipse.smila.processing.pipelets.MimeTypeIdentifyPipelet|MimeTypeIdentifyPipelet]] which uses a MimeTypeIdentifier service)
* Using other technologies such as JNI, RMI, or CORBA to integrate remote or non Java components (As an example consider the integration of [http://www.oracle.com/technologies/embedded/outside-in.html Oracle Outside In Technology].)
+
* Using other technologies such as JNI, RMI, or CORBA to integrate remote or non Java components.
  
 
===== Examples =====
 
===== Examples =====
  
* Typical examples for pipelets are the [[SMILA/Documentation/Bundle_org.eclipse.smila.processing.pipelets.xmlprocessing|XML processing pipelets]]. These lightweight pipelets are used for XML processing (e.g. XSL transformation). Each pipeline uses its own [[SMILA/Glossary#P|pipelet]] instance.
+
* Typical examples for pipelets are the [[SMILA/Documentation/Bundle org.eclipse.smila.processing.pipelets.xmlprocessing|XML processing pipelets]]. These lightweight pipelets are used for XML processing (e.g. XSL transformation). Each pipeline uses its own [[SMILA/Glossary#P|pipelet]] instance.
* A good example for a processing service is the [[SMILA/Documentation/LuceneIndexService|Lucene index service]]. It provides functionality to index [[SMILA/Glossary#R|records]] in [http://lucene.apache.org/java/docs/index.html Lucene] indexes and can be used from multiple [[SMILA/Glossary#P|pipelines]] in parallel.
+
  
 
===== Further reading =====
 
===== Further reading =====
 
Please consult the following how-to tutorials for a more detailed technical description:
 
Please consult the following how-to tutorials for a more detailed technical description:
 
* [[SMILA/Development_Guidelines/How to write a Pipelet|How to write a pipelet]]
 
* [[SMILA/Development_Guidelines/How to write a Pipelet|How to write a pipelet]]
* [[SMILA/Development_Guidelines/How to write a ProcessingService|How to write a processing service]]
+
* [[SMILA/Development_Guidelines/How to integrate the HelloWorld webservice as a Pipelet|How to integrate the HelloWorld '''web service''' as a pipelet]]
* [[SMILA/Development_Guidelines/How to integrate the HelloWorld webservice as a Pipelet|How to integrate the HelloWorld web service as a pipelet]]
+
  
==== Advanced: Integrating remote SMILA processing services ====
+
== Integrating own workers ==
tbd.
+
 
 +
When the desired changes to the functionality are not simple and small or the desired functionality requires more than just accessing the records, it is recommended to integrate your functionality as new [[SMILA/Glossary#W|workers]].
 +
These workers have to be defined in the asynchronous job processing configuration and can then be integrated in workflows.
 +
 
 +
=== Description ===
 +
A worker is an OSGi service, that implements the Worker interface, i.e. provides a name (to collect tasks and to be identified in workflows) and offers a threadsafe <tt>perform</tt> method, which takes a <tt>TaskContext</tt> object (generated by the WorkerManager) as an argument.
 +
 
 +
Normally the worker will handle a bulk of records in the perform method, manipulate the contained records, may access external services or stores and create new bulks of records.
 +
 
 +
The worker does not have to handle tasks or check if tasks are available. These menial tasks are done by the WorkerManager that cares for all registered workers, handles the invocations of their perform methods and copes with input and output bulks, up-scaling or exception handling. The worker just has to provide the perfom method to deal with the records.
 +
 
 +
If the worker requires direct access to the JobManager/TaskManager, e.g. because it is an initial worker and has to get initial tasks to start workflow runs without being triggered by previous workers, it must not register itself as a Worker service but has to tackle TaskManager and JobManager itself. Be careful, this is tedious work! So if possible stick to the Worker interface and let everything else be handled by the WorkerManager.
 +
 
 +
=== Further reading ===
 +
Please consult at least the following pages about asynchronous job processing:
 +
* [[SMILA/Documentation/JobManager|JobManager]]
 +
* [[SMILA/Documentation/WorkerManager|WorkerManager]]
 +
* [[http://build.eclipse.org/rt/smila/javadoc/current/index.html?org/eclipse/smila/taskworker/package-summary.html|TaskWorker JavaDoc]]
 +
* [[SMILA/Development_Guidelines/How_to_write_a_Worker|How to write a '''worker''']]
  
 
== Integrating data sources ==
 
== Integrating data sources ==
  
Due to the architecture of the SMILA connectivity framework it is easy to include additional data sources by providing appropriate implementations of [[SMILA/Glossary#A|agents]] and/or [[SMILA/Glossary#C|crawlers]].  
+
If you like to integrate data sources, you should do so by writing a new Importing-Wokrflow along with apropriate crawler/fetcher workers (and eventually an extractor worker) as is described in [[SMILA/Documentation/HowTo/How_to_add_a_new_Data_Source_to_the_importing_framework|HowTo add a new Data Source to the Importing Framework]].
  
=== Examples ===
+
This also involves integration of own workers (see [[#Integrating own workers|Integrating own workers]] above).
* A typical agent is a FilesystemWatcher. It monitors a folder (or folder structure) for changes (creation, modification, or deletion of files/folders) and reports those actions to SMILA.
+
* Typical crawlers are the FilesystemCrawler or the WebCrawler. The first iterates over a folder structure and sends all encountered files to SMILA. The latter traverses the links of HTML pages, follows links to other HTML pages, and sends these pages as well as other resources (images, PDF files, etc.) to SMILA.
+
  
 
=== Further reading ===
 
=== Further reading ===
 
Please consult the following how-to tutorials for a more detailed technical description:
 
Please consult the following how-to tutorials for a more detailed technical description:
* [[SMILA/Development_Guidelines/How to implement an agent|How to implement an agent]]
+
* [[SMILA/Documentation/HowTo/How_to_add_a_new_Data_Source_to_the_importing_framework|HowTo add a new Data Source to the Importing Framework]]
* [[SMILA/Development_Guidelines/How to implement a crawler|How to implement a crawler]]
+
* [[#Integrating own workers|Integrating own workers]]
  
 
== Integrating alternative implementations of SMILA core components ==
 
== Integrating alternative implementations of SMILA core components ==
  
The component-based architecture of SMILA even allows you to provide your own implementations of SMILA core components. More info coming soon...
+
The component-based architecture of SMILA even allows you to provide your own implementations of SMILA core components, since these are implemented as OSGi service components (see [[SMILA/Development_Guidelines/Declarative_Services|Declarativ Services]]) and can thus be exchanged in a standard way.
 +
* Include a new plug-in that exposes a service implementing the interface of the core component (e.g. <tt>ObjectStoreService</tt>)
 +
* modify the config.ini configuration file in SMILA.application to include and start the new plug-in instead of the plug-in provided by SMILA core.
 +
* build your application and run it.
  
 
=== Examples ===
 
=== Examples ===
  
A typical example is an alternative implementation of the <tt>DeltaIndexingManager</tt> that does not store its state in memory but in the file system or in a database.
+
A typical example could be an alternative implementation of the <tt>ObjectStoreService</tt> that does not store the objects in the file system but in memory or in a database.
  
 
=== Further reading ===
 
=== Further reading ===
tbd.
+
* [[SMILA/Development_Guidelines/Declarative_Services]]
 +
* [http://eclipse.org/equinox Eclipse Equinox]
 +
* [http://live.eclipse.org/node/407 Webinar: Getting started with OSGi]
 +
 
 
[[Category:SMILA]]
 
[[Category:SMILA]]

Latest revision as of 13:58, 23 October 2012

This page summarizes the different types and complexity levels for the integration of components in SMILA.

Contents

[edit] Introduction

Due to its architecture SMILA allows for the easy integration of third-party components into its framework. Actually there are four different possible integration scenarios available that are depicted in the following table.

Integrating a SMILA pipelet
This is probably the most frequently used integration scenario. It allows for the integration or exchange of functionality (services, 3rd party software, etc.) used to process records in the workflow engine.
Integrate-Service 1.1.0.png
The figure demonstrates how you can integrate the functionality of your service or your piece of software to SMILA by adding it to the workflow engine.


Integrating workers
Integrating your own worker implementation is another common scenario for adding functionality or adapting workflows to SMILA.
Integrate-Worker 1.1.0.png
The figure above exemplary shows how you can add your own worker implementation to SMILA. Please note that you also have to add the worker to the job manager configuration files (workers.json, workflows.json) and add your worker as an OSGi service to activate it.


Integrating data sources
Integrating a data source by implementing your own crawler is another scenario for adding functionality to SMILA. By doing so, further data sources can be unlocked to provide additional input to SMILA.
Integrate-DataSources 1.1.0.png
The figure above exemplary shows how you can add your own crawler implementation to SMILA. Please note that eventually you have to add a new Fetcher or extractors or further workers as well and you also have to add a new workflow definition, which is not shown in the figure. This was chosen due to simplicity.


Integrating alternative implementations of SMILA core components
This scenario is particularly intended for the experienced (SMILA) developer and comprises the possibility to exchange existing implementations of the SMILA core components by your own implementations.
Provide-Alternative-To-Core-Component 1.1.0.png
The figure above demonstrates one of the SMILA core components -- data store -- may be exchanged by your own implementation. This serves as examples only, that is, you may also exchange other core components such as the blackboard service.
Note.png
The above figures exemplary demonstrate at which levels in the SMILA architecture an integration of new components is applicable. However, for simplicity reason, we restricted the above figures to the index processing chain while completely ignoring the search processing chain that offers the same integration options (except for the integration of agents and crawlers), but is currently not in the focus of this page.


[edit] Conventions

[edit] Handling of Character Encoding

To make processing of data in SMILA easier: If external data must be converted to a string (e.g. an attribute value), the crawler/fetcher or any other component accessing an external data source should try everything that is possible to ensure that the conversion is done using the correct encoding. For example, HTTP clients should use the encoding reported by the HTTP server. If the data source does not provide information about the character encoding, you can use the class org.eclipse.smila.utils.file.EncodingHelper that tries to convert a byte[] to a string by trying to detect the correct encoding from a byte[] by checking BOMs or checking XML and HTML content for instructions and finally by using UTF-8 or, if this fails, the default platform encoding. You may find this helpful.

On the other hand, if valid string data must be converted to a byte[] (e.g. if it is stored as a attachment after pipelet processing), the conversion must always use UTF-8 encoding.

[edit] Integrating a SMILA pipelet

As already shown in the overview above, SMILA offers the possibility to integrate your own service or piece of software into SMILA BPEL workflows. In SMILA we simply call these workflows pipelines. A pipeline is the definition of a BPEL process (or workflow) that orchestrates pipelets and other BPEL services (e.g. web services).

The default and thus recommended technique to integrate simple and small functionality or software in SMILA is to provide a pipelet. Pipelets are easy to implement as they require only standard Java knowledge. They are not shared between multiple pipelines, even multiple invocations of a Pipelet in the same Pipeline do not share the same instance. The lifecycle and configuration of pipelets is managed by the workflow engine, not by OSGi runtime. Pipelets have full access to SMILA records by using the blackboard service, which makes it easy to read, modify, and store records. For further information on pipelets refer to the Pipelets documentation.

In general pipelets follow the same (sometimes optional) logical steps (of course this depends highly on the business logic to be executed). These steps are:

  • Read the configuration (optional)
  • Read input data from blackboard (optional)
  • Execute the business logic
  • Write result data to blackboard (optional)

In terms of the pipelet that implements the business logic you are totally free to use any desired technology. Some of the posibilities include:

  • Using POJOs (For examples refer to the XML processing pipelets)
  • Using any local available OSGi service (For an example refer to the MimeTypeIdentifyPipelet which uses a MimeTypeIdentifier service)
  • Using other technologies such as JNI, RMI, or CORBA to integrate remote or non Java components.
[edit] Examples
  • Typical examples for pipelets are the XML processing pipelets. These lightweight pipelets are used for XML processing (e.g. XSL transformation). Each pipeline uses its own pipelet instance.
[edit] Further reading

Please consult the following how-to tutorials for a more detailed technical description:

[edit] Integrating own workers

When the desired changes to the functionality are not simple and small or the desired functionality requires more than just accessing the records, it is recommended to integrate your functionality as new workers. These workers have to be defined in the asynchronous job processing configuration and can then be integrated in workflows.

[edit] Description

A worker is an OSGi service, that implements the Worker interface, i.e. provides a name (to collect tasks and to be identified in workflows) and offers a threadsafe perform method, which takes a TaskContext object (generated by the WorkerManager) as an argument.

Normally the worker will handle a bulk of records in the perform method, manipulate the contained records, may access external services or stores and create new bulks of records.

The worker does not have to handle tasks or check if tasks are available. These menial tasks are done by the WorkerManager that cares for all registered workers, handles the invocations of their perform methods and copes with input and output bulks, up-scaling or exception handling. The worker just has to provide the perfom method to deal with the records.

If the worker requires direct access to the JobManager/TaskManager, e.g. because it is an initial worker and has to get initial tasks to start workflow runs without being triggered by previous workers, it must not register itself as a Worker service but has to tackle TaskManager and JobManager itself. Be careful, this is tedious work! So if possible stick to the Worker interface and let everything else be handled by the WorkerManager.

[edit] Further reading

Please consult at least the following pages about asynchronous job processing:

[edit] Integrating data sources

If you like to integrate data sources, you should do so by writing a new Importing-Wokrflow along with apropriate crawler/fetcher workers (and eventually an extractor worker) as is described in HowTo add a new Data Source to the Importing Framework.

This also involves integration of own workers (see Integrating own workers above).

[edit] Further reading

Please consult the following how-to tutorials for a more detailed technical description:

[edit] Integrating alternative implementations of SMILA core components

The component-based architecture of SMILA even allows you to provide your own implementations of SMILA core components, since these are implemented as OSGi service components (see Declarativ Services) and can thus be exchanged in a standard way.

  • Include a new plug-in that exposes a service implementing the interface of the core component (e.g. ObjectStoreService)
  • modify the config.ini configuration file in SMILA.application to include and start the new plug-in instead of the plug-in provided by SMILA core.
  • build your application and run it.

[edit] Examples

A typical example could be an alternative implementation of the ObjectStoreService that does not store the objects in the file system but in memory or in a database.

[edit] Further reading