Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/HowTo/How to integrate the HelloWorld webservice as a Pipelet"

Line 195: Line 195:
 
* Launch SMILA by calling either <tt>eclipse.exe -console</tt> or <tt>launch.cmd</tt>.  
 
* Launch SMILA by calling either <tt>eclipse.exe -console</tt> or <tt>launch.cmd</tt>.  
  
If SMILA is running, you can start a crawling job as described in [[SMILA/Development_Guidelines#Run_and_manage_the_connectivity_framework|Run and manage the connectivity framework]] beginning at step 5).
+
If SMILA is running, you can start a crawling job as described in [[SMILA/Development_Guidelines#Run_and_manage_the_connectivity_framework|Run and manage the connectivity framework]] beginning at step 5.
While crawling your data source you can already search for indexed documents. Open your browser and navigate to [http://localhost:8080/AnyFinder/SearchForm  http://localhost:8080/AnyFinder/SearchForm] and execute a search. In the result table take a look at the attribute '''Title'''. Every '''Title''' should now have the suffix <tt>"modified by HelloWorldPipelet"</tt>, as this was added by the pipelet.
+
While crawling your data source you can already search for indexed documents. Open your browser, navigate to [http://localhost:8080/AnyFinder/SearchForm  http://localhost:8080/AnyFinder/SearchForm] and execute a search. In the result table take a look at the attribute '''Title'''. Every '''Title''' should now have the suffix <tt>"modified by HelloWorldPipelet"</tt>, as this was added by the pipelet.
  
 
=== Troubleshooting ===
 
=== Troubleshooting ===
If there are any problems please take a look at the logfiles <tt>SMILA.log</tt> and <tt>/workspace/.metadata/.log</tt> and feel free to ask for support at the [http://www.eclipse.org/smila/newsgroup.php SMILA Newsgroup].
+
If there are any problems please take a look at the log files <tt>SMILA.log</tt> and <tt>/workspace/.metadata/.log</tt> and feel free to ask for support at the [http://www.eclipse.org/smila/newsgroup.php SMILA Newsgroup].
 
+
  
 
[[Category:SMILA]]
 
[[Category:SMILA]]

Revision as of 11:04, 6 October 2008

This page illustrates all steps that need to be performed in order to integrate the HelloWorld web service as a pipelet in SMILA. For general information on how to integrate components and add functionality to SMILA refer to How to integrate a component in SMILA.

Preparations

It may be helpful to first take a look at the SMILA Development guidelines as many topics that are beyond the scope of this tutorial are illustrated there.

Create new bundle

  • Create a new bundle that should contain your pipelet. Follow the instructions on How to create a bundle and use the following settings:
Project name: org.eclipse.smila.sample.pipelet
Plug-in ID: org.eclipse.smila.sample.pipelet
Plug-in Version: 1.0.0
Plug-in Name: Sample Pipelet Bundle
Plug-in Provider: your name or company
  • Then integrate your new bundle into the SMILA build process. Refer to the instructions on How to integrate a new bundle into build process for details.
  • Edit the file META-INF/MANIFEST.MF and add the following import-package dependencies as those are required to implement the basic functionalities of your pipelet:
Import-Package: org.apache.commons.logging;version="1.1.1",
 org.eclipse.smila.blackboard;version="0.5.0",
 org.eclipse.smila.blackboard.path;version="0.5.0",
 org.eclipse.smila.datamodel.id;version="0.5.0",
 org.eclipse.smila.datamodel.record;version="0.5.0",
 org.eclipse.smila.processing;version="0.5.0",
 org.eclipse.smila.processing;version="0.5.0",
 org.eclipse.smila.processing.configuration;version="0.5.0"
  • To make sure that the PipeletTrackerService detects your new pipelet, add the following line to the file META-INF/MANIFEST.MF. This registers the class that will implement your SMILA pipelet:
SMILA-Pipelets: org.eclipse.smila.sample.pipelet.HelloWorldPipelet

Create Java classes from WSDL

coming soon ...

Implementation

  • Create the package org.eclipse.smila.sample.pipelet and the Java class HelloWorldPipelet.
  • Use the following code as a template for your new class. It contains empty method bodies and a reference to the logger. In the following we are going to gradually replace the comments in this file by the corresponding code snippets. For your convenience you may also download the complete zipped source file from HelloWorldPipelet.zip.
package org.eclipse.smila.sample.pipelet
 
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.eclipse.smila.blackboard.BlackboardService;
import org.eclipse.smila.blackboard.path.Path;
import org.eclipse.smila.datamodel.id.Id;
import org.eclipse.smila.datamodel.record.Literal;
import org.eclipse.smila.datamodel.record.RecordFactory;
import org.eclipse.smila.processing.ProcessingException;
import org.eclipse.smila.processing.SimplePipelet;
import org.eclipse.smila.processing.configuration.PipeletConfiguration;
 
public class HelloWorldPipelet implements SimplePipelet {
 
  // additional member variables 
  private final Log _log = LogFactory.getLog(HelloWorldPipelet.class);
 
  public HelloWorldPipelet(){
  }
 
  public void configure(PipeletConfiguration configuration) throws ProcessingException {
    // read the configuration properties
  }
 
  public Id[] process(BlackboardService blackboard, Id[] recordIds) throws ProcessingException {
    // process the recordIds and create a result
  }
}

Read PipeletConfiguration

  • First let's create two member variables that store the names of the input and output attributes as well as string constants for the property names used in the configuration. Replace the comment "// additional member variables " with the following code snippet.
private final String PROP_IN_ATT_NAME= "IN_ATT_NAME";
private final String PROP_OUT_ATT_NAME= "OUT_ATT_NAME";
 
private String _inAttName;
private String _outAttName;
  • Then we are going to fill those members with the attribute names provided by the PipeletConfiguration in method configure(PipeletConfiguration configuration). The method getPropertyFirstValueNotNull(String) will check that the value of the property is not null. If it is null a ProcessingException will be thrown. In addition we should ensure, that the provided string is not empty or consists of whitespaces only. Replace the comment "// read the configuration properties" with the following code snippet.
_inAttName = (String) configuration.getPropertyFirstValueNotNull(PROP_IN_ATT_NAME);
if (_inAttName.trim().length() == 0) {
    throw new ProcessingException("Property " + PROP_IN_ATT_NAME + " must not be an empty String");
}
 
_outAttName = (String) configuration.getPropertyFirstValueNotNull(PROP_OUT_ATT_NAME);
if (_outAttName.trim().length() == 0) {
    throw new ProcessingException("Property " + PROP_OUT_ATT_NAME + " must not be an empty String");
}

Note: Of course it is also possible to store the PipeletConfiguration in a member variable and access the properties as needed in the process(BlackboardService blackboard, Id[] recordIds) method.

Process IDs and implement exception handling

The method process(BlackboardService blackboard, Id[] recordIds) has two parameters:

The HelloWorld pipelet should therefore iterate over the IDs in the parameter recordIds, get the required data from the record identified by the ID, process this data, and store the result in the record. Let's place a try ... catch() block in the for loop to ensure that errors do only interrupt the processing of the current ID. The comments in the code serve as placeholders for the functionality described in the following sections. At the end we return the unmodified input parameter recordIds as the result of the pipelet. Replace the comment "// process the recordIds and create a result" with the following code snippet.

for (Id id : recordIds) {
    try {
        // Read Input Data
 
        // Process Input Data
 
        // Write Output Data
 
    } catch (final Exception ex) {
        if (_log.isErrorEnabled()) {
            _log.error("error during execution of HelloWorldPipelet with record " + id, ex);
        }
    }
} // for
 
return recordIds;

Note: Most of the time the return value of a pipelet is the unmodified input parameter recordIds. However, in some cases a pipelet may filter record IDs or even create new records. Then the return value has to be adopted appropriately.

Read input data

Now we want to read the data of the attribute with the name stored in _inAttName. Therefore we first have to create a Path object with the attribute's name. Before accessing the literal value we check if the record contains an attribute with the given Path. In this tutorial we know that the value of the attribute is a string value, so we directly access the value by calling the method getStringValue(). Replace the comment "// Read Input Data" with the following code snippet.

String inputValue = "";
final Path path = new Path(_inAttName);
if (blackboard.hasAttribute(id, path)) {
  inputValue = blackboard.getLiteral(id, path).getStringValue();
}

Note: Accessing attribute values can be achieved more generically. Therefore you have to check what data type a certain literal contains using the method getDataType(). Then you can use the appropriate getter method to access the raw data.

Process input data

At this point the HelloWorld web service should be called with the parameter inputValue and the result should be stored in the variable outputValue, using the classes generated from WSDL.
Until this tutorial description is complete, simply assign the content of the variable inputValue to variable outputValue and append a constant string value. Replace the comment "// Process Input Data" with the following code snippet.

    String outputValue = inputValue + " modified by HelloWorldPipelet";

Write output data

Finally, we want to store the content of the variable outputValue in the record attribute with the name contained in variable _outAttName. Therefore we have to create a new Literal object and set its value. Then we only need to set this Literal for the current ID on the black board. Replace the comment "// Write Output Data" with the following code snippet.

final Literal literal = RecordFactory.DEFAULT_INSTANCE.createLiteral();
literal.setStringValue(outputValue);
blackboard.setLiteral(id, new Path(_outAttName), literal);

Note: The method commit(Id) of the blackboard service does not need to be called in each pipelet as it is automatically called at the end of the pipeline.

Configuration and invocation in BPEL

In this tutorial we will integrate the HelloWorld pipelet in the SMILA indexing process just before the record is stored in the Lucene index. With this configuration the input for the HelloWorld pipelet will be read from attribute Title and the modified output will be stored in the same attribute, overwriting the previous value.

  • Edit the file configuration/org.eclipse.smila.processing.bpel/pipelines/addpipeline.bpel and add the following right between the <extensionActivity name="convertDocument"> and the <extensionActivity name="invokeLuceneService"> section.
<extensionActivity name="invokeHelloWorldPipelet">
    <proc:invokePipelet>
        <proc:pipelet class="org.eclipse.smila.sample.pipelet.HelloWorldPipelet" />
        <proc:variables input="request" output="request" />
        <proc:PipeletConfiguration>
            <proc:Property name="IN_ATT_NAME">
                <proc:Value>Title</proc:Value>
            </proc:Property>
            <proc:Property name="OUT_ATT_NAME">
                <proc:Value>Title</proc:Value>
            </proc:Property>
        </proc:PipeletConfiguration>       
    </proc:invokePipelet>
</extensionActivity>

Test your pipelet

To test your pipelet, you have to include the bundle in the OSGi launch configuration:

Run SMILA in eclipse

  • Open Run > Open Run Dialog.
  • In the left window select OSGi Framework > SMILA.
  • In the right window expand Workspace and select org.eclipse.smila.sample.pipelet.
  • Set the Default Auto-Start option to true.
  • Click the Apply button.
  • Launch SMILA by clicking the Run button.

Run SMILA as application

  • Copy your bundle to the directory %SMILA_HOME%/plugins.
  • Add the following XML snippet to the file %SMILA_HOME%/features/org.eclipse.smila.feature_1.0.0/feature.xml:

   <plugin
   id="org.eclipse.smila.sample.pipelet"
   download-size="0"
   install-size="0"
   version="1.0.0"
   unpack="false"/>

  • Launch SMILA by calling either eclipse.exe -console or launch.cmd.

If SMILA is running, you can start a crawling job as described in Run and manage the connectivity framework beginning at step 5. While crawling your data source you can already search for indexed documents. Open your browser, navigate to http://localhost:8080/AnyFinder/SearchForm and execute a search. In the result table take a look at the attribute Title. Every Title should now have the suffix "modified by HelloWorldPipelet", as this was added by the pipelet.

Troubleshooting

If there are any problems please take a look at the log files SMILA.log and /workspace/.metadata/.log and feel free to ask for support at the SMILA Newsgroup.

Back to the top