|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| − | == Bundle: org.eclipse.smila.lucene.pipelets.LuceneIndexPipelet ==
| + | {{Note|Deprecated: |
| − | | + | The Lucene integration is outdated. Please take a look at the new Solr integration <tt>org.eclipse.smila.solr</tt> which is described here: [[SMILA/Documentation/Solr]].}} |
| − | === Description ===
| + | |
| − | This pipelet is used to index SMILA records in a Lucene document index. It supports adding, updating, and deleting records.
| + | |
| − | | + | |
| − | === Configuration ===
| + | |
| − | | + | |
| − | ==== Pipelet Configuration ====
| + | |
| − | The LuceneIndexPipelet uses the following parameters to decide on how to handle a record:
| + | |
| − | {| border = 1 | + | |
| − | !Name!!Value!!Description
| + | |
| − | |-
| + | |
| − | |''indexname''||String||The name of the index to work on
| + | |
| − | |-
| + | |
| − | |''executionMode''||ADD or DELETE||''ADD'' - add or update the record, ''DELETE'' - delete the record from the index
| + | |
| − | |-
| + | |
| − | |''allowDoublets''||Boolean||''true'' - allow doublets in the index, i.e. no check if a document already exists is performed, ''false'' - does not allow for doublets in the index, i.e. do a check if a document already exists and if so delete it first. Default is ''false''.
| + | |
| − | |}
| + | |
| − | The parameters can be set either in the record as part of the <tt>_parameters</tt> attribute (which is a single map with these values set), or in the pipelet configuration in BPEL (see below).
| + | |
| − | | + | |
| − | ==== LuceneService Configuration Files ====
| + | |
| − | | + | |
| − | =====<tt>configuration/org.eclipse.smila.search.datadictionary/DataDictionary.xml</tt>=====
| + | |
| − | This file configures the Lucene index structure and the search template. It is possible to define more than one index here. The index to work on is set in the pipeline by the annotation ''indexName''. The defined "FieldNo" is referenced in the file <tt>Mappings.xml</tt>. For more information about the configuration of <tt>DataDictionary.xml</tt>, see Anyfinder documentation.
| + | |
| − | This file is used to prepare the settings for indices. When an index is needed, it is created automatically on demand, and the configuration is loaded from this file to create the index.
| + | |
| − | (Note: The framework will also create a <tt>Datadictionary.xml</tt> file in the workspace. This file contains the information (settings/configuration) for created indices only).
| + | |
| − | The framework will create an index by itself, when a record is configured to be stored in an index. But the user may as well use ''createIndex'' JMX command to create an index explicitly.
| + | |
| − | | + | |
| − | There are some new optional configuration options available for each index configuration which modify the flushing behavior of the underlying Lucene index. If they are not specified, the old logic (flush after every add/delete) is executed.
| + | |
| − | | + | |
| − | <source lang="xml">
| + | |
| − | ...
| + | |
| − | <Index Name="test_index" ForceFlush="false"
| + | |
| − | RAMBufferSize="20" MaxBufferedDocs="100000"
| + | |
| − | MaxBufferedDeleteTerms="100000">
| + | |
| − | ...
| + | |
| − | </source>
| + | |
| − | | + | |
| − | {| border = 1
| + | |
| − | !Name!!Value!!Description
| + | |
| − | |-
| + | |
| − | |''ForceFlush''||Boolean||''true'' - a flush is forced after a document is added/deleted; ''false'' - flushing is configured by the parameters below. Default is ''true''. If set to ''true'' the parameters below have no effect.
| + | |
| − | |-
| + | |
| − | |''RAMBufferSize''|| Integer || The amount of RAM in MB that may be used before the buffered in-memory documents are flushed. If no value is specified, the Lucene default of 16 MB is used. Refer to the Lucene documentation for details.
| + | |
| − | |-
| + | |
| − | |''MaxBufferedDocs''|| Integer || The minimal number of documents required before the buffered in-memory documents are flushed. If no value is specified, Lucene uses flushing by ''RAMBufferSize'' by default. Refer to the Lucene documentation for details.
| + | |
| − | |-
| + | |
| − | |''MaxBufferedDeleteTerms''|| Integer || The minimal number of delete terms required before the buffered in-memory delete terms are applied and flushed. If no value is specified, Lucene uses flushing by ''RAMBufferSize'' by default. Refer to the Lucene documentation for details.
| + | |
| − | |}
| + | |
| − | | + | |
| − | ;Note:
| + | |
| − | For best performance use ''ForceFlush="false"'' and ''RAMBufferSize'' with a meaningful value depending on your system resources. It is important to understand, that if ''ForceFlush="false"'' then added/deleted documents may not be visible in a search until a final flush is triggered. A flush is always performed when the bundle <tt>org.eclipse.smila.lucene</tt> is stopped. It is also possible to perform a manual flush via the JMX Console.
| + | |
| − | | + | |
| − | =====<tt>configuration/org.eclipse.smila.lucene/Mappings.xml</tt>=====
| + | |
| − | This file configures the mapping of attribute and attachment names to Lucene "FieldNo" (defined in <tt>DataDictionary.xml</tt>). It is possible to define mappings for multiple indices in this file, using the same "indexName" as in the file <tt>DataDictionary.xml</tt>.
| + | |
| − | | + | |
| − | ==== Example ====
| + | |
| − | | + | |
| − | The following example was used in the SMILA example application to index records delivered by the File System or Web Crawler respectively.
| + | |
| − | | + | |
| − | '''addpipeline.bpel'''
| + | |
| − | <source lang="xml">
| + | |
| − | ...
| + | |
| − | <extensionActivity>
| + | |
| − | <proc:invokePipelet name="invokeLuceneService">
| + | |
| − | <proc:pipelet class="org.eclipse.smila.lucene.pipelets.LuceneIndexPipelet" />
| + | |
| − | <proc:variables input="request" output="request" />
| + | |
| − | <proc:configuration>
| + | |
| − | <rec:Val key="indexname">test_index</rec:Val>
| + | |
| − | <rec:Val key="executionMode">ADD</rec:Val>
| + | |
| − | <rec:Val key="allowDoublets">false</rec:Val>
| + | |
| − | </proc:configuration>
| + | |
| − | </proc:invokePipelet>
| + | |
| − | </extensionActivity>
| + | |
| − | ...
| + | |
| − | </source>
| + | |
| − | | + | |
| − | '''DataDictionary.xml'''
| + | |
| − | <source lang="xml">
| + | |
| − | <?xml version="1.0" encoding="UTF-8"?>
| + | |
| − | <AnyFinderDataDictionary
| + | |
| − | xmlns="http://www.anyfinder.de/DataDictionary"
| + | |
| − | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
| + | |
| − | xsi:schemaLocation="http://www.anyfinder.de/DataDictionary ../xml/AnyFinderDataDictionary.xsd"
| + | |
| − | >
| + | |
| − | <Index Name="test_index">
| + | |
| − | <Connection xmlns="http://www.anyfinder.de/DataDictionary/Connection" MaxConnections="5"/>
| + | |
| − | <IndexStructure xmlns="http://www.anyfinder.de/IndexStructure" Name="test_index">
| + | |
| − | <Analyzer ClassName="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
| + | |
| − | <IndexField FieldNo="8" IndexValue="true" Name="MimeType"
| + | |
| − | StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="7" IndexValue="true" Name="Size"
| + | |
| − | StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="6" IndexValue="true" Name="Extension"
| + | |
| − | StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="5" IndexValue="true" Name="Title"
| + | |
| − | StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="4" IndexValue="true" Name="Url"
| + | |
| − | StoreText="true" Tokenize="false" Type="Text">
| + | |
| − | <Analyzer ClassName="org.apache.lucene.analysis.WhitespaceAnalyzer"/>
| + | |
| − | </IndexField>
| + | |
| − | <IndexField FieldNo="3" IndexValue="true" Name="LastModifiedDate"
| + | |
| − | StoreText="true" Tokenize="false" Type="Text"/>
| + | |
| − | <IndexField FieldNo="2" IndexValue="true" Name="Path" StoreText="true"
| + | |
| − | Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="1" IndexValue="true" Name="Filename"
| + | |
| − | StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="0" IndexValue="true" Name="Content"
| + | |
| − | StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | </IndexStructure>
| + | |
| − | <Result>
| + | |
| − | <Field FieldNo="0" Name="ID"/>
| + | |
| − | </Result>
| + | |
| − | <Configuration xmlns="http://www.anyfinder.de/DataDictionary/Configuration"
| + | |
| − | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
| + | |
| − | xsi:schemaLocation="http://www.anyfinder.de/DataDictionary/Configuration ../xml/DataDictionaryConfiguration.xsd"
| + | |
| − | >
| + | |
| − | <DefaultConfig>
| + | |
| − | <Field FieldNo="8">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="7">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="6">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="5">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="4">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="3">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="2">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="1">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="0">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <NodeTransformer
| + | |
| − | xmlns="http://www.anyfinder.de/Search/ParameterObjects"
| + | |
| − | Name="urn:ExtendedNodeTransformer">
| + | |
| − | <ParameterSet xmlns="http://www.brox.de/ParameterSet"/>
| + | |
| − | </NodeTransformer>
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR"
| + | |
| − | Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | </DefaultConfig>
| + | |
| − | </Configuration>
| + | |
| − | </Index>
| + | |
| − | </AnyFinderDataDictionary>
| + | |
| − | </source>
| + | |
| − | | + | |
| − | '''Mappings.xml'''
| + | |
| − | <source lang="xml">
| + | |
| − | <?xml version="1.0" encoding="utf-8" ?>
| + | |
| − | <Mappings xmlns="http://www.eclipse.org/smila/lucene"
| + | |
| − | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
| + | |
| − | xsi:noNamespaceSchemaLocation="schemas/Mappings.xsd"
| + | |
| − | >
| + | |
| − | <Mapping indexName="test_index">
| + | |
| − | <Attributes>
| + | |
| − | <Attribute name="Filename" fieldNo="1" />
| + | |
| − | <Attribute name="Path" fieldNo="2" />
| + | |
| − | <Attribute name="LastModifiedDate" fieldNo="3" />
| + | |
| − | <Attribute name="Url" fieldNo="4" />
| + | |
| − | <Attribute name="Title" fieldNo="5" />
| + | |
| − | <Attribute name="Extension" fieldNo="6" />
| + | |
| − | <Attribute name="Size" fieldNo="7" />
| + | |
| − | <Attribute name="MimeType" fieldNo="8" />
| + | |
| − | </Attributes>
| + | |
| − | <Attachments>
| + | |
| − | <Attachment name="Content" fieldNo="0" />
| + | |
| − | </Attachments>
| + | |
| − | </Mapping>
| + | |
| − | </Mappings>
| + | |
| − | </source>
| + | |
| − | | + | |
| − | | + | |
| − | [[Category:SMILA]] [[Category:SMILA/Pipelet]] | + | |
