|
|
| (31 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | [[Category:SMILA]]
| + | #REDIRECT [[SMILA/Documentation_for_5_Minutes_to_Success]] |
| − | [[Category:HowTo]]
| + | |
| − | | + | |
| − | This page contains installation instructions for the SMILA application which will help you taking the first steps with SMILA.
| + | |
| − | | + | |
| − | == Download and unpack SMILA ==
| + | |
| − | | + | |
| − | [http://www.eclipse.org/smila/downloads.php Download] the SMILA package and unpack it to an arbitrary folder. This will result in the following folder structure:
| + | |
| − | | + | |
| − | [[Image:Installation.png]]
| + | |
| − | | + | |
| − | == Check the preconditions ==
| + | |
| − | | + | |
| − | To be able to follow the steps below, check the following preconditions:
| + | |
| − | | + | |
| − | * You will have to provide a JRE executable to be able to run SMILA. The JVM version should be at least Java 5. <br> Either:
| + | |
| − | ** add the path of your local JRE executable to the PATH environment variable <br>or<br>
| + | |
| − | ** add the argument <tt>-vm <path/to/jre/executable></tt> right at the top of the file <tt>SMILA.ini</tt>. <br>Make sure that <tt>-vm</tt> is indeed the first argument in the file and that there is a line break after it. It should look similar to the following:
| + | |
| − | <tt>
| + | |
| − | -vm
| + | |
| − | d:/java/jre6/bin/java
| + | |
| − | ...
| + | |
| − | </tt>
| + | |
| − | * Since we are going to use <tt>Jconsole</tt> as the JMX client later in this tutorial, it is recommended to install and use a Java SE Development Kit (JDK) and not just a Java SE Runtime Environment (JRE) because the latter does not include this application.
| + | |
| − | *When using the Linux distributable of SMILA, make sure that the file <tt>SMILA</tt> has executable permissions. If not, set it by running the following command in a console:
| + | |
| − | <tt>
| + | |
| − | chmod +x ./SMILA
| + | |
| − | </tt>
| + | |
| − | | + | |
| − | == Start SMILA ==
| + | |
| − | | + | |
| − | To start the SMILA engine, simply double-click the <tt>SMILA.exe</tt> executable. Alternatively, open a command line, navigate to the directory where you extracted the files to, and call the <tt>SMILA.exe</tt> executable. Wait until the engine has been fully started. If everything is fine, you should see output similar to the one on the following screenshot:
| + | |
| − | | + | |
| − | [[Image:Smila-console-0.8.0.png]]
| + | |
| − | | + | |
| − | == Check the log file ==
| + | |
| − | Open the SMILA log file in an editor of your choice to find out what is happening in the background. This file is named <tt>SMILA.log</tt> and can be found in the same directory as the SMILA executable.
| + | |
| − | | + | |
| − | [[Image:Smila-log.png]]
| + | |
| − | | + | |
| − | == Control crawler jobs ==
| + | |
| − | As soon as the SMILA engine is up and running we can start a crawler job. Crawler jobs can be managed via the JMX protocol, therefore you can connect to SMILA using any JMX client you like. We are going to use JConsole in the following because it is included in the Java SE Development Kit.
| + | |
| − | | + | |
| − | Start the JConsole executable in your JDK distribution (<tt><JAVA_HOME>/bin/jconsole</tt>). If the client is up and running, connect to <tt>localhost:9004</tt>.
| + | |
| − | | + | |
| − | [[Image:Jconsole.png-0.8.0.png]]
| + | |
| − | | + | |
| − | Next, switch to the ''MBeans'' tab, expand the ''SMILA'' node in the ''MBeans'' tree on the left-hand side, and click the ''CrawlerController'' node. This node is used to manage and monitor all crawling activities.
| + | |
| − | | + | |
| − | [[Image:Mbeans-overview-0.8.0.png]]
| + | |
| − | | + | |
| − | == Start the File System Crawler ==
| + | |
| − | To start the File System Crawler, select ''SMILA > CrawlerControl > Operations'' on the left-hand side, enter "file" into the text field next to the ''startCrawlerTask'' button, then click the button:
| + | |
| − | | + | |
| − | [[Image:Start-file-crawl-0.8.0.png]]
| + | |
| − | | + | |
| − | You should receive a message similar to the following, indicating that the crawler has been successfully started:
| + | |
| − | | + | |
| − | [[Image:Start-crawl-file-result-0.8.0.png]]
| + | |
| − | | + | |
| − | Now let's check <tt>SMILA.log</tt> to see what has happened:
| + | |
| − | | + | |
| − | [[Image:File-crawl-log.png]]
| + | |
| − | | + | |
| − | The File System Crawler tries to crawl the folder <tt>c:\data</tt> by default. Therefore, it is very likely that do not receive the above results indicating the successful indexing but rather an error message similar to the one shown below, except where it happens that you accidently have a folder <tt>c:\data</tt> on your system that the crawler can find and index:
| + | |
| − | | + | |
| − | <tt>
| + | |
| − | ... ERROR impl.CrawlThread - org.eclipse.smila.connectivity.framework.CrawlerCriticalException: Folder "c:\data" is not found
| + | |
| − | </tt>
| + | |
| − | | + | |
| − | The error message above states that the crawler tried to index a folder at <tt>c:\data</tt> but was not able to find it. To solve this, prepare a folder with sample data, say <tt>/home/johndoe/mydata</tt> or <tt>c:\mydata</tt>, put some dummy text files into it, and configure the File System Crawler to index it as explained in the following section.
| + | |
| − | | + | |
| − | == Configure the File System Crawler ==
| + | |
| − | | + | |
| − | To change the folder which is to be indexed by the File System crawler, open the configuration file at <tt>configuration/org.eclipse.smila.connectivity.framework/file.xml</tt> and modify the ''BaseDir'' attribute by setting its value to an absolute path pointing to the desired directory. Do not forget to save the file:
| + | |
| − | | + | |
| − | <tt>
| + | |
| − | <Process>
| + | |
| − | <BaseDir>/home/johndoe/mydata</BaseDir>
| + | |
| − | ...
| + | |
| − | </Process>
| + | |
| − | </tt>
| + | |
| − | | + | |
| − | Then [[SMILA/Documentation/2011.Simplification/Documentation_for_5_Minutes_to_Success#Start the File System Crawler|start the File System Crawler]] again and check <tt>SMILA.log</tt> for the result.
| + | |
| − | | + | |
| − | {|width="100%" style="background-color:#ffcccc; padding-left:30px;"
| + | |
| − | |
| + | |
| − | Note: Currently, only plain text and HTML files can be crawled and indexed properly.
| + | |
| − | |}
| + | |
| − | | + | |
| − | == Search on index ==
| + | |
| − | To search on the index which was created by the crawlers, point your browser to <tt>http://localhost:8080/SMILA/search</tt>. There are currently two stylesheets from which you can select by clicking the respective links in the upper left corner of the header bar: The ''Default'' stylesheet shows a reduced search form with text fields like ''Query'', ''Result Size'', and ''Index Name'', adequate to query the full-text content of the indexed documents. The ''Advanced'' stylesheet in turn provides a more detailed search form with text fields for meta-data search like for example ''Path'', ''MimeType'', ''Filename'', and other document attributes.
| + | |
| − | | + | |
| − | [[Image:Smila-search-form.png]] | + | |
| − | | + | |
| − | Now, let's try the ''Default'' stylesheet and enter our first simple search using a word that you expect to be contained in your dummy files. In this tutorial, we assume that there is a match for the term "data" in the indexed documents. First, select the index on which you want to search from the ''Indexlist'' column on the left-hand side. Currently, there should be only one in the list, namely an index called "test_index". Note that the selected index name will appear in the ''Index Name'' text field of the search form. Then enter the desired term into the ''Query'' text field. And finally, click ''OK'' to send your query to SMILA. Your result could be similar to the following:
| + | |
| − | | + | |
| − | [[Image:Searching-for-text-in-file.png]]
| + | |
| − | | + | |
| − | Now, let's use the ''Advanced'' stylesheet and search for the name of one the files contained in the indexed folder to check whether it was properly indexed. In our example, we are going to search for a file named <tt>glossary.html</tt>. Click ''Advanced'' to switch to the detailed search form, enter the desired file name into the ''Filename'' text field, then click ''OK'' to submit your search. Your result could be similar to the following:
| + | |
| − | | + | |
| − | [[Image:Searching-by-filename.png]]
| + | |
| − | | + | |
| − | == Configure and run the Web Crawler ==
| + | |
| − | Now that we alreday know how to start and configure the File System Crawler and how to search on indices, configuring and running the Web Crawler is rather straightforward:
| + | |
| − | | + | |
| − | First, let's have a look at the configuration file of the Web Crawler which you can find at <tt>configuration/org.eclipse.smila.connectivity.framework/web.xml</tt>:
| + | |
| − | | + | |
| − | [[Image:Webcrawler-config.png]]
| + | |
| − | | + | |
| − | By default, the Web Crawler is configured to index the URL ''http://wiki.eclipse.org/SMILA''. To change this, set the content of the <tt><Seed></tt> element to the desired web address. If you require further help on this configuration file refer to [[SMILA/Documentation/Web_Crawler|Web Crawler documentation]].
| + | |
| − | | + | |
| − | <tt>
| + | |
| − | <Seeds FollowLinks="NoFollow">
| + | |
| − | <Seed>http://someserver.org/somefile</Seed>
| + | |
| − | </Seeds>
| + | |
| − | </tt>
| + | |
| − | | + | |
| − | To start the crawling process, save the configuration file, go back or reconnect to Jconsole, navigate to ''SMILA'' > ''CrawlerControl'' > ''Operations'', type "web" into the text field next to the <tt>startCrawlerTask</tt> button, then click the button.
| + | |
| − | | + | |
| − | [[Image:Starting-web-crawler-0.8.0.png]]
| + | |
| − | | + | |
| − | Although the default limit for spidered web sites is set to 1,000 in the Web Crawler configuration file, it may take a while for the web crawling job to be finished. Click the <tt>getCrawlerTasksState</tt> button to monitor the job processing if you want to find out when it has finished. This will produce an output similar to the following:
| + | |
| − | | + | |
| − | [[Image:SMILA-One-active-crawl-found-0.8.0.png]]
| + | |
| − | | + | |
| − | If you do not want to wait, you may as well stop the crawling job manually. In order to do this, type "web" into the text field next to the (<tt>stopCrawlerTask</tt>) button, then click this button.
| + | |
| − | | + | |
| − | As soon as the Web Crawler's job has finished, go back to the search form to [[SMILA/Documentation/2011.Simplification/Documentation_for_5_Minutes_to_Success#Search on index|search on the generated index]].
| + | |
| − | | + | |
| − | [[Category:SMILA]]
| + | |
| − | | + | |
| − | == Manage CrawlerController with jmxclient ==
| + | |
| − | | + | |
| − | In addition to managing crawling jobs with JConsole it's also possible to use jmxclient from SMILA distribution. Jmxclient is a console application that allows to manage crawl jobs and create scripts for batch crawlers execution. For more information please check [[SMILA/Documentation/Management#JMX_Client|jmxclient documentation ]]. Jmxclient application is located into <tt>jmxclient</tt> directory. You should use appropriate run script (run.bat or run.sh) to start the application.
| + | |
| − | For example, to start the File System Crawler use the following command:
| + | |
| − | | + | |
| − | <code>
| + | |
| − | ''run crawl file
| + | |
| − | </code>''
| + | |
| − | | + | |
| − | == 5 Minutes for changing the workflow ==
| + | |
| − | In previous sections all data collected by crawlers was processed with the same workflow and indexed into the same index, test_index.
| + | |
| − | It's possible to configure SMILA so that data from different data sources will go through different workflows and will be indexed into different indices. This will require more advanced configuration than before but still is quite simple.
| + | |
| − | Let's create additional workflow for webcrawler records so that webcrawler data will be indexed into separate index, say web_index.
| + | |
| − | | + | |
| − | === Modify Listener rules ===
| + | |
| − | | + | |
| − | First, lets modify the default add rule in Listener and add another rule that will make webcrawler records to be processed by separate BPEL workflow.
| + | |
| − | For more information about Listener, please see the section [[SMILA/Documentation/QueueWorker/Listener|Listener]] of the [[SMILA/Documentation/QueueWorker|QueueWorker]] documentation.
| + | |
| − | Listener configuration is placed at the
| + | |
| − | <tt>configuration/org.eclipse.smila.connectivity.queue.worker.jms/QueueWorkerListenerConfig.xml</tt>
| + | |
| − | Open that file and edit the <tt><Condition></tt> tag of the Default ADD Rule. The result should be as follows:
| + | |
| − | <source lang="xml">
| + | |
| − | <Rule Name="ADD Rule" WaitMessageTimeout="10" Threads="2">
| + | |
| − | <Source BrokerId="broker1" Queue="SMILA.connectivity"/>
| + | |
| − | <Condition>Operation='ADD' and NOT(DataSourceID LIKE 'web%')</Condition>
| + | |
| − | <Task>
| + | |
| − | <Process Workflow="AddPipeline"/>
| + | |
| − | </Task>
| + | |
| − | </Rule>
| + | |
| − | </source>
| + | |
| − | Now add the following new rule to this file:
| + | |
| − | <source lang="xml">
| + | |
| − | <Rule Name="Web ADD Rule" WaitMessageTimeout="10" Threads="2">
| + | |
| − | <Source BrokerId="broker1" Queue="SMILA.connectivity"/>
| + | |
| − | <Condition>Operation='ADD' and DataSourceID LIKE 'web%'</Condition>
| + | |
| − | <Task>
| + | |
| − | <Process Workflow="AddWebPipeline"/>
| + | |
| − | </Task>
| + | |
| − | </Rule>
| + | |
| − | </source>
| + | |
| − | Notice that we modified condition in the ADD Rule to skip webcrawler data. Webcrawler data will be processed by new Web ADD Rule.
| + | |
| − | Web ADD Rule defines that webcrawler data will be processed by AddWebPipeline workflow, so next we need to create AddWebPipeline workflow.
| + | |
| − | | + | |
| − | === Create workflow for the BPEL WorkflowProcessor ===
| + | |
| − | We need to add the AddWebPipeline workflow to BPEL WorkflowProcessor. For more information about BPEL WorkflowProcessor please check the [[SMILA/Documentation/BPEL_Workflow_Processor|BPEL WorkflowProcessor]] documentation.
| + | |
| − | BPEL WorkflowProcessor configuration files are placed at the <tt>configuration/org.eclipse.smila.processing.bpel/pipelines</tt> directory.
| + | |
| − | There is a file <tt>addpipeline.bpel</tt> that defines AddPipeline process. Let's create the <tt>addwebpipeline.bpel</tt> file that will define AddWebPipeline process and put the following code into it:
| + | |
| − | <source lang="xml">
| + | |
| − | <?xml version="1.0" encoding="utf-8" ?>
| + | |
| − | <process name="AddWebPipeline" targetNamespace="http://www.eclipse.org/smila/processor"
| + | |
| − | xmlns="http://docs.oasis-open.org/wsbpel/2.0/process/executable"
| + | |
| − | xmlns:xsd="http://www.w3.org/2001/XMLSchema"
| + | |
| − | xmlns:proc="http://www.eclipse.org/smila/processor"
| + | |
| − | xmlns:rec="http://www.eclipse.org/smila/record">
| + | |
| − |
| + | |
| − | <import location="processor.wsdl" namespace="http://www.eclipse.org/smila/processor"
| + | |
| − | importType="http://schemas.xmlsoap.org/wsdl/" />
| + | |
| − |
| + | |
| − | <partnerLinks>
| + | |
| − | <partnerLink name="Pipeline" partnerLinkType="proc:ProcessorPartnerLinkType" myRole="service" />
| + | |
| − | </partnerLinks>
| + | |
| − |
| + | |
| − | <extensions>
| + | |
| − | <extension namespace="http://www.eclipse.org/smila/processor" mustUnderstand="no" />
| + | |
| − | </extensions>
| + | |
| − |
| + | |
| − | <variables>
| + | |
| − | <variable name="request" messageType="proc:ProcessorMessage" />
| + | |
| − | </variables>
| + | |
| − |
| + | |
| − | <sequence>
| + | |
| − | <receive name="start" partnerLink="Pipeline" portType="proc:ProcessorPortType" operation="process"
| + | |
| − | variable="request" createInstance="yes" />
| + | |
| − |
| + | |
| − | <!-- only process text based content, skip everything else -->
| + | |
| − | <if name="conditionIsText">
| + | |
| − | <condition>
| + | |
| − | contains($request.records/rec:Record[1]/rec:A[@n="MimeType"]/rec:L/rec:V, "text/")
| + | |
| − | </condition>
| + | |
| − | <sequence name="processTextBasedContent">
| + | |
| − |
| + | |
| − | <!-- extract txt from html files -->
| + | |
| − | <if name="conditionIsHtml">
| + | |
| − | <condition>
| + | |
| − | ($request.records/rec:Record[1]/rec:A[@n="MimeType"]/rec:L/rec:V[contains(., "text/html")])
| + | |
| − | or
| + | |
| − | ($request.records/rec:Record[1]/rec:A[@n="MimeType"]/rec:L/rec:V[contains(., "text/xml")])
| + | |
| − | </condition>
| + | |
| − | </if>
| + | |
| − |
| + | |
| − | <extensionActivity>
| + | |
| − | <proc:invokePipelet name="invokeHtml2Txt">
| + | |
| − | <proc:pipelet class="org.eclipse.smila.processing.pipelets.HtmlToTextPipelet" />
| + | |
| − | <proc:variables input="request" output="request" />
| + | |
| − | <proc:PipeletConfiguration>
| + | |
| − | <proc:Property name="inputType">
| + | |
| − | <proc:Value>ATTACHMENT</proc:Value>
| + | |
| − | </proc:Property>
| + | |
| − | <proc:Property name="outputType">
| + | |
| − | <proc:Value>ATTACHMENT</proc:Value>
| + | |
| − | </proc:Property>
| + | |
| − | <proc:Property name="inputName">
| + | |
| − | <proc:Value>Content</proc:Value>
| + | |
| − | </proc:Property>
| + | |
| − | <proc:Property name="outputName">
| + | |
| − | <proc:Value>Content</proc:Value>
| + | |
| − | </proc:Property>
| + | |
| − | <proc:Property name="meta:title">
| + | |
| − | <proc:Value>Title</proc:Value>
| + | |
| − | </proc:Property>
| + | |
| − | </proc:PipeletConfiguration>
| + | |
| − | </proc:invokePipelet>
| + | |
| − | </extensionActivity>
| + | |
| − |
| + | |
| − | <extensionActivity>
| + | |
| − | <proc:invokePipelet name="invokeLucenePipelet">
| + | |
| − | <proc:pipelet class="org.eclipse.smila.lucene.pipelets.LuceneIndexPipelet" />
| + | |
| − | <proc:variables input="request" output="request" />
| + | |
| − | <proc:setAnnotations>
| + | |
| − | <rec:An n="org.eclipse.smila.lucene.LuceneIndexService">
| + | |
| − | <rec:V n="indexName">web_index</rec:V>
| + | |
| − | <rec:V n="executionMode">ADD</rec:V>
| + | |
| − | </rec:An>
| + | |
| − | </proc:setAnnotations>
| + | |
| − | </proc:invokePipelet>
| + | |
| − | </extensionActivity>
| + | |
| − |
| + | |
| − | </sequence>
| + | |
| − | </if>
| + | |
| − |
| + | |
| − | <reply name="end" partnerLink="Pipeline" portType="proc:ProcessorPortType"
| + | |
| − | operation="process" variable="request" />
| + | |
| − | <exit />
| + | |
| − | </sequence>
| + | |
| − | </process>
| + | |
| − | </source>
| + | |
| − | | + | |
| − | Note that we use "web_index" index name for the LuceneService in the code above:
| + | |
| − | <source lang="xml">
| + | |
| − | <rec:An n="org.eclipse.smila.lucene.LuceneIndexService">
| + | |
| − | <rec:V n="indexName">web_index</rec:V>
| + | |
| − | <rec:V n="executionMode">ADD</rec:V>
| + | |
| − | </rec:An>
| + | |
| − | </source>
| + | |
| − | | + | |
| − | We need to add our pipeline description to the <tt>deploy.xml</tt> file placed in the same directory. Add the following code to the end of <tt>deploy.xml</tt> before the closing <tt></deploy></tt> tag:
| + | |
| − | <source lang="xml">
| + | |
| − | <process name="proc:AddWebPipeline">
| + | |
| − | <in-memory>true</in-memory>
| + | |
| − | <provide partnerLink="Pipeline">
| + | |
| − | <service name="proc:AddWebPipeline" port="ProcessorPort" />
| + | |
| − | </provide>
| + | |
| − | </process>
| + | |
| − | </source>
| + | |
| − | | + | |
| − | Now we need to add our web_index to LuceneIndexService configuration.
| + | |
| − | | + | |
| − | === Configure LuceneIndexService ===
| + | |
| − | For more information about LuceneIndexService, please see [[SMILA/Documentation/LuceneIndexService|LuceneIndexService]]
| + | |
| − | | + | |
| − | Let's configure our web_index index structure and search template. Add the following code to the end of <tt>configuration/org.eclipse.smila.search.datadictionary/DataDictionary.xml</tt> file before the closing <tt></AnyFinderDataDictionary></tt> tag:
| + | |
| − | <source lang="xml">
| + | |
| − | <Index Name="web_index">
| + | |
| − | <Connection xmlns="http://www.anyfinder.de/DataDictionary/Connection" MaxConnections="5"/>
| + | |
| − | <IndexStructure xmlns="http://www.anyfinder.de/IndexStructure" Name="web_index">
| + | |
| − | <Analyzer ClassName="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
| + | |
| − | <IndexField FieldNo="8" IndexValue="true" Name="MimeType" StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="7" IndexValue="true" Name="Size" StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="6" IndexValue="true" Name="Extension" StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="5" IndexValue="true" Name="Title" StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="4" IndexValue="true" Name="Url" StoreText="true" Tokenize="false" Type="Text">
| + | |
| − | <Analyzer ClassName="org.apache.lucene.analysis.WhitespaceAnalyzer"/>
| + | |
| − | </IndexField>
| + | |
| − | <IndexField FieldNo="3" IndexValue="true" Name="LastModifiedDate" StoreText="true" Tokenize="false" Type="Text"/>
| + | |
| − | <IndexField FieldNo="2" IndexValue="true" Name="Path" StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="1" IndexValue="true" Name="Filename" StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | <IndexField FieldNo="0" IndexValue="true" Name="Content" StoreText="true" Tokenize="true" Type="Text"/>
| + | |
| − | </IndexStructure>
| + | |
| − | <Configuration xmlns="http://www.anyfinder.de/DataDictionary/Configuration" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
| + | |
| − | xsi:schemaLocation="http://www.anyfinder.de/DataDictionary/Configuration ../xml/DataDictionaryConfiguration.xsd">
| + | |
| − | <DefaultConfig>
| + | |
| − | <Field FieldNo="8">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="7">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="6">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="5">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="4">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="3">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="2">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="1">
| + | |
| − | <FieldConfig Constraint="optional" Weight="1" xsi:type="FTText">
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="OR" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | <Field FieldNo="0">
| + | |
| − | <FieldConfig Constraint="required" Weight="1" xsi:type="FTText">
| + | |
| − | <NodeTransformer xmlns="http://www.anyfinder.de/Search/ParameterObjects" Name="urn:ExtendedNodeTransformer">
| + | |
| − | <ParameterSet xmlns="http://www.brox.de/ParameterSet"/>
| + | |
| − | </NodeTransformer>
| + | |
| − | <Parameter xmlns="http://www.anyfinder.de/Search/TextField" Operator="AND" Tolerance="exact"/>
| + | |
| − | </FieldConfig>
| + | |
| − | </Field>
| + | |
| − | </DefaultConfig>
| + | |
| − | </Configuration>
| + | |
| − | </Index>
| + | |
| − | </source>
| + | |
| − | Now we need to add mapping of attribute and attachment names to Lucene "FieldNo" defined in <tt>DataDictionary.xml</tt>. Open <tt>configuration/org.eclipse.smila.lucene/Mappings.xml</tt> file and add the following code to the end of file before closing <tt></Mappings></tt> tag:
| + | |
| − | <source lang="xml">
| + | |
| − | <Mapping indexName="web_index">
| + | |
| − | <Attributes>
| + | |
| − | <Attribute name="Filename" fieldNo="1" />
| + | |
| − | <Attribute name="Path" fieldNo="2" />
| + | |
| − | <Attribute name="LastModifiedDate" fieldNo="3" />
| + | |
| − | <Attribute name="Url" fieldNo="4" />
| + | |
| − | <Attribute name="Title" fieldNo="5" />
| + | |
| − | <Attribute name="Extension" fieldNo="6" />
| + | |
| − | <Attribute name="Size" fieldNo="7" />
| + | |
| − | <Attribute name="MimeType" fieldNo="8" />
| + | |
| − | </Attributes>
| + | |
| − | <Attachments>
| + | |
| − | <Attachment name="Content" fieldNo="0" />
| + | |
| − | </Attachments>
| + | |
| − | </Mapping>
| + | |
| − | </source>
| + | |
| − | | + | |
| − | === Put it all together ===
| + | |
| − | Ok, now it seems that we have finally finished configuring SMILA for using separate workflows for file system and web crawling and index data from these crawlers into different indices.
| + | |
| − | Here is what we have done so far:
| + | |
| − | # We modified Listener rules in order to use different workflows for web and file system crawling.
| + | |
| − | # We created a new BPEL workflow for the Web Crawler.
| + | |
| − | # We added the <tt>web_index</tt> index to the Lucence configuration.
| + | |
| − | Now we can start SMILA again and observe what will happen when starting the Web Crawler.
| + | |
| − | | + | |
| − | {|width="100%" style="background-color:#d8e4f1; padding-left:30px;"
| + | |
| − | |
| + | |
| − | It's very important to shutdown SMILA engine and restart afterwards because modified configurations will load only on startup.
| + | |
| − | |}
| + | |
| − | | + | |
| − | Now we can also search on the web_index from browser:
| + | |
| − | | + | |
| − | [[Image:Web_index-search.png]]
| + | |
| − | | + | |
| − | == Configuration overview ==
| + | |
| − | | + | |
| − | SMILA configuration files are placed into <tt>configuration</tt> directory of the SMILA application.
| + | |
| − | Following figure shows configuration files relevant to this tutorial, regarding SMILA components and data lifecycle. SMILA components names are black-colored, directories containing configuration files and filenames are blue-colored.
| + | |
| − | | + | |
| − | [[Image:Smila-configuration-overview.jpg]]
| + | |
