Difference between revisions of "SMILA/Specifications/2 Nodes"

Revision as of 07:05, 17 November 2008

Use case

1. XmlStorage is not started
2. First node runs CrawlerController, Crawlers and DeltaIndexing
3. Second node runs Broker, Processing and Indexing

Configuration

Both nodes

Both nodes must use the same binary storage configuration to have access to records attachments. To configure binary storage, open file configuration/org.eclipse.smila.binarystorage/configuration/binarystorageconfig.properties , uncomment property path and set it to some existing path that will be accessible by both SMILA instances, for example path=e:/storage.

First node

Here is bundles configuration for starting SMILA on the first node, where crawling jobs will run. Please note that probably some bundles that are not really needed for crawling, currently must be started to satisfy dependencies of other bundles (this must be fixed of course)

osgi.bundles=org.eclipse.core.runtime@:start, \
org.eclipse.equinox.common@2:start, \
org.eclipse.update.configurator@3:start, \
org.eclipse.equinox.util@2:start, \
org.eclipse.equinox.ds@3:start, \
#com.sleepycat.dbxml@4:start, \
org.eclipse.smila.utils@3:start, \
org.eclipse.smila.management@3:start, \
org.eclipse.smila.connectivity.deltaindexing@4:start, \
org.eclipse.smila.management.jmx@4:start, \
org.eclipse.smila.management.crawlercontroller@4:start, \
#org.eclipse.smila.connectivity.queue.broker.main@4:start, \
org.eclipse.smila.xmlstorage@4:start, \
#org.eclipse.smila.xmlstorage.bdb@4:start, \
org.eclipse.smila.processing@4:start, \
org.eclipse.smila.processing.bpel@4:start, \
#org.eclipse.smila.processing.pipelets.mimetype@4:start, \
#org.eclipse.smila.lucene@4:start, \
org.eclipse.smila.connectivity.framework@4:start, \
org.eclipse.smila.connectivity.impl@4:start, \
org.eclipse.smila.connectivity.framework.crawler.filesystem@4:start, \
org.eclipse.smila.connectivity.framework.crawler.web@4:start, \
org.eclipse.smila.connectivity.queue.worker@4:start, \
org.eclipse.smila.binarystorage@4:start, \
org.eclipse.smila.blackboard@4:start, \
#org.eclipse.smila.tomcat@4:start

First node doesn't need Listener, so Listener rules must be commented into ListenerConfig.xml

Filters are not needed for the Router, so comment or remove Fetch element.

<Rule Name="Default ADD Rule">
  <Condition>Operation='ADD'</Condition>
  <Task BlackboardSync="true"> 
    <!-- Fetch Filter="no-filter"/ -->
    <Send BrokerId="broker1" Queue="SMILA.connectivity">
    </Send>
  </Task>
</Rule>
 
 
<Rule Name="Default DELETE Rule">
  <Condition>Operation='DELETE'</Condition>
  <Task BlackboardSync="true">
  <!-- Fetch Filter="no-filter"/ -->
  <Send BrokerId="broker1" Queue="SMILA.connectivity">
  </Send>
  </Task>
</Rule>

Broker conection must be configured properly in the ConnectionConfig.xml in case when Broker runs on remote computer:

<ConnectionConfig Id="broker1">
  <URL>tcp://localhost:61616</URL>
  <User>any</User>
  <Password>any</Password>
  <ConnectionFactory>org.apache.activemq.ActiveMQConnectionFactory</ConnectionFactory>
</ConnectionConfig>

Separate nodes will provide separate management functionality, so JMX port must be different on both nodes. It can be changed in the SMILA.ini:

-Dcom.sun.management.jmxremote.port=10001

Second node

Here is the second node bundles configuration:

osgi.bundles=org.eclipse.core.runtime@:start, \
org.eclipse.equinox.common@2:start, \
org.eclipse.update.configurator@3:start, \
org.eclipse.equinox.util@2:start, \
org.eclipse.equinox.ds@3:start, \
#com.sleepycat.dbxml@4:start, \
org.eclipse.smila.utils@3:start, \
org.eclipse.smila.management@3:start, \
#org.eclipse.smila.connectivity.deltaindexing@4:start, \
org.eclipse.smila.management.jmx@4:start, \
org.eclipse.smila.management.crawlercontroller@4:start, \
org.eclipse.smila.connectivity.queue.broker.main@4:start, \
org.eclipse.smila.xmlstorage@4:start, \
#org.eclipse.smila.xmlstorage.bdb@4:start, \
org.eclipse.smila.processing@4:start, \
org.eclipse.smila.processing.bpel@4:start, \
org.eclipse.smila.processing.pipelets.mimetype@4:start, \
org.eclipse.smila.lucene@4:start, \
org.eclipse.smila.connectivity.framework@4:start, \
org.eclipse.smila.connectivity.impl@4:start, \
#org.eclipse.smila.connectivity.framework.crawler.filesystem@4:start, \
#org.eclipse.smila.connectivity.framework.crawler.web@4:start, \
org.eclipse.smila.connectivity.queue.worker@4:start, \
org.eclipse.smila.binarystorage@4:start, \
org.eclipse.smila.blackboard@4:start, \
org.eclipse.smila.tomcat@4:start

It must use JMX port that differs from the first node, and must NOT use filters in Listener configuration too.