Jump to: navigation, search

Difference between revisions of "SMILA/FAQ"

(I get classloading errors in invocations of my own Pipelet/Processing Service when running SMILA outside the IDE. In the IDE it works)
m (I implemented/deployed an OSGi Service but it seems that it isn't activated)
(34 intermediate revisions by 8 users not shown)
Line 2: Line 2:
  
 
'''General Hint:''' When you have problems during a SMILA launch / run, please have a look at the '''SMILA log''' file first. (<tt><SMILA>/SMILA.log</tt>)
 
'''General Hint:''' When you have problems during a SMILA launch / run, please have a look at the '''SMILA log''' file first. (<tt><SMILA>/SMILA.log</tt>)
 +
 +
== General ==
 +
 +
=== Where do I find the log file? ===
 +
 +
The SMILA log file's name is <tt>SMILA.log</tt> and is to be found at the applications working folder:
 +
 +
If you extracted SMILA from a zip, you can find the log file in the folder, where you extracted the SMILA.
 +
<pre>
 +
/<SMILA>
 +
  /about_files
 +
  /configuration
 +
  /features
 +
  /jmxclient
 +
  /plugins
 +
  /workspace
 +
  .eclipseproduct
 +
  ...
 +
  SMILA
 +
  SMILA.ini
 +
  -> SMILA.log <-
 +
</pre>
 +
 +
If you started SMILA from within the Eclipse IDE using the launcher, you can find the log file at the project <tt>SMILA.application</tt> in your workspace.
 +
 +
=== How can I see that SMILA started correctly? ===
 +
 +
1. Open your browser at <code>http://localhost:8080/smila</code>, you should see a system state overview and links to different APIs.
 +
 +
2. You should see no stacktraces in the log ;) and it should end with an entry like the following if SMILA has just started:
 +
 +
<pre>
 +
... INFO  ... internal.HttpServiceImpl    - HTTP server started successfully on port 8080.
 +
</pre>
  
 
== Building SMILA ==
 
== Building SMILA ==
Line 17: Line 51:
 
# Save and build again.
 
# Save and build again.
  
 +
 +
=== Build fails with 'java.net.MalformedURLException: no protocol: ${eclipseBaseURL}' message ===
 +
 +
The complete error looks like this:
 +
 +
<source lang="text">
 +
...
 +
build.core:
 +
    [java] Buildfile: /home/scum36/Eclipse/helios-sr2-build/plugins/org.eclipse.pde.build_3.6.2.R36x_20110203/scripts/productBuild/productBuild.xml
 +
    [java]
 +
    [java] main:
 +
    [java]
 +
    [java] preBuild:
 +
    [java]
 +
    [java] preSetup:
 +
    [java]
 +
    [java] checkLocalMaps:
 +
    [java]
 +
    [java] getMapFiles:
 +
    [java]      [cvs] Caught exception: Cannot run program "cvs" (in directory "/home/scum36/workspace/SMILA/smila.build/maps"): java.io.IOException: error=2, No such file or directory
 +
    [java]
 +
    [java] postSetup:
 +
    [java]
 +
    [java] checkLocalBase:
 +
    [java]
 +
    [java] getBaseComponents:
 +
    [java]
 +
    [java] BUILD FAILED
 +
    [java] /home/scum36/Eclipse/helios-sr2-build/plugins/org.eclipse.pde.build_3.6.2.R36x_20110203/scripts/productBuild/productBuild.xml:33: The following error occurred while executing this line:
 +
    [java] /home/scum36/Eclipse/helios-sr2-build/plugins/org.eclipse.pde.build_3.6.2.R36x_20110203/scripts/build.xml:53: The following error occurred while executing this line:
 +
    [java] /home/scum36/workspace/SMILA/SMILA.builder/customTargets.xml:80: The following error occurred while executing this line:
 +
    [java] /home/scum36/workspace/SMILA/SMILA.builder/customTargets.xml:25: java.net.MalformedURLException: no protocol: ${eclipseBaseURL}
 +
    [java]
 +
    [java] Total time: 0 seconds
 +
    [java] An error has occurred. See the log file
 +
    [java] /home/scum36/workspace/.metadata/.log.
 +
</source>
 +
 +
 +
Probably you did forget to copy <code>build.properties.template</code> to <code>build.properties</code> in <code>SMILA.builder</code> and adapt it. See
 +
[http://wiki.eclipse.org/SMILA/Development_Guidelines/Howto_build_a_SMILA-Distribution#Setting_the_Target_Build_Platform How to build SMILA].
  
 
== Launching SMILA ==
 
== Launching SMILA ==
Line 46: Line 121:
 
|}
 
|}
  
For the complete documentation on eclipse runtime options please see: http://help.eclipse.org/ganymede/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/misc/runtime-options.html
+
For the complete documentation on eclipse runtime options please see: http://help.eclipse.org/indigo/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/misc/runtime-options.html
  
 
=== Bundles ===
 
=== Bundles ===
  
==== Launching crawler bundles - How to solve: "Could not find crawler id" error message? ====
+
==== new bundle was not started  ====
  
While launching SMILA I receive the following error message:
+
After launching SMILA my new bundle doesn't seem to be started.
  
<code>Could not find crawler id</code>
+
If you started '''<tt>SMILA.launch</tt>''' in eclipse to launch SMILA: The launcher didn't start your new bundle. Try this:
  
If you started '''<tt>SMILA.launch</tt>''' to launch SMILA: The launcher didn't start your new crawler bundle. Try this:
+
# Add your bundle by selecting "Run Configurations" in eclipse and choose your SMILA profile.
 
+
# Add your bundle by selecting "Open Run dialog" in eclipse and choose your SMILA profile.
+
 
# Select your bundle in the list and set the checkmark.
 
# Select your bundle in the list and set the checkmark.
 
# Set the start level to "4" and the autostart to "true".
 
# Set the start level to "4" and the autostart to "true".
Line 65: Line 138:
  
 
<ol>
 
<ol>
<li>Open the file <tt>config.ini</tt> and add your bundle as shown below:<br>
+
<li>Open the file <tt>configuration/config.ini</tt> and add your bundle as shown below:<br>
 
<code>org.eclipse.smila.connectivity.framework.crawler.owncrawler@5:start, \</code></li>
 
<code>org.eclipse.smila.connectivity.framework.crawler.owncrawler@5:start, \</code></li>
 
<li>Open the <tt>build.properties</tt> file of your bundle and include the folders <tt>schemas/</tt>, <tt>OSGI-INF/</tt>, and the file <tt>plugin.xml</tt>.</li>
 
<li>Open the <tt>build.properties</tt> file of your bundle and include the folders <tt>schemas/</tt>, <tt>OSGI-INF/</tt>, and the file <tt>plugin.xml</tt>.</li>
 
</ol>
 
</ol>
 
==== I changed the implementation of a bundle, deployed it and restarted SMILA, but SMILA still seems to use the old bundle ====
 
 
Close SMILA, delete the following directories in your configuration folder and restart SMILA again:
 
* <tt>org.eclipse.core.runtime/</tt>
 
* <tt>org.eclipse.equinox.app/</tt>
 
* <tt>org.eclipse.osgi/</tt>
 
* <tt>org.eclipse.update/</tt>
 
  
 
==== I changed bundle settings in my <tt>config.ini</tt>, but after SMILA restart nothing changed ====
 
==== I changed bundle settings in my <tt>config.ini</tt>, but after SMILA restart nothing changed ====
  
Check your <tt>config.ini</tt> for unusual whitespaces (e.g. a tab) between the (edited) bundle entries - and remove them.  
+
Check your <tt>config.ini</tt> for unusual whitespaces (e.g. a tab) between the (edited) bundle entries or after the backslashes on line ends - and remove them.  
  
 
If that doesn't help, see question (resp. answer) before.
 
If that doesn't help, see question (resp. answer) before.
  
  
=== JConsole ===
+
== Configuring/Running SMILA ==
  
==== Why is the ''SMILA'' package not in the JConsole tree? ====
+
=== Basics ===
  
I've started <tt>SMILA.exe</tt> but the SMILA package isn't in the tree of JConsole.
+
==== How do I change the HTTP port of SMILA? ====
  
To solve this try the following:
+
The HTTP port is configured in fie <tt>configuration/org.eclipse.smila.clusterconfig.simple/clusterconfig.json</tt>. Change it there and restart SMILA.
  
# Create a new connection.
+
See [[SMILA/Documentation/JettyHttpServer#Setting_the_HTTP_port]] for details.
# Change your connection by setting the port "9004" on the ''Remote'' tab.
+
# Click the ''Connect'' button, switch to the ''MBeans'' tab, and check the tree again.
+
 
+
==== Why is the ''SMILA/CrawlerController'' MBean not in the JConsole tree? ====
+
 
+
Check if all needed bundles are active. Open the equinox console and type:
+
* <tt>ss crawler</tt>
+
* <tt>ss deltaindexing.impl</tt>
+
 
+
If one of these is not active:
+
* check the <tt>configuration/config.ini</tt>
+
* check the log file for errors
+
 
+
==== Cannot connect with JConsole to a remote machine running Ubuntu 8.04 or newer ====
+
 
+
For some reason the JConsole cannot make a '''''remote''''' connection to JVM running on Ubuntu 8.04 or newer installations. (We did some tests with SuSE Linux and had no problems.)
+
 
+
This can be easily circumvented by modifying <tt>/etc/hosts</tt> file.
+
Simply replace <tt>127.0.1.1</tt> with the real IP address of your Ubuntu machine and you're ready to go.
+
 
+
For example, if you have a line in /etc/hosts that looks like this:
+
 
+
<tt>127.0.1.1 jupiter</tt>        (Where "jupiter" it the actual name of the Ubuntu machine in this example. You will almost certainly have some other name here ;-)
+
 
+
then replace it with:
+
 
+
<tt>192.168.220.101 jupiter</tt>  (Where 192.168.220.101 is the actual IP address of the Ubuntu machine in this example.)
+
 
+
== Configuring/Running SMILA ==
+
  
 
=== Crawler ===
 
=== Crawler ===
  
==== I tried to crawl/index a data source, JConsole says "Crawl ... sucessfully started" but nothing seems to happen ====
+
==== I started a crawl job, the job was started successfully, but nothing seems to happen ====
  
Check if your queue is receiving records:
+
Check the SMILA log file for errors.
# Open the JConsole's MBeans tab and go to: <code>org.apache.activemq/localhost/Queue/<queue>/Attributes</code>
+
# if <code>EnqueueCount</code> is increased your queue is receiving records, so it's not a crawler issue.
+
  
If your queue didn't receive any records from the crawler:
+
Check the state of the crawl job in [http://localhost:8080/smila/jobmanager/jobs jobmanager state API]
# Did you crawl the data source before? If so, try a <code>clearAll</code> on the DeltaIndexing-MBean in your JConsole
+
* if the state in your crawl job is "RUNNING", check the mode it has been started
# Check your crawler configuration file (see <tt>configuration/org.eclipse.smila.connectivity.framework/</tt>)
+
** if it says {{code|"mode": "STANDARD"}}, stop the job run ({{code|POST http://localhost:8080/smila/jobmanager/jobs/<job name>/<job run id>/finish}}) and start it again with mode "runOnce", e.g. {{code|POST http://localhost:8080/smila/jobmanager/jobs/<job name> {"mode": "runOnce"}}}
# Check the queue routing configuration: <tt>org.eclipse.smila.connectivity.queue.worker.jms/QueueWorkerRouterConfig.xml</tt>
+
* if the state is FINISHING
# Check the queue connection: <tt>org.eclipse.smila.connectivity.queue.worker.jms/QueueWorkerConnectionConfig.xml</tt>
+
** check the job into which the crawl job pushes its records for processed records or errors.
# If queue is running in a separate SMILA instance (e.g. on another PC), make sure that "Queue-SMILA" is started before "Crawler-SMILA"
+
* if everything up to now is OK
 +
** just wait some time, the autocommit handler maybe hasn't yet committed the data to the Solr index
 +
** check {{code|autoCommit}} settings in the <tt>solrconfig.xml</tt> file of your Solr core.
  
 
=== Pipeline ===  
 
=== Pipeline ===  
Line 147: Line 183:
 
See entry: <code>pipeline.timeout</code>
 
See entry: <code>pipeline.timeout</code>
  
 +
==== My xpath expression doesnt work ====
 +
# Check that the expression is correct with a XPath tool
 +
# ensure that the attributes are not filtered out, see [[SMILA/Documentation/HowTo/How_to_filter_and_access_record_data_in_BPEL]]
  
=== Lucene Indexing / Search ===
+
'''Helpful log points'''
 +
{{CTable}}
 +
| Area || id || level || Comment
 +
|-
 +
| BPEL || org.eclipse.smila.processing.bpel.PipeletManager  || debug || Log BPEL request XML before and after pipelet invocation. Note: this contains the XML for the records but only their filtered version.
 +
|-
 +
| BPEL || org.eclipse.smila.processing.bpel.PipeletManager  || trace|| Log full XML of records after pipelet invocation. Note: this is the real thing unfiltered.
 +
|}
  
==== How can I browse/search an existing Lucene index without SMILA (for debug purposes)? ====
+
== Implementing Pipelets / OSGi Services / Bundles ==
  
Try [http://www.getopt.org/luke/ LUKE]
+
=== Pipelets ===
  
==== Why are attributes in my Lucene index missing / permuted? ====
+
==== I want to use the <code>ConfigUtils</code> class in my Pipelet to read the configuration, where do I have to put my configuration files? ====
  
Open your configuration files:
+
Configuration files are searched for in the following order:
* <tt>configuration/org.eclipse.smila.lucene/Mappings.xml</tt>
+
# <tt><SMILA>/configuration/<bundle-name>/<config-file></tt>
* <tt>configuration/org.eclipse.smila.search.datadictionary/DataDictionary.xml</tt>
+
# <tt><config-file></tt> in the root path of the bundle jar-file
  
Check:
+
See [[SMILA/Project_Concepts/Simple_configuration_handler|Configuration Handler]] for more information.
* if all index fields are specified in these two files
+
* if the field numbers are compatible to each other
+
  
Still the same problem?
+
==== I get classloading errors in invocations of my own Pipelet when running SMILA outside the IDE. In the IDE it works ====
* Close SMILA, remove file (if exists): <tt>workspace/org.eclipse.smila.search.datadictionary/DataDictionary.xml</tt> and restart SMILA
+
  
==== SMILA doesn't return search results, although I see appropriate entries for (some fields of) my query in the Lucene index ====
+
The error could look like this:
  
Open the file: <tt>configuration/org.eclipse.smila.search.datadictionary/DataDictionary.xml</tt>
+
<code>
* Check the <code>Constraint</code> entries for all fields: If a <code>Constraint</code> is set <code>required</code> and this field is set in your query, the query value ''must'' match the indexed value! (regardless of other fields)
+
2010-11-19 11:28:36,101 ERROR [ODEServerImpl-1
 +
]  vpu.JacobVPU                                  - Method "run" in class "org.apache.ode.bpel.rtrep.v2.EXTENSIONACTIVITY" threw an unexpected exception.
 +
java.lang.LinkageError: loader constraint violation: loader (instance of org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) previously initiated loading for a different type with name "org/w3c/dom/Document"
 +
</code>
  
Still problems? Try removing the workspace version of that file if it exists:
+
We are not completely sure, why this happens, but a solution is to set this system property in the SMILA.ini file:
* Close SMILA, remove <tt>workspace/org.eclipse.smila.search.datadictionary/DataDictionary.xml</tt>, and restart SMILA.
+
  
==== Test search application (http://localhost:8080/SMILA/search) returns <tt>HTTP Status 500</tt> with the stack trace====
+
<code>-Dosgi.java.profile.bootdelegation=override</code>
  
This error usually happens when no index has been build but the user tries to do the search (directly after installing SMILA) by calling the test search application. Prior to 0.5 M3 user gets in this case a long stack trace with the real cause of the problem at the end of it: <tt>org.eclipse.smila.search.index.IndexException: index does not exist in data dictionary [test_index]</tt>. This means that the test index named "test_index" has not been created yet. This can easily be done by starting a crawler or an agent. In 0.5 M3 we improved the error handling in search servlet so that only error messages are displayed and highlighted. (Stack trace can still be seen by simply clicking on "Details...".)
+
Thamks to Bogdan Sacaleanu for the solution. See this [http://smila.markmail.org/thread/sj4vhcikq2wndtdp thread in the smila-dev mailing list] for additional details.
  
== Implementing Pipelets / Processing Services / Bundles ==
 
  
=== Configuration ===
+
=== OSGI-Services ===
  
==== I want to use the <code>ConfigUtils</code> class in my Processing Service to read the configuration, where do I have to put my configuration files? ====
+
==== I implemented/deployed an OSGi Service but it seems that it isn't activated  ====
  
Configuration files are searched for in the following order:
+
Check your bundle, it should contain a file like that:
# <tt><SMILA>/configuration/<bundle-name>/<config-file></tt>
+
# <tt><config-file></tt> in the root path of the bundle jar-file
+
  
 +
*<tt>OSGI-INF/&lt;myService&gt;.xml</tt>
  
=== Deploy / Launch ===
+
In this file your service has to be referenced. If you have copied the file from some other service, be sure to change the component name in the root element to something unique, because DS does not start multiple services with the same component name.
  
==== I implemented/deployed a Processing Service in a new bundle but SMILA log says that it couldn't be found ====
+
*<tt>&lt;component name="&lt;myService&gt;" immediate="true"&gt;</tt>
  
Check your new bundle, it should contain a file like that:
+
Also the file has to be referenced in the META-INF / MANIFEST.MF file of your bundle as a service component:  
  
* <tt>OSGI-INF/<myService>.xml</tt>
+
*<tt>Service-Component: OSGI-INF/&lt;myService&gt;.xml</tt>
  
In this file your new processing service has to be referenced. Also the file has to be referenced from the MANIFEST.MF file of your bundle as a service component:
+
On the "Build" page of the manifest editor, you must add the OSGI-INF directory to the binary build.  
  
<tt>Service-Component: OSGI-INF/<myService>.xml</tt>
+
And finally, your bundle has to be started at SMILA launch, by adding it to the <tt>configuration/config.ini</tt>:<br> <tt>... </tt><br> <tt>&lt;bundle-id&gt;@4:start, \</tt><br> <tt>... </tt>  
  
On the "Build" page of the manifest editor, you must add the OSGI-INF directory to the binary build. And finally, your bundle has to be started at SMILA launch, e.g. by adding it to the config.ini.
+
If you are using '''<tt>SMILA.launch</tt>''' to launch SMILA in eclipse IDE, you have to open the run/debug configuration of SMILA, check the new bundle and set Auto-Start to "true".  
  
 +
If you checked all the things above and it still doesn't work:
  
==== I get classloading errors in invocations of my own Pipelet/Processing Service when running SMILA outside the IDE. In the IDE it works ====
+
*Implement an <tt>activate()</tt> method in your service - if not already there, and check if it's called at startup.
 +
*If your service depends on other services, check if those are activated
 +
*If your service uses other services, check the naming of the set/unset methods in the Java code vs. those specified in the <tt>&lt;myService&gt;.xml</tt>
 +
*If your services uses 3rd-party-jars, make sure that they are specified in the <tt>Bundle-ClassPath</tt> section of your MANIFEST.MF (Do not reference the lib folder here, instead reference all jars, e.g. <tt>Bundle-ClassPath: ., &lt;lib1.jar&gt;, &lt;lib2.jar&gt;</tt>). Make also sure, that they are added to the binary build ("Build" page of the manifest editor).
 +
*If your service has super-classes you may need to include <tt>Import-Package:</tt> declarations of the super-classes in your service implementation class even if there are no compile errors.
 +
*Use the OSGi console, e.g. via: <tt>telnet &lt;host&gt; 9005</tt>
 +
**<tt>ss &lt;bundle-ID&gt;</tt> - check if your bundle is in the list and ACTIVE (the bundle-ID is the "Bundle-SymbolicName" from the MANIFEST.MF)
 +
***if it isn't in the list, the bundle is not correctly deployed
 +
***if it isn't ACTIVE but only RESOLVED it's not started (-&gt; see hints above)
 +
**<tt>bundle &lt;id&gt;</tt> - check if your service is listed here in the "Registered Services" section (the "id" can be taken from the "ss" output)
 +
***if it isn't there, your service is not correctly deployed (-&gt; check all points above)
  
The error could look like this:
+
=== General ===
  
<code>
+
==== I get classloading errors when I try to access an external Web Service using JAX-WS ====
2010-11-19 11:28:36,101 ERROR [ODEServerImpl-1
+
]  vpu.JacobVPU                                  - Method "run" in class "org.apache.ode.bpel.rtrep.v2.EXTENSIONACTIVITY" threw an unexpected exception.
+
java.lang.LinkageError: loader constraint violation: loader (instance of org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) previously initiated loading for a different type with name "org/w3c/dom/Document"
+
</code>
+
  
We are not completely sure, why this happens, but a solution is to set this system property in the SMILA.ini file:
+
Class loading problems often occur when using third party libraries that use the "thread context classloader" in OSGi, and standard implementations of Java specifications by Sun/Oracle (or other non-OSGi-aware parties...) do this very often.  
  
<code>-Dosgi.java.profile.bootdelegation=override</code>
+
The only solution we currently know of is to wrap the critical section (in this case it's the construction of the webservice client class) in a piece of code like this:
  
Thamks to Bogdan Sacaleanu for the solution. See this [http://smila.markmail.org/thread/sj4vhcikq2wndtdp thread in the smila-dev mailing list] for additional details.
+
<source lang="java">
 +
ClassLoader tccl = Thread.currentThread().getContextClassLoader();
 +
try {
 +
    Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
 +
    /*
 +
    * initialize the client
 +
    */
 +
} finally {
 +
    Thread.currentThread().setContextClassLoader(tccl);
 +
}
 +
</source>
 +
 
 +
See also [http://www.eclipse.org/forums/index.php/t/266362/ this forum thread].
 +
 
 +
 
 +
==== More classloader errors ====
 +
 
 +
If you get an error like this:
 +
 
 +
<source lang="text">
 +
Caused by: java.lang.LinkageError: loader constraint violation: when resolving field "DATETIME" the class loader (instance of
 +
org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) of the referring class, javax/xml/datatype/DatatypeConstants, and the class loader (instance of <bootloader>) for the field's resolved type, javax/xml/namespace/QName, have different Class objects for that type
 +
</source>
  
 +
the reason is most likely that you have classes in the bundle classpath of your bundle that contains classes which are also part of the JDK runtime library (java.*, javax.*). Remove these classes from the bundle and it should work.
  
 +
See also [http://www.eclipse.org/forums/index.php/t/266362/ this forum thread].
  
 
[[Category:SMILA]]
 
[[Category:SMILA]]

Revision as of 04:35, 11 June 2013

This pages contains the frequently asked questions of the SMILA project.

General Hint: When you have problems during a SMILA launch / run, please have a look at the SMILA log file first. (<SMILA>/SMILA.log)

General

Where do I find the log file?

The SMILA log file's name is SMILA.log and is to be found at the applications working folder:

If you extracted SMILA from a zip, you can find the log file in the folder, where you extracted the SMILA.

/<SMILA>
  /about_files
  /configuration
  /features
  /jmxclient
  /plugins
  /workspace
  .eclipseproduct
  ...
  SMILA
  SMILA.ini
  -> SMILA.log <-

If you started SMILA from within the Eclipse IDE using the launcher, you can find the log file at the project SMILA.application in your workspace.

How can I see that SMILA started correctly?

1. Open your browser at http://localhost:8080/smila, you should see a system state overview and links to different APIs.

2. You should see no stacktraces in the log ;) and it should end with an entry like the following if SMILA has just started:

... INFO  ... internal.HttpServiceImpl    - HTTP server started successfully on port 8080.

Building SMILA

I receive an Out of Memory error? What can I do?

While building with SMILA.builder I receive the following errror message:

Build Failed - Out of Memory - Java heap space

The reason for this is that Ant hasn´t enough heap space to build the project. You will have to expand the heap space by setting the VM arguments accordingly. In eclipse try the following:

  1. Click Open external tools dialog and select your Ant build profile.
  2. Switch to the JRE tab and add the following VM arguments: -Xms40m -Xmx512m.
  3. Save and build again.


Build fails with 'java.net.MalformedURLException: no protocol: ${eclipseBaseURL}' message

The complete error looks like this:

...
build.core:
     [java] Buildfile: /home/scum36/Eclipse/helios-sr2-build/plugins/org.eclipse.pde.build_3.6.2.R36x_20110203/scripts/productBuild/productBuild.xml
     [java] 
     [java] main:
     [java] 
     [java] preBuild:
     [java] 
     [java] preSetup:
     [java] 
     [java] checkLocalMaps:
     [java] 
     [java] getMapFiles:
     [java]       [cvs] Caught exception: Cannot run program "cvs" (in directory "/home/scum36/workspace/SMILA/smila.build/maps"): java.io.IOException: error=2, No such file or directory
     [java] 
     [java] postSetup:
     [java] 
     [java] checkLocalBase:
     [java] 
     [java] getBaseComponents:
     [java] 
     [java] BUILD FAILED
     [java] /home/scum36/Eclipse/helios-sr2-build/plugins/org.eclipse.pde.build_3.6.2.R36x_20110203/scripts/productBuild/productBuild.xml:33: The following error occurred while executing this line:
     [java] /home/scum36/Eclipse/helios-sr2-build/plugins/org.eclipse.pde.build_3.6.2.R36x_20110203/scripts/build.xml:53: The following error occurred while executing this line:
     [java] /home/scum36/workspace/SMILA/SMILA.builder/customTargets.xml:80: The following error occurred while executing this line:
     [java] /home/scum36/workspace/SMILA/SMILA.builder/customTargets.xml:25: java.net.MalformedURLException: no protocol: ${eclipseBaseURL}
     [java] 
     [java] Total time: 0 seconds
     [java] An error has occurred. See the log file
     [java] /home/scum36/workspace/.metadata/.log.


Probably you did forget to copy build.properties.template to build.properties in SMILA.builder and adapt it. See How to build SMILA.

Launching SMILA

Linux

How to start/stop and manage SMILA as a background process on a Linux machine?

Since the default configuration (stored in SMILA.ini) of the OSGi runtime (in our case Equinox) launcher expects that you execute it in foreground and therefore have an OSGi console running in your shell and listening to the standard input, the first thing we have to do is to advise the launcher (and thereby Equinox) to listen on some TCP port instead. This is done by adding a new line with the port number just after the "-console" line.

For example, to set console to listen at TCP port 9999, SMILA.ini would look like this:

-console
9999
...

Now, after SMILA has been started with “$ nohup ./SMILA &”, the console can be accessed from any computer simply by opening a telnet session:

$ telnet <smila_host_name> <console_port>


Symbol comment vote.png If you are logged in via telnet and just want to close this connection and not to stop Equinox running SMILA, than just type “disconnect”. Otherwise, if you want to stop SMILA and close the connection, use “close” - as you normally would do on a console running in your shell.

For the complete documentation on eclipse runtime options please see: http://help.eclipse.org/indigo/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/misc/runtime-options.html

Bundles

new bundle was not started

After launching SMILA my new bundle doesn't seem to be started.

If you started SMILA.launch in eclipse to launch SMILA: The launcher didn't start your new bundle. Try this:

  1. Add your bundle by selecting "Run Configurations" in eclipse and choose your SMILA profile.
  2. Select your bundle in the list and set the checkmark.
  3. Set the start level to "4" and the autostart to "true".

If you started SMILA.EXE to launch SMILA: Your bundle isn`t defined in config.ini or the start level isn´t correct. Try this:

  1. Open the file configuration/config.ini and add your bundle as shown below:
    org.eclipse.smila.connectivity.framework.crawler.owncrawler@5:start, \
  2. Open the build.properties file of your bundle and include the folders schemas/, OSGI-INF/, and the file plugin.xml.

I changed bundle settings in my config.ini, but after SMILA restart nothing changed

Check your config.ini for unusual whitespaces (e.g. a tab) between the (edited) bundle entries or after the backslashes on line ends - and remove them.

If that doesn't help, see question (resp. answer) before.


Configuring/Running SMILA

Basics

How do I change the HTTP port of SMILA?

The HTTP port is configured in fie configuration/org.eclipse.smila.clusterconfig.simple/clusterconfig.json. Change it there and restart SMILA.

See SMILA/Documentation/JettyHttpServer#Setting_the_HTTP_port for details.

Crawler

I started a crawl job, the job was started successfully, but nothing seems to happen

Check the SMILA log file for errors.

Check the state of the crawl job in jobmanager state API

  • if the state in your crawl job is "RUNNING", check the mode it has been started
  • if the state is FINISHING
    • check the job into which the crawl job pushes its records for processed records or errors.
  • if everything up to now is OK
    • just wait some time, the autocommit handler maybe hasn't yet committed the data to the Solr index
    • check autoCommit settings in the solrconfig.xml file of your Solr core.

Pipeline

Why do I get a timeout exception during a (long running) pipeline execution?

In SMILA there's a timeout configured for a pipeline execution:

configuration/org.eclipse.smila.processing.bpel/processor.properties

See entry: pipeline.timeout

My xpath expression doesnt work

  1. Check that the expression is correct with a XPath tool
  2. ensure that the attributes are not filtered out, see SMILA/Documentation/HowTo/How_to_filter_and_access_record_data_in_BPEL

Helpful log points

Area id level Comment
BPEL org.eclipse.smila.processing.bpel.PipeletManager debug Log BPEL request XML before and after pipelet invocation. Note: this contains the XML for the records but only their filtered version.
BPEL org.eclipse.smila.processing.bpel.PipeletManager trace Log full XML of records after pipelet invocation. Note: this is the real thing unfiltered.

Implementing Pipelets / OSGi Services / Bundles

Pipelets

I want to use the ConfigUtils class in my Pipelet to read the configuration, where do I have to put my configuration files?

Configuration files are searched for in the following order:

  1. <SMILA>/configuration/<bundle-name>/<config-file>
  2. <config-file> in the root path of the bundle jar-file

See Configuration Handler for more information.

I get classloading errors in invocations of my own Pipelet when running SMILA outside the IDE. In the IDE it works

The error could look like this:

2010-11-19 11:28:36,101 ERROR [ODEServerImpl-1 ] vpu.JacobVPU - Method "run" in class "org.apache.ode.bpel.rtrep.v2.EXTENSIONACTIVITY" threw an unexpected exception. java.lang.LinkageError: loader constraint violation: loader (instance of org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) previously initiated loading for a different type with name "org/w3c/dom/Document"

We are not completely sure, why this happens, but a solution is to set this system property in the SMILA.ini file:

-Dosgi.java.profile.bootdelegation=override

Thamks to Bogdan Sacaleanu for the solution. See this thread in the smila-dev mailing list for additional details.


OSGI-Services

I implemented/deployed an OSGi Service but it seems that it isn't activated

Check your bundle, it should contain a file like that:

  • OSGI-INF/<myService>.xml

In this file your service has to be referenced. If you have copied the file from some other service, be sure to change the component name in the root element to something unique, because DS does not start multiple services with the same component name.

  • <component name="<myService>" immediate="true">

Also the file has to be referenced in the META-INF / MANIFEST.MF file of your bundle as a service component:

  • Service-Component: OSGI-INF/<myService>.xml

On the "Build" page of the manifest editor, you must add the OSGI-INF directory to the binary build.

And finally, your bundle has to be started at SMILA launch, by adding it to the configuration/config.ini:
...
<bundle-id>@4:start, \
...

If you are using SMILA.launch to launch SMILA in eclipse IDE, you have to open the run/debug configuration of SMILA, check the new bundle and set Auto-Start to "true".

If you checked all the things above and it still doesn't work:

  • Implement an activate() method in your service - if not already there, and check if it's called at startup.
  • If your service depends on other services, check if those are activated
  • If your service uses other services, check the naming of the set/unset methods in the Java code vs. those specified in the <myService>.xml
  • If your services uses 3rd-party-jars, make sure that they are specified in the Bundle-ClassPath section of your MANIFEST.MF (Do not reference the lib folder here, instead reference all jars, e.g. Bundle-ClassPath: ., <lib1.jar>, <lib2.jar>). Make also sure, that they are added to the binary build ("Build" page of the manifest editor).
  • If your service has super-classes you may need to include Import-Package: declarations of the super-classes in your service implementation class even if there are no compile errors.
  • Use the OSGi console, e.g. via: telnet <host> 9005
    • ss <bundle-ID> - check if your bundle is in the list and ACTIVE (the bundle-ID is the "Bundle-SymbolicName" from the MANIFEST.MF)
      • if it isn't in the list, the bundle is not correctly deployed
      • if it isn't ACTIVE but only RESOLVED it's not started (-> see hints above)
    • bundle <id> - check if your service is listed here in the "Registered Services" section (the "id" can be taken from the "ss" output)
      • if it isn't there, your service is not correctly deployed (-> check all points above)

General

I get classloading errors when I try to access an external Web Service using JAX-WS

Class loading problems often occur when using third party libraries that use the "thread context classloader" in OSGi, and standard implementations of Java specifications by Sun/Oracle (or other non-OSGi-aware parties...) do this very often.

The only solution we currently know of is to wrap the critical section (in this case it's the construction of the webservice client class) in a piece of code like this:

ClassLoader tccl = Thread.currentThread().getContextClassLoader();
try {
    Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
    /*
     * initialize the client
     */
} finally {
    Thread.currentThread().setContextClassLoader(tccl);
}

See also this forum thread.


More classloader errors

If you get an error like this:

Caused by: java.lang.LinkageError: loader constraint violation: when resolving field "DATETIME" the class loader (instance of 
 org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) of the referring class, javax/xml/datatype/DatatypeConstants, and the class loader (instance of <bootloader>) for the field's resolved type, javax/xml/namespace/QName, have different Class objects for that type

the reason is most likely that you have classes in the bundle classpath of your bundle that contains classes which are also part of the JDK runtime library (java.*, javax.*). Remove these classes from the bundle and it should work.

See also this forum thread.