Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/HowTo/How to implement a crawler"

(Java implementations)
(Java implementations)
Line 77: Line 77:
 
* it is not required to implement a BundleActivator (this may change if SCA Nodes are used, then it may be required to register the Crawler with an SCA Domain)
 
* it is not required to implement a BundleActivator (this may change if SCA Nodes are used, then it may be required to register the Crawler with an SCA Domain)
 
* create a top level folder <tt>OSGI-INF</tt>
 
* create a top level folder <tt>OSGI-INF</tt>
* create a Component Description file in <tt>OSGI-INF</tt>. You can name the file as you like, but it is good practice to name it like the Crawler. Therein you have to provide a unique component name, it should be the same as the Crawler's name followed by DS (for DeclarativeService). Then you have to provide your implementation class and the service interface class, which is always <tt>org.eclipse.eilf.connectivity.framework.Crawler</tt>. Make sure to set attribute "immediate" of tag "component" to <tt>false</tt> and attribute "servicefactory" of tag "service " to <tt>true</tt>. This is required to let SCA dynamically create ServiceReferences. Finally you have to provide a value for the property <tt>org.eclipse.eilf.connectivity.framework.crawler.type</tt>. This is used by SCA to find the correct OSGi Declarative Service during runtime. The value has to be unique (it is used in SCA contribution files) and should be named like the Crawler. Here is an example:
+
* create a Component Description file in <tt>OSGI-INF</tt>. You can name the file as you like, but it is good practice to name it like the Crawler. Therein you have to provide a unique component name, it should be the same as the Crawler's name followed by DS (for DeclarativeService). Then you have to provide your implementation class and the service interface class, which is always <tt>org.eclipse.eilf.connectivity.framework.Crawler</tt>.  
  
 
'''filesystemcrawler.xml'''
 
'''filesystemcrawler.xml'''
 
<source lang="xml">
 
<source lang="xml">
 
<?xml version="1.0" encoding="UTF-8"?>
 
<?xml version="1.0" encoding="UTF-8"?>
<component name="FileSystemCrawlerDS" immediate="false">
+
<component name="FileSystemCrawlerDS" immediate="true">
 
     <implementation class="org.eclipse.eilf.connectivity.framework.crawler.filesystem.FileSystemCrawler" />
 
     <implementation class="org.eclipse.eilf.connectivity.framework.crawler.filesystem.FileSystemCrawler" />
     <service servicefactory="true">
+
     <service>
 
         <provide interface="org.eclipse.eilf.connectivity.framework.Crawler"/>
 
         <provide interface="org.eclipse.eilf.connectivity.framework.Crawler"/>
 
     </service>
 
     </service>
Line 105: Line 105:
 
// implementation goes here ...
 
// implementation goes here ...
 
}
 
}
 +
</source>
 +
 +
Make sure to set attribute "immediate" of tag "component" to <tt>false</tt> and attribute "servicefactory" of tag "service " to <tt>true</tt>. This is required to let SCA dynamically create ServiceReferences. Finally you have to provide a value for the property <tt>org.eclipse.eilf.connectivity.framework.crawler.type</tt>. This is used by SCA to find the correct OSGi Declarative Service during runtime. The value has to be unique (it is used in SCA contribution files) and should be named like the Crawler. Here is an example:
 +
 +
'''filesystemcrawler.xml'''
 +
<source lang="xml">
 +
<?xml version="1.0" encoding="UTF-8"?>
 +
<component name="FileSystemCrawlerDS" immediate="false">
 +
    <implementation class="org.eclipse.eilf.connectivity.framework.crawler.filesystem.FileSystemCrawler" />
 +
    <service servicefactory="true">
 +
        <provide interface="org.eclipse.eilf.connectivity.framework.Crawler"/>
 +
    </service>
 +
    <property name="org.eclipse.eilf.connectivity.framework.crawler.type" value="filesystemcrawler"/>
 +
</component>
 
</source>
 
</source>
  

Revision as of 11:11, 18 September 2008

Java implementations

  • Create a new eclipse Project using the Plug-in Project Wizard and select "Equinox" as OSGi framework
  • name the project using the prefix org.eclipse.eilf.connectivity.framework.crawler.
  • edit the manifest and add the follwing to Imported Packages
    • org.eclipse.eilf.connectivity.framework
    • org.eclipse.eilf.connectivity.framework.utils
    • org.eclipse.eilf.datamodel.record
    • org.osoa.sca.annotations
    • com.sun.xml.bind.v2;version="2.1.6"
    • javax.xml.bind;version="2.1.0"
    • javax.xml.bind.annotation;version="2.1.0"
    • javax.xml.bind.annotation.adapters;version="2.1.0"
    • javax.xml.stream;version="1.0.0"
    • org.apache.commons.logging;version="1.1.1"
  • edit the manifest and add the follwing to <tt>Require-Bundle: (it MUST be imported via reguire bundle instead of package import)</tt>
    • org.eclipse.eilf.connectivity.framework.indexorder
  • Add "code/gen" to source folders (build.properties : source.. = code/src/,code/gen/)
    • click right on your bundle
    • go to the menu item New
    • click on Source folder
    • folder name: code/gen
  • Copy content of the folder "schema-compile-runtime\schema-pattern\" into your crawler bundle folder
  • Complile schema into JAXB classes by running schema.cmd ( "schema-compile-runtime" folder should be located on the projects level - it will be used for compilation )
    • start schema.cmd from cmd console, to see the result or error messages
  • Implement XSD schema for crawler configuration using template "schemas\TemplateIndexOrder.xsd"
    • create a new class (IndexOrderSchemaImpl) which implements interface IndexOrderSchema
    • use method String getSchemaLocation() to return "schemas/TemplateIndexOrder.xsd"
package org.eclipse.eilf.connectivity.framework.crawler.filesystem;
 
import org.eclipse.eilf.connectivity.framework.indexorder.IndexOrderSchema;
 
/**
 * The Class IndexOrderSchemaImpl.
 */
public class IndexOrderSchemaImpl implements IndexOrderSchema {
 
  /**
   * {@inheritDoc}
   * 
   * @see org.eclipse.eilf.connectivity.framework.indexorder.IndexOrderSchema#getSchemaLocation()
   */
  public String getSchemaLocation() {
    return "schemas/filesystemIndexOrder.xsd";
  }
}


  • Implement extension for "org.eclipse.eilf.connectivity.framework.indexorder.schema" with the bundle name used as ID and NAME
    • check schema class and change if it is necessary
<plugin>
   <extension
         id="org.eclipse.eilf.connectivity.framework.crawler.filesystem"
         name="org.eclipse.eilf.connectivity.framework.crawler.filesystem"
         point="org.eclipse.eilf.connectivity.framework.indexorder.schema">
      <schema
            class="org.eclipse.eilf.connectivity.framework.crawler.filesystem.IndexOrderSchemaImpl">
      </schema>
   </extension>
</plugin>
  • Note: if you renamed schema file name, it should be fixed inside
    • plug-in implementation
    • TemplateIndexOrder.jxb (it also should be also renamed with the same name as schema)
    • schema.cmd
  • implement your crawler in a new Class extending org.eclipse.eilf.connectivity.framework.AbstractCrawler


OSGi and Declarative Service requirements

  • it is not required to implement a BundleActivator (this may change if SCA Nodes are used, then it may be required to register the Crawler with an SCA Domain)
  • create a top level folder OSGI-INF
  • create a Component Description file in OSGI-INF. You can name the file as you like, but it is good practice to name it like the Crawler. Therein you have to provide a unique component name, it should be the same as the Crawler's name followed by DS (for DeclarativeService). Then you have to provide your implementation class and the service interface class, which is always org.eclipse.eilf.connectivity.framework.Crawler.

filesystemcrawler.xml

<?xml version="1.0" encoding="UTF-8"?>
<component name="FileSystemCrawlerDS" immediate="true">
    <implementation class="org.eclipse.eilf.connectivity.framework.crawler.filesystem.FileSystemCrawler" />
    <service>
         <provide interface="org.eclipse.eilf.connectivity.framework.Crawler"/>
    </service>
    <property name="org.eclipse.eilf.connectivity.framework.crawler.type" value="filesystemcrawler"/>
</component>
  • add a Service-Component entry to your manifest file, e.g.
Service-Component: OSGI-INF/filesystemcrawler.xml

SCA requirements

Most requirements for SCA are already handled by the base class org.eclipse.eilf.connectivity.framework.AbstractCrawler. You should annotate your implementation with @AllowsPassByReference, to allow SCA to pass service parameters by reference when service interactions are within the same adress space. Here is an example:

@AllowsPassByReference
public class FileSystemCrawler extends AbstractCrawler {
// implementation goes here ...
}

Make sure to set attribute "immediate" of tag "component" to false and attribute "servicefactory" of tag "service " to true. This is required to let SCA dynamically create ServiceReferences. Finally you have to provide a value for the property org.eclipse.eilf.connectivity.framework.crawler.type. This is used by SCA to find the correct OSGi Declarative Service during runtime. The value has to be unique (it is used in SCA contribution files) and should be named like the Crawler. Here is an example:

filesystemcrawler.xml

<?xml version="1.0" encoding="UTF-8"?>
<component name="FileSystemCrawlerDS" immediate="false">
    <implementation class="org.eclipse.eilf.connectivity.framework.crawler.filesystem.FileSystemCrawler" />
    <service servicefactory="true">
         <provide interface="org.eclipse.eilf.connectivity.framework.Crawler"/>
    </service>
    <property name="org.eclipse.eilf.connectivity.framework.crawler.type" value="filesystemcrawler"/>
</component>

IndexOrderConfiguration

Index Order Configaration based on XSD schema redefinition of abstract "RootIndexOrderConfiguration" schema

<xs:schema elementFormDefault="qualified" attributeFormDefault="unqualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
	<xs:redefine schemaLocation="RootIndexOrderConfiguration.xsd">
		<xs:complexType name="Process">
			<xs:annotation>
				<xs:documentation>Process Specification</xs:documentation>
			</xs:annotation>
			<xs:complexContent>
				<xs:extension base="Process">
					<\!--define process here -->
				</xs:extension>
			</xs:complexContent>
		</xs:complexType>
		<xs:complexType name="Attribute">
			<xs:complexContent>
				<xs:extension base="Attribute">
					<\!--define attribute here -->
				</xs:extension>
			</xs:complexContent>
		</xs:complexType>
	</xs:redefine>
</xs:schema>
  • Developer should define redefinition of "Process" and "Attribute" nodes for crawler specific information.
  • Cofigure all dependencies in MANIFEST.MF (see main section on the top)
Require-Bundle: org.eclipse.eilf.connectivity.framework.indexorder
Import-Package: com.sun.xml.bind.v2;version="2.1.6",
 javax.xml.bind;version="2.1.0",
 javax.xml.bind.annotation;version="2.1.0",
 javax.xml.bind.annotation.adapters;version="2.1.0",
 javax.xml.stream;version="1.0.0",
 org.apache.commons.io;version="1.4.0",
 org.apache.commons.logging;version="1.1.1",
 org.eclipse.eilf.connectivity.framework,
 org.eclipse.eilf.connectivity.framework.utils,
 org.eclipse.eilf.datamodel.record,
 org.osoa.sca.annotations
  • Compile schema into JAXB classes (see main section on the top)
  • Enjoy

Back to the top