Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Triquetrum/Kepler"

(Hadoop dependency)
(LibraryAttribute was missing)
Line 143: Line 143:
  
 
=== Various Ptolemy Classes are Missing ===
 
=== Various Ptolemy Classes are Missing ===
DDFDirector, various actor.hoc and KeplerDocumentationAttribute classes are missing, so we copy them over.
+
DDFDirector, various actor.hoc, KeplerDocumentationAttribute and LibraryAttribute classes are missing, so we copy them over.
  
 
'''FIXME''': These should be added as a bundle to Triquetrum.
 
'''FIXME''': These should be added as a bundle to Triquetrum.
Line 151: Line 151:
 
   ptolemy/actor/util/DFUtilities.java \
 
   ptolemy/actor/util/DFUtilities.java \
 
   ptolemy/actor/lib/hoc/{Case,CaseDirector,MultiCompositeActor,MultiCompositePort,Refinement,RefinementPort}.java \
 
   ptolemy/actor/lib/hoc/{Case,CaseDirector,MultiCompositeActor,MultiCompositePort,Refinement,RefinementPort}.java \
   ptolemy/actor/util/Time.java ptolemy/math/ExtendedMath.java
+
   ptolemy/actor/util/Time.java ptolemy/math/ExtendedMath.java \
   ptolemy/vergil/basic/{DocAttribute,KeplerDocumentationAttribute}.java) \
+
   ptolemy/vergil/basic/{DocAttribute,KeplerDocumentationAttribute}.java \
 +
  ptolemy/moml/LibraryAttribute.java) \
 
   | (cd ~/src/workspaceTriq03JunB/org.kepler.triquetrum.ddp-common/src/; tar -xvf -)
 
   | (cd ~/src/workspaceTriq03JunB/org.kepler.triquetrum.ddp-common/src/; tar -xvf -)
 
</pre>
 
</pre>

Revision as of 18:55, 6 June 2016

This page is a work in progress.

The goal of this page is to illustrate how to use Kepler Scientific Workflow System Actors in Triquetrum.

Note that the Kepler Scientific Workflow System is not the same as the Kepler release of Eclipse. In this page, the term Kepler is used to mean the Kepler Scientific Workflow System.

Kepler

The Kepler Scientific Workflow System has actors that we would like to use in Triquetrum. In this case, we would like to use Kepler's Distributed Data-Parallel (DDP) actors.

For a brief overview of DDP, see:

The biggest issue with the integration is that Kepler's classes are not set up for OSGi. Triquetrum/Extending Triquetrum discusses how to set up the packages.

Kepler Installation

  1. See Kepler and Eclipse for how to download and configure Kepler. To get the DDP work, we use the biokepler configuration, so:
  mkdir kepler
  cd kepler
  svn co https://code.kepler-project.org/code/kepler/trunk/modules/build-area
  cd build-area/
  ant change-to -Dsuite=biokepler  
  ant clean-cache
  ant eclipse
  1. Import the Kepler projects into a new workspace that is not the same as where Triquetrum is set up
  2. Create a Kepler Run configuration an run it:
  • project: biokepler
  • main class: org.kepler.Kepler

Kepler Fails to start: ProvKAREntryHandler$Factory

If, during startup, we get this message:

java.lang.ClassNotFoundException: org.kepler.kar.handlers.ProvKAREntryHandler$Factory
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

The solution is to select use the biokepler project, not the kepler project.

Run the DDP demo

Kepler will start up. Close the splash window and under Components, double click on DDP and then wordcount-execution-choice.xml. Below is a screen shot of the initial Kepler window:

KeplerMainDDPWordCount.png

The wordcount demo has multiple ways of counting words. The default way is to use Unix commands.

Below is a run that counts words using Unix commands:

KeplerWorkflowSystemLocalExecution.png

To use the other ways of counting words, double click on the Word Count actor and change the Choice parameter.

Our first goal is to get Kepler running and count words using Unix commands.

Creating a new Kepler Bundle

See Creating an actor bundle.

  1. Create the plug-in,
    1. In Eclipse: File -> New Plug-in Project
    2. In the dialog, set the Project Name: to org.kepler.triquetrum.ddp
    3. Click Next, then Finish
  2. In the org.kepler.triquetrum.ddpDemo package, update MANIFEST.MF:
Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 ptolemy.gui,
 ptolemy.core,
 ptolemy.moml

and

Import-Package: org.apache.commons.lang;version="2.6.0",
  org.osgi.framework;version="1.8.0",
  org.ptolemy.classloading;version="11.0.0",
  org.ptolemy.classloading.osgi;version="11.0.0",
  org.ptolemy.commons;version="11.0.0",
  ptolemy.actor;version="11.0.0",
  ptolemy.actor.gui;version="11.0.0",
  ptolemy.data;version="11.0.0",
  ptolemy.data.expr;version="11.0.0",
  ptolemy.data.type;version="11.0.0",
  ptolemy.gui;version="11.0.0",
  ptolemy.kernel;version="11.0.0",
  ptolemy.kernel.util;version="11.0.0",
  ptolemy.moml;version="11.0.0",
  ptolemy.util;version="11.0.0"
Export-Package: org.kepler;version="1.0.0"

FIXME: The Import-Package settings are probably not necessary.

  1. Create the p2.inf file (See Creating an actor bundle)
    1. Open the org.eclipse.triquetrum.workflow.actor.plot project, click on META-INF, highlight the p2.inf file and select copy
    2. In the org.kepler.triquetrum.ddp project click on META-INF and paste
    3. The p2.inf file will be created.

Create ExecutionChoice.java

  1. In the org.kepler.triquetrum.ddp project, right click on the src icon and click New -> Class
  2. Change the package to org.kepler.ddp.actor
  3. We will update the enclosing type later
  4. Use File -> Open to open the kepler/ddp-common/src/org/kepler/ddp/actor/ExecutionChoice.java file, copy and paste it in to the newly created file
  5. There will be lots of errors.

Fix the compilation errors in ExecutionChoice.java

Below are the steps taken to fix the compilation errors in ExecutionChoice.java

The import org.apache.commons.io.FileUtils cannot be found

The error:

import org.apache.commons.io.FilenameUtils;

Solution: Add the following to org.kepler.triquetrum.ddp MANIFEST.MF under Require-Bundle:

org.apache.commons.io,

Various Kepler classes are missing

In ExecutionChoice, these imports are failing:

 import org.kepler.build.modules.Module;
 import org.kepler.build.modules.ModuleTree;
 import org.kepler.ddp.gui.ExecutionChoiceEditorFactory;
 import org.kepler.ddp.gui.ExecutionChoiceEditorPane;
 import org.kepler.provenance.ProvenanceRecorder;
 import org.kepler.reporting.ReportingListener

The solution is to copy them over. An alternative would be to use the Eclipse projects for these Kepler projects.

(cd ~/src/kepler/build-area/src; \
 tar -cf - org/kepler/build/modules/{Module,ModuleTree}.java) \
 | (cd ~/src/workspaceTriq03JunB/org.kepler.triquetrum.ddp/src/; tar -xvf -)

(cd ~/src/kepler/ddp-common/src; \
 tar -cf - org/kepler/ddp/gui/ExecutionChoiceEditor{Factory,Pane}.java) \
 | (cd ~/src/workspaceTriq03JunB/org.kepler.triquetrum.ddp/src/; tar -xvf -)

(cd ~/src/kepler/provenance/src; \
 tar -cf - org/kepler/provenance/ProvenanceRecorder.java) \
 | (cd ~/src/workspaceTriq03JunB/org.kepler.triquetrum.ddp/src/; tar -xvf -)

(cd ~/src/kepler/reporting/src; \
 tar -cf - org/kepler/reporting/ReportingListener.java) \
 | (cd ~/src/workspaceTriq03JunB/org.kepler.triquetrum.ddp/src/; tar -xvf -)

Various Ptolemy Classes are Missing

DDFDirector, various actor.hoc, KeplerDocumentationAttribute and LibraryAttribute classes are missing, so we copy them over.

FIXME: These should be added as a bundle to Triquetrum.

(cd $PTII; \
  tar -cf - ptolemy/domains/ddf/kernel/{DDFDirector,ActorEnablingStatus}.java \
  ptolemy/actor/util/DFUtilities.java \
  ptolemy/actor/lib/hoc/{Case,CaseDirector,MultiCompositeActor,MultiCompositePort,Refinement,RefinementPort}.java \
  ptolemy/actor/util/Time.java ptolemy/math/ExtendedMath.java \
  ptolemy/vergil/basic/{DocAttribute,KeplerDocumentationAttribute}.java \
  ptolemy/moml/LibraryAttribute.java) \
  | (cd ~/src/workspaceTriq03JunB/org.kepler.triquetrum.ddp-common/src/; tar -xvf -)

Hadoop

The error is:

 Description	Resource	Path	Location	Type
Configuration cannot be resolved to a type	DDPDataSink.java	/org.kepler.triquetrum.ddpDemo/src/org/kepler/ddp/actor/pattern	line 258	Java Problem

The issue is that these imports in DDPDataSink are not present

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.FileUtil;
 import org.apache.hadoop.fs.Path;

One solution would be to import the jars from ~/KeplerData/workflows/module/hadoop/tools/share/hadoop/

A better solution would be to find a hadoop Eclipse bundle upon which we could depend.

Kepler provides a hadoop project that could be helpful here!

Solution: Right click on the org.kepler.triquetrum.ddpDemo project -> Build Path -> Configure Build Path and add the following package dependencie

  • ddp-common
  • hadoop

Resources

Back to the top