|
|
(21 intermediate revisions by one other user not shown) |
Line 14: |
Line 14: |
| The biggest issue with the integration is that Kepler's classes are not set up for OSGi. [[Triquetrum/Extending Triquetrum]] discusses how to set up the packages. | | The biggest issue with the integration is that Kepler's classes are not set up for OSGi. [[Triquetrum/Extending Triquetrum]] discusses how to set up the packages. |
| | | |
− | === Kepler Installation === | + | == Status == |
| + | As of June 7, 2016, porting the Kepler ExecutionChoice actor to Triquetrum is partially complete. The ExecutionChoice actor was chosen because it is an arbitrarily complex actor that would not be an easy port. The primary issue is that the ExecutionChoice code uses the Swing-based BasicGraphFrame code that is not present in Triquetrum. The next step would be to get the ptolemy.actor.lib.hoc.Case actor working in Triquetrum because the Case actor has internal models that need to be viewable. Right now, the UI uses ptolemy.vergil.modal.CaseGraphFrame and CaseGraphTableau, which use BasicGraphFrame, which is not present in Triquetrum because Triquetrum does not use the Swing-based viewer. After that, the next step would be to refactor ExecutionChoice to follow the Case example where the UI is in ptolemy.vergil.modal.CaseGraphFrame and CaseGraphTableau. |
| + | |
| + | The current work is in checked in to a repository, see [[Triquetrum/Kepler Triquetrum Repository]]. |
| + | |
| + | == Kepler Installation == |
| # See [https://kepler-project.org/developers/reference/kepler-and-eclipse Kepler and Eclipse] for how to download and configure Kepler. To get the DDP work, we use the biokepler configuration, so: | | # See [https://kepler-project.org/developers/reference/kepler-and-eclipse Kepler and Eclipse] for how to download and configure Kepler. To get the DDP work, we use the biokepler configuration, so: |
| mkdir kepler | | mkdir kepler |
Line 23: |
Line 28: |
| ant clean-cache | | ant clean-cache |
| ant eclipse | | ant eclipse |
− | # Import the Kepler projects into the same workspace as to where Triquetrum is set up | + | # Import the Kepler projects into a new workspace that is not the same as where Triquetrum is set up |
| # Create a Kepler Run configuration an run it: | | # Create a Kepler Run configuration an run it: |
| * project: biokepler | | * project: biokepler |
| * main class: org.kepler.Kepler | | * main class: org.kepler.Kepler |
| | | |
− | ==== Kepler Fails to start: ProvKAREntryHandler$Factory ====
| + | === Kepler Fails to start: ProvKAREntryHandler$Factory === |
| If, during startup, we get this message: | | If, during startup, we get this message: |
| | | |
Line 52: |
Line 57: |
| Our first goal is to get Kepler running and count words using Unix commands. | | Our first goal is to get Kepler running and count words using Unix commands. |
| | | |
− | == Creating a new Kepler Bundle ==
| |
− | See [[Triquetrum/Extending_Triquetrum#Creating_an_actor_bundle | Creating an actor bundle]].
| |
− |
| |
− | # Create the plug-in,
| |
− | ## In Eclipse: File -> New Plug-in Project
| |
− | ## In the dialog, set the Project Name: to org.kepler.triquetrum.ddpDemo
| |
− | ## Click Finish
| |
− | # In the org.kepler.triquetrum.ddpDemo package, update MANIFEST.MF:
| |
− | <pre>
| |
− | Import-Package: org.apache.commons.lang;version="2.6.0",
| |
− | org.osgi.framework;version="1.8.0",
| |
− | org.ptolemy.classloading;version="11.0.0",
| |
− | org.ptolemy.classloading.osgi;version="11.0.0",
| |
− | org.ptolemy.commons;version="11.0.0",
| |
− | ptolemy.actor;version="11.0.0",
| |
− | ptolemy.actor.gui;version="11.0.0",
| |
− | ptolemy.data;version="11.0.0",
| |
− | ptolemy.data.expr;version="11.0.0",
| |
− | ptolemy.data.type;version="11.0.0",
| |
− | ptolemy.gui;version="11.0.0",
| |
− | ptolemy.kernel;version="11.0.0",
| |
− | ptolemy.kernel.util;version="11.0.0",
| |
− | ptolemy.moml;version="11.0.0",
| |
− | ptolemy.util;version="11.0.0"
| |
− | Export-Package: org.kepler;version="1.0.0"
| |
− | </pre>
| |
− | # Create the p2.inf file (See [[Triquetrum/Extending_Triquetrum#Creating_an_actor_bundle | Creating an actor bundle]])
| |
− | ## Open the org.eclipse.triquetrum.workflow.actor.plot project, click on META-INF, highlight the p2.inf file and select copy
| |
− | ## In the org.kepler.triquetrum.ddpDemo project click on META-INF and paste
| |
− | ## The p2.inf file will be created.
| |
− |
| |
− | == What Kepler classes are necessary? ==
| |
− |
| |
− | '''FIXME: It turns out that doing the steps below is probably not the right way to add actors. Instead, a better way is to add each actor and the classes the actor depends on'''
| |
− |
| |
− | The next step is to determine what Kepler classes are necessary.
| |
− |
| |
− | The demo primarily uses [http://www.biokepler.org/userguide#execchoice org.kepler.ddp.actor.ExecutionChoice], which is a composite actor that contains submodels that implement the different ways the actor can be executed.
| |
− |
| |
− | The -verbose option of the java command will report what classes are loaded.
| |
− |
| |
− | Unfortunately, it seems that adding -verbose to the run configuration in Eclipse does not report the Kepler classes that are loaded, so we use the command line
| |
− |
| |
− | To run the model without the UI from the command line
| |
− |
| |
− | cd kepler/build_area
| |
− | ant run -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml
| |
− |
| |
− | Edit kepler/build-area/src/org/kepler/build/Run.java and insert the following into the runSuite() method:
| |
− | java.createJvmarg().setLine("-verbose");
| |
− |
| |
− | Compile the build system:
| |
− | ant -f kepler-tasks.xml
| |
− |
| |
− | Run the model and save the output to a file:
| |
− | ant run -v -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml >& /tmp/run.txt
| |
− |
| |
− | To see how many org.kepler classes are loaded (the sed command converts any inner classes that have $ in their name to the parent class, so org.kepler.util.sql.Table$IndexType becomes org.kepler.util.sql.Table)
| |
− | grep 'Loaded' /tmp/run.txt | grep org.kepler | awk '{print $3}' | sed 's/\$.*\//' | sort | uniq | sed 's@\.@/@g' | awk '{print $1 ".java"}' > /tmp/files
| |
− |
| |
− | '''FIXME''': Kepler classes should not be in org.kepler because the Kepler project does not have the kepler.org domain name.
| |
− |
| |
− | The above returns 188 files. Not all of these classes will be necessary at runtime when running the model, but the important thing here is to make a guess at what classes are necessary.
| |
− |
| |
− | The script below, when run in the kepler directory, will create a tar file with all the .java files:
| |
− | <pre>
| |
− | #!/bin/sh
| |
− |
| |
− | if [ ! -d files ]; then
| |
− | mkdir files
| |
− | fi
| |
− |
| |
− | files=`cat /tmp/files`
| |
− | for file in $files
| |
− | do
| |
− | srcs=`ls -1d */src`
| |
− | for src in $srcs
| |
− | do
| |
− | if [ -f $src/$file ]; then
| |
− | (cd $src; tar -cf - $file) | (cd files; tar -xvf -)
| |
− | break;
| |
− | fi
| |
− | done
| |
− | done
| |
− |
| |
− | (cd files; tar -cf ../files.tar .)
| |
− | ls -l files.tar
| |
− | </pre>
| |
− |
| |
− | Place the above in a file called kepler/findKeplerFiles.
| |
− |
| |
− | Run the script with:
| |
− | cd kepler
| |
− | sh ./findKeplerFiles
| |
− |
| |
− | Create copies of the files in the org.kepler.triquetrum.ddpDemo module (your pathname will vary)
| |
− | cat files.tar | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src; tar -xf -)
| |
− |
| |
− | '''FIXME''' This is really wrong because we are creating one bundle for all the Kepler files used by one demo. A better way would create multiple bundles for the various Kepler modules.
| |
− |
| |
− | == Tracking Down Dependencies ==
| |
− | Right click on the org.kepler.triquetrum.ddpDemo project and select refresh. In the Problems tab, there will be many compilation problems. Below are the problems we saw and the solutions:
| |
− |
| |
− | === _controller cannot be resolved in KeplerGraphFrame ===
| |
− | The error is:
| |
− | Description Resource Path Location Type
| |
− | _controller cannot be resolved to a variable KeplerGraphFrame.java /org.kepler.triquetrum.ddpDemo/src/org/kepler/gui line 245 Java Problem
| |
− |
| |
− | KeplerGraphFrame extends ptolemy.vergil.actor.ActorGraphFrame. Triquetrum does not use ActorGraphFrame
| |
− |
| |
− | The solution is to remove org.kepler.triquetrum.ddpDemo/src/org/kepler/gui/KeplerGraphFrame.java
| |
− |
| |
− | === _debugging cannot be resolved in ExecutionChoiceDirector ===
| |
− | The error is:
| |
− | Description Resource Path Location Type
| |
− | _debugging cannot be resolved to a variable ExecutionChoiceDirector.java /org.kepler.triquetrum.ddpDemo/src/org/kepler/ddp/actor line 101 Java Problem
| |
− |
| |
− | The problem is that ExecutionChoiceDirector extends CaseDirector, which is not found.
| |
− |
| |
− | Looking at the errors in the imports, we see we need Case and Refinement as well. Case extends MultiInstanceComposite, so we need that.
| |
− |
| |
− | The solution is to copy over these files:
| |
− | (cd $PTII; tar -cf - ptolemy/actor/lib/hoc/{Case,CaseDirector,MultiCompositeActor,MultiCompositePort,Refinement,RefinementPort}.java ptolemy/actor/util/Time.java ptolemy/math/ExtendedMath.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src/; tar -xvf -)
| |
− |
| |
− | === DDFDirector is needed by ExecutionChoiceDirector ===
| |
− | Copy over DDFDirector and ActorEnablingStatus, which is needed by DDFDirector. Also copy over DFUtilities, NotSchedulableException, Time and ExtendedMath:
| |
− | (cd $PTII; tar -cf - ptolemy/actor/lib/hoc/{Case,CaseDirector,MultiCompo\
| |
− | siteActor,MultiCompositePort,Refinement,RefinementPort}.java ptolemy/actor/util/Ti\
| |
− | me.java ptolemy/math/ExtendedMath.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.\
| |
− | triquetrum.ddpDemo/src/; tar -xvf -)
| |
− |
| |
− | === ParameterPort and PortParameter are needed by ExecutionChoiceDirector ===
| |
− | The error is:
| |
− | Description Resource Path Location Type
| |
− | Access restriction: The type 'ParameterPort' is not API (restriction on required library
| |
− | '/Users/cxh/src/workspaceTriq04Mar/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool /plugins/ptolemy.core_11.0.0.201605041253.jar') ExecutionChoiceDirector.java
| |
− | /org.kepler.triquetrum.ddpDemo/src/org/kepler/ddp/actor line 44 Java Problem
| |
− |
| |
− | '''FIXME''': The issue here is that PortParameter and ParameterPort are in the ptolemy.core module, but that module is not available
| |
− |
| |
− | The workaround is to copy them:
| |
− | (cd $PTII; tar -cf - ptolemy/actor/parameters/{ParameterPort,PortParameter}.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src/; tar -xvf -)
| |
− |
| |
− | It turns out that we need a number of classes from ptolemy.core. Unfortunately, adding
| |
− | ptolemy.core;version="11.0.0",
| |
− | to the MANIFEST.MF file results in an error message: "No available bundle exports ptolemy.core"
| |
− |
| |
− | However, the org.kepler.triquetrum.ddpDemo project lists the following as a Plug-in Dependency
| |
− | /Users/cxh/src/workspaceTriq04Mar/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool/plugins/ptolemy.core_11.0.0.201605041253.jar
| |
− |
| |
− | That jar file has the following MANIFEST.MF:
| |
− | <pre>
| |
− | Manifest-Version: 1.0^M
| |
− | Bundle-SymbolicName: ptolemy.core^M
| |
− | Export-Package: org.ptolemy.classloading;version="11.0.0",org.ptolemy.^M
| |
− | classloading.osgi;version="11.0.0",ptolemy.actor;version="11.0.0",pto^M
| |
− | lemy.actor.parameters;version="11.0.0",ptolemy.actor.sched;version="1^M
| |
− | 1.0.0",ptolemy.actor.util;version="11.0.0",ptolemy.data;version="11.0^M
| |
− | .0",ptolemy.data.expr;version="11.0.0",ptolemy.data.type;version="11.^M
| |
− | 0.0",ptolemy.data.unit;version="11.0.0",ptolemy.graph;version="11.0.0^M
| |
− | ",ptolemy.graph.analysis;version="11.0.0",ptolemy.graph.analysis.anal^M
| |
− | yzer;version="11.0.0",ptolemy.graph.analysis.strategy;version="11.0.0^M
| |
− | ",ptolemy.graph.mapping;version="11.0.0",ptolemy.kernel;version="11.0^M
| |
− | .0",ptolemy.kernel.attributes;version="11.0.0",ptolemy.kernel.undo;ve^M
| |
− | rsion="11.0.0",ptolemy.kernel.util;version="11.0.0",ptolemy.math;vers^M
| |
− | ion="11.0.0",ptolemy.util;version="11.0.0"^M
| |
− | Bundle-Name: Ptolemy Core^M
| |
− | Bundle-Version: 11.0.0.201605041253^M
| |
− | Bundle-ManifestVersion: 2^M
| |
− | Bundle-RequiredExecutionEnvironment: JavaSE-1.7^M
| |
− | Bundle-Activator: ptolemy.kernel.activator.Activator^M
| |
− | Eclipse-BuddyPolicy: registered^M
| |
− | Bundle-Vendor: CHESS^M
| |
− | Import-Package: org.osgi.framework;version="1.3.0",org.osgi.util.track^M
| |
− | er;version="1.5.1",org.ptolemy.commons;version="11.0.0"^M
| |
− | </pre>
| |
− |
| |
− | However, if I right click on the org.kepler.triquetrum.ddpDemo project, select Build Path -> Configure Build Path, then under the Libraries tab, expand Plug-in Dependencies, scroll to ptolemy.core..., expand it and expand the Access rules tab, I can see that ptolemy/actor/parameters/* is not present
| |
− |
| |
− | [[File:TriquetrumKeplerPtolemyCoreAccessRules.png|thumb|ptolemy.core Access rules]]
| |
− |
| |
− | The workaround is to add ptolemy/actor/parameters/* and move it above the forbidden rule.
| |
− |
| |
− | For ptolemy.core, add:
| |
− | * ptolemy/actors/parameters/*
| |
− | * ptolemy/graph/*
| |
− |
| |
− | For ptolemy.moml, add:
| |
− | * ptolemy/moml/filters/*
| |
− |
| |
− | For ptolemy.gui, add
| |
− | * ptolemy/actor/gui/style/*
| |
− |
| |
− | However, it seems like the above changes are not persistent?
| |
− |
| |
− | FIXME: Where are these Access rules set?
| |
− |
| |
− | === org.apache.commons.io.FilenameUtils is missing from ExecutionChoice ===
| |
− | The error:
| |
− | import org.apache.commons.io.FilenameUtils;
| |
− | Solution: Add the following to org.kepler.triquetrum.ddpDemo MANIFEST.MF:
| |
− | org.apache.commons.io;version="2.2.0",
| |
− |
| |
− |
| |
− | === ExecutionChoice uses lots of GUI code ===
| |
− |
| |
− | The imports below are missing from ExecutionChoice:
| |
− |
| |
− | import org.kepler.gui.KeplerGraphFrame;
| |
− | import org.kepler.gui.ModelToFrameManager;
| |
− | import org.kepler.gui.frame.TabbedKeplerGraphFrame;
| |
− |
| |
− | import ptolemy.actor.gui.EditorFactory;
| |
− |
| |
− | import ptolemy.gui.ComponentDialog;
| |
− |
| |
− | import ptolemy.gui.ExtensionFilenameFilter;
| |
− | import ptolemy.gui.JFileChooserBugFix;
| |
− | import ptolemy.gui.PtFileChooser;
| |
− | import ptolemy.gui.PtGUIUtilities;
| |
− | import ptolemy.gui.Query;
| |
− |
| |
− | '''FIXME''': Not sure what to do about these. It seems like ExecutionChoice has a very tight connection to GUI classes like KeplerGraphFrame that are not present in Triquetrum
| |
− |
| |
− | === Hadoop ===
| |
− | The error is:
| |
− | Description Resource Path Location Type
| |
− | Configuration cannot be resolved to a type DDPDataSink.java /org.kepler.triquetrum.ddpDemo/src/org/kepler/ddp/actor/pattern line 258 Java Problem
| |
− |
| |
− | The issue is that these imports in DDPDataSink are not present
| |
− | import org.apache.hadoop.conf.Configuration;
| |
− | import org.apache.hadoop.fs.FileSystem;
| |
− | import org.apache.hadoop.fs.FileUtil;
| |
− | import org.apache.hadoop.fs.Path;
| |
− |
| |
− | One solution would be to import the jars from ~/KeplerData/workflows/module/hadoop/tools/share/hadoop/
| |
− |
| |
− | A better solution would be to find a hadoop Eclipse bundle upon which we could depend.
| |
− |
| |
− | Kepler provides a hadoop project that could be helpful here!
| |
− |
| |
− | Solution: Right click on the org.kepler.triquetrum.ddpDemo project -> Build Path -> Configure Build Path and add the following package dependencies
| |
− | * configuration-manager (For apache logging) '''FIXME''': Might be able to get this elsewhere
| |
− | * ddp-common
| |
− | * hadoop
| |
− |
| |
− | === ExecutionChoice Revisited ===
| |
− | After looking at the number of changes above, a different approach of just adding the ExecutionChoice actor might be in order.
| |
− |
| |
− | The first step was to modify the Plug-in dependencies for ptolemy.core, ptolemy.actor and ptolemy.gui as above.
| |
− |
| |
− | ExecutionChoice depends on KeplerDocumentationAttribute, which depends on DocAttribute, so we copy those files:
| |
− | (cd $PTII; tar -cf - ptolemy/vergil/basic/{DocAttribute,KeplerDocumentationAttribute}.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src/; tar -xvf -)
| |
− |
| |
− | == Providing the ExecutionChoice actor to the run time ==
| |
− | Based on [[Triquetrum/Extending_Triquetrum#Providing_the_actor_class_to_the_runtime | Providing the actor class to the runtime]]
| |
− |
| |
− | Create a package called org.kepler.ddp.actor.activator and then create Activator.java in that package:
| |
− | <pre>
| |
− | package org.kepler.ddp.actor.activator;
| |
− |
| |
− |
| |
− | import org.kepler.ddp.actor.ExecutionChoice;
| |
− | import org.osgi.framework.BundleActivator;
| |
− | import org.osgi.framework.BundleContext;
| |
− | import org.osgi.framework.ServiceRegistration;
| |
− | import org.osgi.framework.Version;
| |
− | import org.ptolemy.classloading.ModelElementClassProvider;
| |
− | import org.ptolemy.classloading.osgi.DefaultModelElementClassProvider;
| |
− | import org.ptolemy.commons.ThreeDigitVersionSpecification;
| |
− | import org.ptolemy.commons.VersionSpecification;
| |
− |
| |
− | public class Activator implements BundleActivator {
| |
− | public void start(BundleContext context) throws Exception {
| |
− |
| |
− | // FIXME figure out a more compact way to create a version-aware provider,
| |
− | // that uses the bundle version but is not too dependent on OSGi APIs itself.
| |
− | Version bundleVersion = context.getBundle().getVersion();
| |
− | VersionSpecification providerVersion = new ThreeDigitVersionSpecification(
| |
− | bundleVersion.getMajor(),
| |
− | bundleVersion.getMinor(),
| |
− | bundleVersion.getMicro(),
| |
− | bundleVersion.getQualifier());
| |
− |
| |
− | _apSvcReg = context.registerService(ModelElementClassProvider.class.getName(),
| |
− | new DefaultModelElementClassProvider(providerVersion, ExecutionChoice.class),
| |
− | null);
| |
− | }
| |
− |
| |
− | public void stop(BundleContext context) throws Exception {
| |
− | _apSvcReg.unregister();
| |
− | }
| |
− |
| |
− | // private stuff
| |
− | /** The svc registration for the actor provider */
| |
− | private ServiceRegistration<?> _apSvcReg;
| |
− |
| |
− | }
| |
− | </pre>
| |
− |
| |
− | == Packaging the plugin(s) for your RCP editor ==
| |
− | See [[https://wiki.eclipse.org/Triquetrum/Extending_Triquetrum#Packaging_the_plugin.28s.29_for_your_RCP_editor | Packaging the plugin(s) for your RCP editor]]
| |
− |
| |
− | Create a simple (non-Java plugin)
| |
| == Resources == | | == Resources == |
| + | * [[Triquetrum/Kepler Triquetrum Repository]] - The current effort to use Kepler actors in Triquetrum. (Start here) |
| + | * [[Triquetrum/Kepler Feature]] - How the [[Triquetrum/Kepler Triquetrum Repository]] was created |
| + | * [[Triquetrum/Kepler Tree Shaking]] - a first attempt at including all of the classes that were used by Kepler at run time. |
| * [https://github.com/eclipse/triquetrum/issues/84 Bug 84: Test out integrating Kepler code into Triquetrum: Use DDP/Hadoop as driver.] | | * [https://github.com/eclipse/triquetrum/issues/84 Bug 84: Test out integrating Kepler code into Triquetrum: Use DDP/Hadoop as driver.] |
| * [[Triquetrum/Extending Triquetrum]] | | * [[Triquetrum/Extending Triquetrum]] |
| + | |
| + | [[Category:Triquetrum]] |
This page is a work in progress.
Note that the Kepler Scientific Workflow System is not the same as the Kepler release of Eclipse. In this page, the term Kepler is used to mean the Kepler Scientific Workflow System.
The biggest issue with the integration is that Kepler's classes are not set up for OSGi. Triquetrum/Extending Triquetrum discusses how to set up the packages.
As of June 7, 2016, porting the Kepler ExecutionChoice actor to Triquetrum is partially complete. The ExecutionChoice actor was chosen because it is an arbitrarily complex actor that would not be an easy port. The primary issue is that the ExecutionChoice code uses the Swing-based BasicGraphFrame code that is not present in Triquetrum. The next step would be to get the ptolemy.actor.lib.hoc.Case actor working in Triquetrum because the Case actor has internal models that need to be viewable. Right now, the UI uses ptolemy.vergil.modal.CaseGraphFrame and CaseGraphTableau, which use BasicGraphFrame, which is not present in Triquetrum because Triquetrum does not use the Swing-based viewer. After that, the next step would be to refactor ExecutionChoice to follow the Case example where the UI is in ptolemy.vergil.modal.CaseGraphFrame and CaseGraphTableau.
The solution is to select use the biokepler project, not the kepler project.
Kepler will start up. Close the splash window and under Components, double click on DDP and then wordcount-execution-choice.xml. Below is a screen shot of the initial Kepler window:
The wordcount demo has multiple ways of counting words. The default way is to use Unix commands.
To use the other ways of counting words, double click on the Word Count actor and change the Choice parameter.
Our first goal is to get Kepler running and count words using Unix commands.