Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Triquetrum/Kepler"
(In the org.kepler.triquetrum.ddpDemo package, update MANIFEST.MF:) |
(Copy the necessary files over.) |
||
Line 51: | Line 51: | ||
Our first goal is to get Kepler running and count words using Unix commands. | Our first goal is to get Kepler running and count words using Unix commands. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== Creating a new Kepler Bundle == | == Creating a new Kepler Bundle == | ||
Line 113: | Line 84: | ||
## The p2.inf file will be created. | ## The p2.inf file will be created. | ||
+ | == What Kepler classes are necessary? == | ||
+ | The next step is to determine what Kepler classes are necessary. | ||
+ | |||
+ | The demo primarily uses [http://www.biokepler.org/userguide#execchoice org.kepler.ddp.actor.ExecutionChoice], which is a composite actor that contains submodels that implement the different ways the actor can be executed. | ||
+ | |||
+ | |||
+ | The -verbose option of the java command will report what classes are loaded. | ||
+ | |||
+ | Unfortunately, it seems that adding -verbose to the run configuration in Eclipse does not report the Kepler classes that are loaded, so we use the command line | ||
+ | |||
+ | To run the model without the UI from the command line | ||
+ | |||
+ | cd kepler/build_area | ||
+ | ant run -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml | ||
+ | |||
+ | Edit kepler/build-area/src/org/kepler/build/Run.java and insert the following into the runSuite() method: | ||
+ | java.createJvmarg().setLine("-verbose"); | ||
+ | |||
+ | Compile the build system: | ||
+ | ant -f kepler-tasks.xml | ||
+ | |||
+ | Run the model and save the output to a file: | ||
+ | ant run -v -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml >& /tmp/run.txt | ||
+ | |||
+ | To see how many org.kepler classes are loaded (the sed command converts any inner classes that have $ in their name to the parent class, so org.kepler.util.sql.Table$IndexType becomes org.kepler.util.sql.Table) | ||
+ | grep 'Loaded' /tmp/run.txt | grep org.kepler | awk '{print $3}' | sed 's/\$.*\//' | sort | uniq | sed 's@\.@/@g' | awk '{print $1 ".java"}' > /tmp/files | ||
+ | |||
+ | The above returns 188 files. Not all of these classes will be necessary at runtime when running the model, but the important thing here is to make a guess at what classes are necessary. | ||
+ | |||
+ | The script below, when run in the kepler directory, will create a tar file with all the .java files: | ||
+ | <pre> | ||
+ | #!/bin/sh | ||
+ | |||
+ | if [ ! -d files ]; then | ||
+ | mkdir files | ||
+ | fi | ||
+ | |||
+ | files=`cat /tmp/files` | ||
+ | for file in $files | ||
+ | do | ||
+ | srcs=`ls -1d */src` | ||
+ | for src in $srcs | ||
+ | do | ||
+ | if [ -f $src/$file ]; then | ||
+ | (cd $src; tar -cf - $file) | (cd files; tar -xvf -) | ||
+ | break; | ||
+ | fi | ||
+ | done | ||
+ | done | ||
+ | |||
+ | (cd files; tar -cf ../files.tar .) | ||
+ | ls -l files.tar | ||
+ | </pre> | ||
+ | |||
+ | Place the above in a file called kepler/findKeplerFiles. | ||
+ | |||
+ | Run the script with: | ||
+ | cd kepler | ||
+ | sh ./findKeplerFiles | ||
+ | Create copies of the files in the org.kepler.triquetrum.ddpDemo module (your pathname will vary) | ||
+ | cat files.tar | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src; tar -xf -) | ||
Revision as of 17:43, 1 June 2016
This page is a work in progress.
The goal of this page is to illustrate how to use Kepler Scientific Workflow System Actors in Triquetrum.
Note that the Kepler Scientific Workflow System is not the same as the Kepler release of Eclipse. In this page, the term Kepler is used to mean the Kepler Scientific Workflow System.
Contents
Kepler
The Kepler Scientific Workflow System has actors that we would like to use in Triquetrum. In this case, we would like to use Kepler's Distributed Data-Parallel (DDP) actors.
For a brief overview of DDP, see:
- Ilkay Altintas, Daniel Crawl, Jianwu Wang. A Distributed Data-Parallel Execution Framework in the Kepler Scientific Workflow System. October 16, 2015. Presented at the Eleventh Biennial Ptolemy Miniconference, Berkeley.
The biggest issue with the integration is that Kepler's classes are not set up for OSGi. Triquetrum/Extending Triquetrum discusses how to set up the packages.
Kepler Installation
- See Kepler and Eclipse for how to download and configure Kepler. To get the DDP work, we use the biokepler configuration, so:
mkdir kepler cd kepler svn co https://code.kepler-project.org/code/kepler/trunk/modules/build-area cd build-area/ ant change-to -Dsuite=biokepler ant clean-cache ant eclipse
- Import the Kepler projects into the same workspace as to where Triquetrum is set up
- Create a Kepler Run configuration an run it:
- project: biokepler
- main class: org.kepler.Kepler
Kepler Fails to start: ProvKAREntryHandler$Factory
If, during startup, we get this message:
java.lang.ClassNotFoundException: org.kepler.kar.handlers.ProvKAREntryHandler$Factory at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
The solution is to select use the biokepler project, not the kepler project.
Run the DDP demo
Kepler will start up. Close the splash window and under Components, double click on DDP and then wordcount-execution-choice.xml. Below is a screen shot of the initial Kepler window:
The wordcount demo has multiple ways of counting words. The default way is to use Unix commands.
Below is a run that counts words using Unix commands:
To use the other ways of counting words, double click on the Word Count actor and change the Choice parameter.
Our first goal is to get Kepler running and count words using Unix commands.
Creating a new Kepler Bundle
- Create the plug-in,
- In Eclipse: File -> New Plug-in Project
- In the dialog, set the Project Name: to org.kepler.triquetrum.ddpDemo
- Click Finish
- In the org.kepler.triquetrum.ddpDemo package, update MANIFEST.MF:
Import-Package: org.apache.commons.lang;version="2.6.0", org.eclipse.draw2d, org.eclipse.nebula.visualization.xygraph.dataprovider, org.eclipse.nebula.visualization.xygraph.figures, org.eclipse.swt.layout, org.eclipse.swt.widgets, org.eclipse.triquetrum.workflow.ui;version="1.0.0", org.osgi.framework;version="1.8.0", org.ptolemy.classloading;version="11.0.0", org.ptolemy.classloading.osgi;version="11.0.0", org.ptolemy.commons;version="11.0.0", ptolemy.actor;version="11.0.0", ptolemy.data;version="11.0.0", ptolemy.data.expr;version="11.0.0", ptolemy.data.type;version="11.0.0", ptolemy.kernel;version="11.0.0", ptolemy.kernel.util;version="11.0.0", ptolemy.util;version="11.0.0" Export-Package: org.kepler.triquetrum.ddpDemo;version="1.0.0"
- Create the p2.inf file (See Creating an actor bundle)
- Open the org.eclipse.triquetrum.workflow.actor.plot project, click on META-INF, highlight the p2.inf file and select copy
- In the org.kepler.triquetrum.ddpDemo project click on META-INF and paste
- The p2.inf file will be created.
What Kepler classes are necessary?
The next step is to determine what Kepler classes are necessary.
The demo primarily uses org.kepler.ddp.actor.ExecutionChoice, which is a composite actor that contains submodels that implement the different ways the actor can be executed.
The -verbose option of the java command will report what classes are loaded.
Unfortunately, it seems that adding -verbose to the run configuration in Eclipse does not report the Kepler classes that are loaded, so we use the command line
To run the model without the UI from the command line
cd kepler/build_area ant run -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml
Edit kepler/build-area/src/org/kepler/build/Run.java and insert the following into the runSuite() method:
java.createJvmarg().setLine("-verbose");
Compile the build system:
ant -f kepler-tasks.xml
Run the model and save the output to a file:
ant run -v -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml >& /tmp/run.txt
To see how many org.kepler classes are loaded (the sed command converts any inner classes that have $ in their name to the parent class, so org.kepler.util.sql.Table$IndexType becomes org.kepler.util.sql.Table)
grep 'Loaded' /tmp/run.txt | grep org.kepler | awk '{print $3}' | sed 's/\$.*\//' | sort | uniq | sed 's@\.@/@g' | awk '{print $1 ".java"}' > /tmp/files
The above returns 188 files. Not all of these classes will be necessary at runtime when running the model, but the important thing here is to make a guess at what classes are necessary.
The script below, when run in the kepler directory, will create a tar file with all the .java files:
#!/bin/sh if [ ! -d files ]; then mkdir files fi files=`cat /tmp/files` for file in $files do srcs=`ls -1d */src` for src in $srcs do if [ -f $src/$file ]; then (cd $src; tar -cf - $file) | (cd files; tar -xvf -) break; fi done done (cd files; tar -cf ../files.tar .) ls -l files.tar
Place the above in a file called kepler/findKeplerFiles.
Run the script with:
cd kepler sh ./findKeplerFiles
Create copies of the files in the org.kepler.triquetrum.ddpDemo module (your pathname will vary)
cat files.tar | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src; tar -xf -)