Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

PTP/user-guides/jaxb etfw workflow

Overview

The intent of this user guide is to provide information on writing ETFw XML workflows using Resource Manager (RM) JAXB by going through the various parts of the workflow. This user guide is not yet complete so if you are currently using this page as a reference for building JAXB workflows, take note that some of how this information is organized may change.

Extension Point

The extension point for adding new JAXB ETFw workflows is org.eclipse.ptp.etfw.jaxb.workflows. The following information is required for a new workflow:

  • name - the name of the tool. This is displayed in the tool selection dropdown menu (e.g. TAU, PPW_UPC, etc).
  • file - the path to the XML file containing the ETFw workflow.
<extension point="org.eclipse.ptp.etfw.jaxb.workflows"
  name="TAU"
  file="TAU.xml">
</extension>

Creating an ETFw Workflow

This section explains how to create new ETFw workflows, provide reference information for the XML tags inside an ETFw workflow, and show some snippets of the TAU xml file. Each ETFw workflow begins with the top-level tag <etfwTool>. Within this top-level tag can be one or more Control Data (<controlData>), Build Tool (<buildTool>), Execution Tool (<execTool>) and/or Post Processing Tool (<postProcTool>) tags.

<etfwTool>
  <controlData>
  ...
  </controlData>
  <buildTool>
  ...
  </buildTool>
  <execTool>
  ...
  </execTool>
  <postProcTool>
  ...
  </postProcTool>
</etfwTool>

Control Data

For those that are familiar with creating target configurations, this is probably a familiar term. The control data tag contains all of the attributes that are associated with the workflow. The attributes are used to control UI actions and store information based on user selections. Below is a snippet from the ETFw TAU XML workflow. These attributes were selected because they illustrate how environment variables are stored, how UI options can be translated into program arguments and how a program argument can be paired with another attribute that has the value to pass in. In the next few sub-sections, each set of attributes are explained in more detail with the corresponding XML UI code.

<controlData>
  <!-- This is how an environment variable, TAU_SYNCHRONIZE_CLOCKS is set -->
  <attribute name="TAU Runtime.performance.options.environmentvariables.configuration_id_TAU_SYNCHRONIZE_CLOCKS" type="boolean" translateBooleanAs="1, ">
    <rm:default>1</rm:default>
  </attribute>
  <!-- This is how a program option is set, e.g. -optVerbose, which gets passed as a command line argument -->
  <attribute name="Tau Compiler.performance.options.configuration_id_-OPTVERBOSE_ARGUMENT_DEFAULT" type="boolean" translateBooleanAs="-optVerbose, ">
    <rm:description>Turn on verbose debugging message</rm:description>
    <rm:tooltip>Turn on verbose debugging message</rm:tooltip>
    <rm:default> </rm:default>
  </attribute>
  <!-- These last two attributes contain the program argument, -optPdtDir=, and the value to set it equal to -->
  <attribute name="Tau Compiler.performance.options.configuration_id_-OPTPDTDIR_ARGUMENT" type="boolean" translateBooleanAs="-optPdtDir=, ">
    <rm:description></rm:description>
    <rm:tooltip></rm:tooltip>
    <rm:default> </rm:default>
  </attribute>
  <attribute name="Tau Compiler.performance.options.configuration_id_-OPTPDTDIR_ARGUMENT_SAVED" type="string">
    <rm:default></rm:default>
  </attribute>
</controlData>

Environment Variables Attributes

The primary parts of the attribute for setting environment variables are:

  • name
    • TAU Runtime.performance.options.environmentvariables.configuration_id_ is the key and should match the option panes <configVarId /> tag to which it belongs.
    • TAU_SYNCHRONIZE_CLOCKS is the environment variable to set.
  • type
    • boolean
    • string
  • translateBooleanAs - In many cases, environment attributes are tied to checkbox UI options that returns a boolean that needs to be translated to something else (e.g. 1 or 0, or even a string). In the case of this attribute above, the boolean is translated as 1 for true, empty for false.

Below is a snippet of the xml workflow that would create the checkbox attribute and associate it with a tool pane. The critical parts to understand are the <configVarId> and the <rm:widget> tags. The other XML tags are explained later in this document.

<execTool tool-name="TAU Runtime">
  <global>
    <toolPanes virtual="false" embedded="false">
      <configId>TAU Runtime.performance.options.configuration_id_</configId>
      <configVarId>TAU Runtime.performance.options.environmentvariables.configuration_id_</configVarId>
      <optionPane>
        <rm:title>Tau Runtime</rm:title>
        <rm:layout>
          <rm:grid-layout />
        </rm:layout>
        <rm:composite>
          <rm:layout>
            <rm:grid-layout numColumns="3" makeColumnsEqualWidth="false" />
	  </rm:layout>
          <rm:layout-data>
            <rm:grid-data horizontalAlign="SWT.FILL" verticalAlign="SWT.FILL" grabExcessHorizontal="true" />
          </rm:layout-data>
          <rm:widget type="checkbox" title="Synchronize Clocks" style="SWT.LEFT" attribute="TAU Runtime.performance.options.environmentvariables.configuration_id_TAU_SYNCHRONIZE_CLOCKS">
	    <rm:layout-data>
              <rm:grid-data horizontalAlign="SWT.FILL" grabExcessHorizontal="false" horizontalSpan="3" />
            </rm:layout-data>
  	  </rm:widget>
        </rm:composite>
      </optionPane>
    </toolPanes>
  </global>
</execTool>

Program Argument Attributes

This section discusses how UI options get translated into program arguments using the above control data as an example. In the Control Data, the option -optVerbose should get passed as a program argument based on a boolean translation (similar to the environment variable). As with the environment variable, the main parts of the attribute are:

  • name
    • Tau Compiler.performance.options.configuration_id_ is the key and should match the option panes <configId /> tag to which it belongs. This lets the system know that it is a program argument for this tools option pane and the value should be passed to the tool.
    • -OPTVERBOSE_ARGUMENT_DEFAULT - Unlike the environment variable, this string can be anything since it doesn't have any significance.
  • type
    • boolean
    • string
  • translateBooleanAs - Similar to environment variables, attributes containing program arguments are primarily tied to checkbox UI widgets that returns a boolean that needs to be translated to something else (e.g. a string value). In the case of this attribute above, the boolean is translated as -optVerbose for true, empty for false.

Below is a snippet of the xml workflow that would create the checkbox widget for this program argument and associate it with the specified tool pane. The critical parts to understand are the <configId> and the <rm:widget> tags. The other XML tags are explained later in this document.

<toolPanes virtual="false" embedded="false" pane-name="Tau Compiler" prepend-with="-tau_options=" enclose-with="'" separate-with=" ">
  <configId>Tau Compiler.performance.options.configuration_id_</configId>
  <configVarId>Tau Compiler.performance.options.environmentvariables.configuration_id_</configVarId>
  <optionPane>
    <rm:title>Tau Compiler</rm:title>
    <rm:layout>
      <rm:grid-layout />
    </rm:layout>
    <rm:composite>
      <rm:layout>
        <rm:grid-layout numColumns="3" makeColumnsEqualWidth="false" />
      </rm:layout>
      <rm:layout-data>
        <rm:grid-data horizontalAlign="SWT.FILL" verticalAlign="SWT.FILL" grabExcessHorizontal="true" />
      </rm:layout-data>
      <rm:widget type="checkbox" title="Verbose" style="SWT.LEFT" attribute="Tau Compiler.performance.options.configuration_id_-OPTVERBOSE_ARGUMENT_DEFAULT">
        <rm:layout-data>
          <rm:grid-data horizontalAlign="SWT.FILL" grabExcessHorizontal="false" horizontalSpan="3" />
        </rm:layout-data>
      </rm:widget>
    </rm:composite>
  </optionPane>
</toolPanes>

Paired Program Argument Attributes

This section discusses the other type of program arguments, paired program arguments. That is, when you have something similar to the previous discussion about program arguments, but you want to pass a value for that argument. For example, looking at the control data, we want to pass a value for the program argument -optPdtDir= where one UI widget contains the program argument and another contains the value to pass. The last two attributes in the control data above represent paired program arguments and the details about how they are paired are highlighted below.

First attribute:

  • name
    • Tau Compiler.performance.options.configuration_id_-OPTPDTDIR_ARGUMENT

Second attribute:

  • name
    • Tau Compiler.performance.options.configuration_id_-OPTPDTDIR_ARGUMENT_SAVED

As with normal program arguments, the first part of the name must match the value found in the tool pane's <configId /> field so that the system knows that the program argument is for the tool that the pane is associated with. The difference is that the second key that has the value must end with _SAVED. When parsing program arguments, the system looks at the launch configuration to see if there is another attribute that matches the original key value, but ends with _SAVED. This way it knows that -optPdtDir= should be paired with whatever value is in the second attribute. Below is a snippet of the xml workflow that would create the checkbox widget for this program argument and a browse widget that would allow the user to either browse for the location of PDT or enter it. The critical parts to understand are the <configId> and the pair of <rm:widget> and <rm:browse> tags. The first is for the checkbox and the other for a special widget for the browse button / Text field. The other XML tags are explained later in this document.

<rm:widget type="checkbox" title="PdtDir" buttonId="PdtDir" style="SWT.LEFT" attribute="Tau Compiler.performance.options.configuration_id_-OPTPDTDIR_ARGUMENT">
  <rm:layout-data>
    <rm:grid-data horizontalAlign="SWT.FILL" grabExcessHorizontal="false" horizontalSpan="1" />
  </rm:layout-data>
</rm:widget>
 
<rm:browse title="Browse..." localOnly="true" directory="true" textStyle="SWT.BORDER" attribute="Tau Compiler.performance.options.configuration_id_-OPTPDTDIR_ARGUMENT_SAVED">
  <rm:text-layout-data>
    <rm:grid-data horizontalSpan="1" horizontalAlign="SWT.FILL" verticalAlign="SWT.FILL" grabExcessHorizontal="true" />
  </rm:text-layout-data>
  <rm:button-layout-data>
    <rm:grid-data horizontalSpan="1" horizontalAlign="SWT.FILL" verticalAlign="SWT.FILL" grabExcessHorizontal="false" />
  </rm:button-layout-data>
  <rm:text-control-state>
    <rm:enable-if button="PdtDir" selected="true" />
  </rm:text-control-state>
  <rm:button-control-state>
    <rm:enable-if button="PdtDir" selected="true" />
  </rm:button-control-state>
</rm:browse>

ETFw Tag Reference

  • <etfwTool> - Top level tag for enclosing an ETFw workflow.
  • <controlData> - Contains all of the attributes associated with the workflow. These are tied to UI widgets to save information based on user selections.
    • <attribute name="" type="" translateBooleanAs="">
  • <buildTool> - Represents a compilation step in the workflow.
    • <global> - Some environment variables do not logically fit with the invocation of a particular tool. In that case, the environment variables and the UI for defining which variables to set, can be placed in the global construct rather than a particular tools panel. Currently, the global tag is only used by the <execTool> tag.
      • <toolPanes virtual="" embedded="" pane-name="" prepend-with="" enclose-with="" separate-with="">
        • <optionPane> - this is where the User Interface is defined by adding widgets. It represents a UI tab from the RM JAXB schema.
    • <cc-compiler>
    • <cxx-compiler>
    • <f90-compiler>
    • <all-compilers>
      • <toolPanes virtual="" embedded="" pane-name="" prepend-with="" enclose-with="" separate-with="">
        • <optionPane> - this is where the User Interface is defined by adding widgets. It represents a UI tab from the RM JAXB schema.
  • <execTool> - Represents a execution step in the workflow.
    • <global> - Some environment variables do not logically fit with the invocation of a particular tool. In that case, the environment variables and the UI for defining which variables to set, can be placed in the global construct rather than a particular tools panel. Currently, the global tag is only used by the <execTool> tag.
      • <toolPanes virtual="" embedded="" pane-name="" prepend-with="" enclose-with="" separate-with="">
        • <configId> - unique string identifying the tool pane that all program argument attributes should start with so they can be associated with the correct tool. See example 1.
        • <configVarId> - unique string identifying the tool pane so that all environment variable attributes can be associated with the correct tool.
        • <optionPane> - this is where the User Interface is defined by adding widgets. It represents a UI tab from the RM JAXB schema.
    • <execUtils>
      • <toolPanes virtual="" embedded="" pane-name="" prepend-with="" enclose-with="" separate-with="">
  • <postProcTool> - Represents a post-processing step in the workflow.
    • <global> - Some environment variables do not logically fit with the invocation of a particular tool. In that case, the environment variables and the UI for defining which variables to set, can be placed in the global construct rather than a particular tools panel. Currently, the global tag is only used by the <execTool> tag.
    • <analysisCommands>
      • <toolPanes virtual="" embedded="" pane-name="" prepend-with="" enclose-with="" separate-with="">

Appendix 1: Converting ETFw to Resource Manager (RM) JAXB

Example 1 - PerfSuite

This section will provide detailed steps for creating an ETFw xml workflow for the PerfSuite profiling tool. This example assumes that PerfSuite is compiled and running on your machine with PAPI support. Most of PerfSuite's options can either be passed to the psrun executable at the command line or set as environment variables. For this example, most will be passed at the command line and one will be set as an environment variable to illustrate how the ETFw JAXB system works.

The profiling tool from PerfSuite we want to run is psrun. The configurable commands are listed below under three categories, environment variables, program arguments and paired program arguments:

  • Environment Variables
    • File Format - output the results in xml or plain text.
  • Program Arguments
    • MD5 checksum - include MD5 checksums in output
  • Paired Program Arguments
    • Configuration - specify the configuration file to use.
    • Domain - select the counting domain (user, kernel, all)
    • POSIX - enable POSIX thread support
    • Resource Usage - include resource usage statistics
    • Resource Usage Only - include only resource usage statistics
    • Write Signal Numbers - Specify signal numbers to write before process exits.
    • Overflow Threshold - specify overflow threshold to use for profiling

The environment variables will be set in the environment prior to launch, program arguments and paired program arguments will get passed to the psrun executable at the command line. For example, to specify include md5 checksums and set the domain to kernel would be:

psrun --md5 -d kernel shallow

where shallow is the name of the program executable to profile. Now that we know the inputs we need to tie to the User Interface (UI), we can start building the XML workflow that will tie the UI selections to program arguments or environment variables.

Since PerfSuite is ran with the program executable as input, we don't need to concern ourselves with the buildTool and postPorcTool tags. We only need the execTool tag. The basic workflow would be something similar to the one below:

<etfwTool xmlns="http://org.eclipse.ptp/etfw" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:rm="org.eclipse.ptp"
	xsi:schemaLocation="etfw_tool_type.xsd" name="perfsuite" prepend-execution="false" recompile="false" explicit-execution="false">
 
  <!-- attributes will be stored in the control data --->
  <controlData>
 
  </controlData>
 
  <!-- This is the tool to execute -->
  <execTool tool-name="PerfSuite">
 
  </execTool>
 
</etfwTool>

For simplicity, we will just add the necessary pieces to perform the previous example (e.g. psrun --md5 -d kernel shallow) and specify that we want to have the output in xml (by specifying the environment variable PS_HWPC_FORMAT.

<etfwTool xmlns="http://org.eclipse.ptp/etfw" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:rm="org.eclipse.ptp"
	xsi:schemaLocation="etfw_tool_type.xsd" name="perfsuite" prepend-execution="false" recompile="false" explicit-execution="false">
 
  <!-- attributes will be stored in the control data --->
  <controlData>
    <attribute name="PERFSUITE_psrun.environmentvariables.configuration_id_PS_HWPC_FORMAT" type="choice">
      <rm:description>Specify the output format (xml or plain text)</rm:description>
      <rm:choice>xml,text</rm:choice>
      <rm:default>xml</rm:default>
    </attribute>
    <attribute name="PERFSUITE_psrun.configuration_id_-MD5" type="boolean" translateBooleanAs="--md5, ">
      <rm:description>Include MD5 checksums in output</rm:description>
      <rm:default> </rm:default>
    </attribute>
    <attribute name="PERFSUITE_psrun.configuration_id_-DOMAIN" type="boolean" translateBooleanAs="-d, ">
      <rm:description>Specify domain</rm:description>
      <rm:default> </rm:default>
    </attribute>
    <attribute name="PERFSUITE_psrun.configuration_id_-DOMAIN_SAVED" type="choice">
      <rm:description>Specified domain</rm:description>
      <rm:choice>user,kernel,all</rm:choice>
      <rm:default>user</rm:default>
    </attribute>
  </controlData>
 
  <execTool tool-name="PerfSuite">
    <!-- This is the set of tools to execute, in this case just one tool is defined -->
    <execUtils tool-command="psrun">
      <!-- A single tool could have multiple panes, in this case there is just one -->
      <toolPanes virtual="false" embedded="false">
        <configId>PERFSUITE_psrun.configuration_id_</configId>
        <configVarId>PERFSUITE_psrun.environmentvariables.configuration_id_</configVarId>
 
        <!-- This is the actual UI tab that will be displayed with the defined UI widgets -->
        <optionPane>
          <rm:title>PerfSuite</rm:title>
          <rm:layout>
            <rm:grid-layout />
          </rm:layout>
          <rm:composite>
            <rm:layout>
              <rm:grid-layout numColumns="3" makeColumnsEqualWidth="false" />
            </rm:layout>
            <rm:layout-data>
              <rm:grid-data horizontalAlign="SWT.FILL" verticalAlign="SWT.FILL" grabExcessHorizontal="true" />
            </rm:layout-data>
            <rm:widget type="combo" title="File Format" readOnly="true" style="SWT.LEFT"
              attribute="PERFSUITE_psrun.environmentvariables.configuration_id_PS_HWPC_FORMAT">
              <rm:layout-data>
                <rm:grid-data widthHint="150" horizontalAlign="SWT.FILL" grabExcessHorizontal="false" horizontalSpan="2" />
              </rm:layout-data>
              <rm:tooltip>Output format, either xml or text</rm:tooltip>
            </rm:widget>
            <rm:widget type="checkbox" title="Domain" buttonId="domain-button" style="SWT.LEFT" attribute="PERFSUITE_psrun.configuration_id_-DOMAIN">
              <rm:layout-data>
                <rm:grid-data widthHint="150" horizontalAlign="SWT.FILL" grabExcessHorizontal="false" horizontalSpan="1" />
              </rm:layout-data>
              <rm:tooltip>Select counting domain (user, kernel, all)</rm:tooltip>
            </rm:widget>
            <rm:widget type="combo" readOnly="true" style="SWT.LEFT" attribute="PERFSUITE_psrun.configuration_id_-DOMAIN_SAVED">
              <rm:layout-data>
                <rm:grid-data widthHint="150" horizontalAlign="SWT.FILL" grabExcessHorizontal="true" horizontalSpan="2" />
              </rm:layout-data>
              <rm:control-state>
                <rm:enable-if button="domain-button" selected="true" />
              </rm:control-state>
            </rm:widget>
            <rm:widget type="checkbox" title="MD5 checksum" style="SWT.LEFT" attribute="PERFSUITE_psrun.configuration_id_-MD5">
              <rm:layout-data>
                <rm:grid-data widthHint="150" horizontalAlign="SWT.FILL" grabExcessHorizontal="true" horizontalSpan="3" />
              </rm:layout-data>
              <rm:tooltip>Include MD5 checksums in output</rm:tooltip>
            </rm:widget>
          </rm:composite>
        </optionPane>
      </toolPanes>
    </execUtils>
  </execTool>
</etfwTool>

Using the above XML workflow as a reference, let's break down where the pieces of the command psrun --md5 -d kernel shallow are in the workflow. If you find the execUtils tag, you will see that it contains the tool-command we want to run, 'psrun. The rest of what is contained between the tags allows the user to select the additional input flags. For example, looking at the checkbox widget with title MD5 checksum, you will see that it is tied to the attribute PERFSUITE_psrun.configuration_id_-MD5. If you find that attribute in the controlData section, it translates to --md5 when it is selected and the empty string when it is not selected. In addition, notice that the first part of the attribute, PERFSUITE_psrun.configuration_id_ corresponds to the configId for that tool's pane. This is how the system understands which attributes belong to which tools so the correct tool gets the --md5 input flag.

The rest of the inputs, including the environment variable for specifying text or xml output can be deduced by reviewing the sections above about setting environment variables and pairing attributes and then locating the UI widgets in the XML workflow.

TBD

References

Back to the top