Difference between revisions of "PTP/release notes/2.1"

Revision as of 15:54, 11 March 2009

What's new in PTP 2.1

We are very pleased to announce the 2.1 release of PTP. This release includes a wide variety of improvements to the core platform, as well as adding two significant new areas of functionality. The first of these is the Remote Development Tools (RDT) which enables C and C++ projects to be located on a remote machine, yet still be accessible from within the Eclipse environment. The second is the contribution of the Cell B.E. IDE which enables PTP to be used for the development of applications for the Cell Broadband Engine. We hope that you find the improvements to PTP beneficial to the development of your parallel applications.

PTP Core

• Resource Managers
  • New more flexible resource manager for Open MPI. This resource manager does not require a proxy agent to be running on the target system, and can be use as a template to target other MPI implementations. It also works with Open MPI 1.2 and 1.3.
  • Improvements to the Parallel Environment resource manager, as well as support for the new debugger.
  • 2.1.1 adds MPICH2 resource manager
• Parallel Debugger
  • Removed the dependency on MPI. This allows the debugger to work with virtually any MPI implementation.
  • Flexible architecture allows different routing and communication layers
  • Full asynchronous command support
• Parallel Language Development Tools (PLDT)
  • Unified Parallel C (UPC) support
  • MPI code templates
• External Tools Framework (formerly the Performance Tools Framework)
  • Added support for PAPI Component counter selection
  • Added parametric analysis capabilities (Supported primarily for TAU at this time)
  • Significant modifications to increase work-flow flexibility
• Services Model
  • Allows the user to configure which services in their system are mapped to which providers and locations (e.g. to a remote machine).
  • Allows ISVs and tool implementers to define new services, and contribute providers to any defined services.

Remote Development Tools (RDT)

• • Enables C and C++ projects to be located on a remote machine
  • Provides remote indexing and parsing services
  • New remote C/C++ project wizard
  • Automatic source code delta handling (the index is automatically updated when files in your project are added/removed/changed)
  • Remote "scanner info" support to allow the user to define include paths and defined preprocessor macros as a context for the parser to operate
  • Remote Search, Call Hierarchy, Navigation (e.g. Go To Declaration), Content Assist, Type Hierarchy
  • Remote Standard Make for building remote makefile-based projects

Cell B.E. IDE

• • Custom source code templates for Cell Broadband Engine development
  • Full configurable build properties for PPU and SPU using GNU and XL compilers
  • Managed Build for PPU and SPU using GNU and XL compilers
  • Remote launch and debug of Cell Applications
  • Support for PPU & SPU combined remote debugger
  • Cell performance tools support
  • ALF programming model support
  • IBM PDT (Performance Debug Tools) instrumentation plugins..
  • Mambo Simulator plugins
  • Five pre-configured Cell projects

Prerequisites

Install OpenMPI

If you are using OpenMPI, install it per directions on the OpenMPI website. Note that previous versions of PTP required you to install OpenMPI --with-devel-headers but this is no longer necessary. In fact pre-built binaries available from many sources (and shipped with many systems) should also work. (Your mileage may vary. Please let us know.)

So a basic install for (e.g.) OpenMPI should consist of

• download the openmpi-1.3.tar.gz file
• untar it: tar -xzvf openmpi-1.3.tar.gz and cd to the directory it creates
• ./configure
• sudo make install

Then to check your install:

• ompi_info should return version info including version of 1.3

Build and run a small MPI test case, for example: testMPI.c

• mpicc -o testMPI testMPI.c
• mpirun -np 4 testMPI

Installation

The installation process for PTP depends on a number of factors, including the versions of software you have installed on your system, your operating system, and the types of target systems you want to use. The basic installation steps are:

1. Install Java
2. Install Eclipse
3. Install CDT (C/C++ Development Tools)
4. Install PTP (including RDT)

To complete the installation, you will probably also need to build and install a small set of platform-specific runtime components.

These steps are detailed below:

Install Java

Ensure that a Java Runtime Environment (JRE) version 1.5 or higher is installed. This can be done with the command:

   java -version

In addition an IBM, Sun, or equivalent JRE is required; The gcj-based Java shipped with most Linux distributions does not work with Eclipse. If in doubt, run the above command and check if there is a reference to gcj. If you are using the Linux operating system, download and install either the Sun Java Runtime Environment or the IBM Java Runtime Environment.

Install Eclipse

Install Eclipse 3.4 and CDT 5.0.1 or later, for your architecture. Although PTP can be used in certain limited applications with Eclipse 3.3, only Eclipse 3.4 will be described here.

Note that RDT requires CDT 5.0.2.

You have two choices for Eclipse installation:

Full Eclipse Install 

Install the full Eclipse SDK ("Classic"). Go to Eclipse downloads and download "Eclipse classic" for your platform. This is the full SDK, but will also require the installation of CDT.

Eclipse IDE for C/C++ Development 

This provides the easiest, lightweight, smaller footprint installation, and now includes CDT 5.0.2. If you only plan to do C/C++ development, this is all you need. This is about 1/3 to 1/4 the size of the full SDK, and doesn't include the Java or Eclipse Plug-in development tools. The UI is simpler since you don't have all those additional options. Go to Eclipse downloads and download "Eclipse IDE for C/C++ Developers." Note that this is the bare bones install, does not include the UPC support etc. Read on for adding this support.

Unpack and launch Eclipse

After downloading and extracting Eclipse (zip or tar file), there's nothing else to the installation process from the Eclipse side. In the extracted folder, simply run the eclipse executable file. It will ask for a workspace location; the default is fine.

Next you will install one or more of CDT, PTP, and RDT. Note that you can install them all at once by adding the update sites to the Update manager at the same time if you like. If you add one update site and check the features you want, then add another update site, be sure to check that the features you checked in the first one are still selected.

Install C/C++ Development Toolkit (CDT)

This adds C/C++ tools to your base Eclipse SDK distribution, or if you installed the "Eclipse IDE for C/C++ Developers" above, will give you the opportunity to install additional tools, e.g. UPC tools.

There are three methods for installing CDT, depending if you are installing 5.0.1 (basic or with optional features) or the unreleased 5.0.2 version.

Installing CDT 5.0.2 Basic Features

Installation of CDT 5.0.2 via the Ganymede update site that is already pre-installed into your Eclipse distribution - installs the basic features of CDT, without the option to install optional features.

1. In Eclipse, select Help > Software Updates...
2. At top, make sure the "Available Software" tab is selected.
3. Open the "Ganymede Update Site" item.
4. Select the "C and C++ Development Tools" checkbox.
5. Click "Install..." and follow the instructions.
6. When asked, click "Yes" to restart the Eclipse workbench.

Installing CDT 5.0.2 including Optional Features

If you want to install features other than the basics ones (or even if you don't), use the complete CDT update site as follows:

1. In Eclipse, select Help > Software Updates...
2. At top, make sure the "Available Software" tab is selected.
3. Select the "Manage Sites..." button on the right (If you started by installing the "Eclipse C/C++ IDE" then skip this; the cdt site is already in the list.
4. Check the box next to http://download.eclipse.org/tools/cdt/releases/ganymede and select OK.
5. Expand this item and select the features you want.
   • Select them all if you are unsure. You may want to *not* select "Eclipse CDT Testing Feature" as it adds testing pages to wizards, etc. which can be confusing for the new user.
6. Click "Install..." and follow the instructions.
7. When asked, click "Yes" to restart the Eclipse workbench.

Install Remote System Explorer (RSE)

If you plan to use the Remote Development Tools (RDT), or wish to use PTP with the Remote System Explorer (RSE), it is recommended that you install it manually. Although the update manager should automatically install RSE if required, it is not always successful.

You can do this from Eclipse as follows:

1. Open Help > Software Updates...
2. At top, click on "Available Software" tab.
3. Open the "Ganymede Update Site" item.
4. Open the "Remote Access and Device Development" item.
5. Select "Remote System Explorer End-User Runtime" item.
6. Select "RSE User Actions" item.
7. Click "Install..." and follow the instructions.
8. When asked, click "Yes" to restart the Eclipse workbench.

Install Parallel Tools Platform (PTP)

Installation of PTP is in two parts. The first part comprises installing PTP into your Eclipse distribution on the local machine. The second part comprises installing optional components on the target machine you wish to use with PTP. This target machine could be the same as that running Eclipse, or a remote system.

NOTE: Installing the Remote Development Tools (RDT) is covered in another section below.

Installing PTP

You can do this from Eclipse as follows:

1. Open Help > Software Updates...
2. At top, click on "Available Software" tab.
3. Click the "Add Site..." button then enter http://download.eclipse.org/tools/ptp/releases/2.1 in the text field. Click OK.
4. Expand the site you just added, and select the features you wish to install.
   • Just select them all if you are unsure.
   • Select "Parallel Tools Platform 2.1" if you want PTP.
   • Select "Remote Development Tools" if you want RDT.
5. Click the "Install..." button and follow the instructions.
6. When asked, click "Yes" to restart the Eclipse workbench.

Installing PTP Server Components

PTP comes with optional components that support different resource managers (MPICH2, LoadLeveler and Parallel Environment), as well as a parallel debugger (called the SDM). These components must be installed on the target machine before PTP can be used.

If the target machine and the local machine are the same, skip to Step 2 below.

Step 1: First, you need to copy some files from your Eclipse installation to the target system. In the following steps, YYYYMMDDHHMM corresponds to the numeric build date of the PTP release. For example, 200810311901.

1. Locate the plugins directory contained in the Eclipse installation directory eclipse.
2. Copy the following directories and their contents to a known location on the target machine. The only requirement is that they are all in the same location. YYYYMM
   • org.eclipse.ptp.utils_2.1.0.YYYYMMDDHHMM
   • org.eclipse.ptp.proxy_2.1.0.YYYYMMDDHHMM
   • org.eclipse.ptp.debug.sdm_2.1.0.YYYYMMDDHHMM
3. If you wish to use PTP with IBM LoadLeveler, copy the following directory and its contents:
   • org.eclipse.ptp.rm.ibm.ll.proxy_2.1.0.YYYYMMDDHHMM
4. If you wish to use PTP with IBM Parallel Environment, copy the following directory and its contents:
   • org.eclipse.ptp.rm.ibm.pe.proxy_2.1.0.YYYYMMDDHHMM
5. If you wish to use PTP with MPICH2, copy the following directory and its contents:
   • org.eclipse.ptp.rm.mpich2.proxy_2.1.0.YYYYMMDDHHMM
6. Finally, copy the following directory and contents, replacing <os> and <arch> with the target system's operating system and architecture respectively:
   • org.eclipse.ptp.<os>.<arch>_2.1.0.YYYYMMDDHHMM

Step 2: To complete the installation on the target system, change to the org.eclipse.ptp.<os>.<arch>_2.1.0.YYYYMMDDHHMM directory and run the command:

     sh BUILD

This should build and install the executables. Check the bin directory to see if there are executables in it (e.g. ptp_ibmpe_proxy or sdm). If you see any errors here, please refer to the Trouble Shooting section below.

The server executables can reside in any convenient directory. When you create a resource manager, you will need to navigate to the this directory (the bin directory by default) and select the appropriate resource manager proxy agent (if it requires one). When you configure a debug launch configuration, you will need to select the sdm executable in this directory.

Install Remote Development Tools (RDT)

Installation of RDT is in two parts. The first part comprises installing RDT into your Eclipse distribution on the local machine. The second part comprises installing components on the target machine you wish to use with RDT to host your remote projects.

Installing RDT

If you didn't install RDT when you installed PTP, you can do this any time from Eclipse as follows:

1. Open Help > Software Updates...
2. At top, click on "Available Software" tab.
3. Expand the http://download.eclipse.org/tools/ptp/releases/2.1 site you added previously
   • If you haven't added the update site yet, click the "Add Site..." button and enter http://download.eclipse.org/tools/ptp/releases/2.1 in the text field. Click OK.
4. Select "Remote Development Tools" from the update site.
5. Click the "Install..." button and follow the instructions.
6. When asked, click "Yes" to restart the Eclipse workbench.

Installing RDT Server Components

RDT provides advanced parsing and indexing functionality for C/C++ projects in a remote client-server scenario. The RDT server component, which contains the parser and index database, must be located on the same remote machine as the project files.

The RDT server component provides the following functionality:

• Editing of remote files.
• Creation of an index database from C/C++ source code files located on the remote machine.
• Advanced search and code assist functionality based on the contents of the remote index database.
• Remote C/C++ projects require a connection to be established with a remote RDT server. This guide describes how to install and run the RDT server.

Note: The RDT server component is based on the RSE dstore server. Therefore setting up the RDT server is very similar to setting up the RSE dstore server. The guide to setting up the RSE dstore server can be found here, however this section contains documentation specific to the RDT server.

See Server Installation on Unix, Linux and MacOS X for installing on Unix, Linux or MacOS X systems.

See Server Installation on Windows for installing on Windows systems.

Server Installation on Unix, Linux and MacOS X

The following documentation explains how to install the Linux or UNIX server code, start the server daemon, and make a connection to a remote Linux or UNIX server.

Prerequisites

To use the Remote System Explorer communications server daemon you need to install Perl. Using the daemon helps eliminate some of the manual steps when you connect to the server.

Installing the server code

1. Ensure that Perl is installed.
2. Ensure that a Java Runtime Environment (JRE) version 1.5 or higher is installed. An IBM, Sun or equivalent JRE is required; The gcj-based Java shipped with most Linux distributions does not work. If in doubt, run the command java -version and check if there is a reference to gcj.
3. Choose a directory where you want to install the server code. These instructions will use the /home/user/ directory as an example, but you are free to use any directory. When the server archive file is expanded it will create a directory named rdt-server in the location where it is expanded.
4. Go to the PTP downloads page and download the RDT server package that suits your operating system to this directory.
5. Switch to the /home/user/ directory.
6. Run the following command in the /home/user directory to extract the server code from the package appropriate to your operating system. For example, for Linux this command is:

    tar -xvf rdt-server-linux-1.0.tar

1. A directory named /home/user/rdt-server will be created that contains the server files.

Starting the server

You can start the RSE communications server with the server daemon, or manually. Before starting the server, make sure the Java command is in your path, you can do this by running the following command:

   java -version

You should see something similar to the following:

   java version "1.5.0_16"
   Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
   Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
   

If you receive a "command not found" error, then try creating a symbolic link to the java command in /usr/bin by running the following command:

   ln -s  /opt/IBMJava2-141/jre/bin/java /usr/bin/java

To start the server with the server daemon

Ensure that you are running using the root user ID. (If the daemon is not run under root, it will be unable to authenticate connecting users.) Run the following commands:

   su -l root
   cd /home/user/rdt-server
   perl ./daemon.pl [daemonPort] [serverPortRange]

Note that the server daemon runs on port 4075 by default. You can pass the optional daemonPort argument to force a different port if you want. If your daemon runs behind a firewall, you may want to specify the optional serverPortRange argument to restrict selected server ports to the range given:

   perl ./daemon.pl 4075 10000-10010

To start the server manually

Note: In the following discussion we assume that the RSE server has been installed on Linux. If you are running on a UNIX system the script name is server.sh" rather than server.pl.

If you do not have root access on a remote machine, you can start the server manually for your particular user id only. Run the following commands:

   cd /home/user/rdt-server
   perl ./server.pl [port]

These commands run the server.pl script located in the /opt/rseserver directory. The port parameter to the server.pl script is optional. If you do not specify a port, then the server will pick the first one available and print the port number to standard out. By default, it is usually 4033. If you would like to use a different port, you will then have to enter this port number in port property for the Files subsystem for your connection in the Remote System Explorer (see Connecting to the Remote Server, below). Otherwise, you do not need to change this property.

Note: When you connect RSE to the server, the server will terminate as soon as you disconnect the client. The daemon, however, will not terminate.

Server Installation on Windows

The following documentation explains how to install the Windows server code, start the server daemon, and make a connection to a remote Windows server.

Installing the server code

1. Choose a directory where you want to install the server code. These instructions will use the C:/ directory as an example, but you are free to use any directory. When the server archive file is expanded it will create a directory named rdt-server in the location where it is expanded.
2. Go to the PTP downloads page and download the RDT Windows server package.
3. Copy the file rdt-server-windows-1.0.zip to the <code>C:\ directory (this could be on a different machine).
4. Use an unzip utility to extract the server code.
5. A directory named C:\rdt-server will be created that contains the server files.

Starting the server

You can start the RDT communications server manually, or as a daemon.

To start the server as a daemon

Note that the server daemon does not enforce any user authentication. If you run the server daemon, any user can connect to the machine, work with the file system and run commands. Use of the server daemon on Windows systems is not recommended.

Simply double click the daemon.bat program to start a server daemon. You can edit the daemon.bat file to change properties for the daemon, like a specific daemon port to use or to force a port range for the server (in order to comply with firewalls).

The server daemon runs on port 4075 by default. You can pass the optional daemonPort argument to force a different port if you want. If your daemon runs behind a firewall, you may want to specify the optional serverPortRange argument to restrict selected server ports to the range given:

   daemon.bat 4075 10000-10010

To start the server manually

Simply double click on the server.bat program to start the RDT server. The server will pick the first port available and print the port number. By default, it is usually 4033. You will then have to enter this port number in port property for the Files subsystem for your connection in the Remote System Explorer.

For security reasons, the server will only wait a limited time until a client connects (12000 seconds by default). In order to start the server with an exactly specified port or timeout, open a Windows command prompt and enter:

   c:
   cd \rdt-server
   server.bat [port] [timeout]

When you connect RDT to the server, the server will terminate as soon as you disconnect the client. The daemon, however, will not terminate.

Feature Descriptions

Test your installation

The easiest way to test your installation is to create and run an MPI application.

1. Start Eclipse. (You may need to use the -clean parameter to force it to recognize new features and plug-ins).
2. Switch to the C/C++ perspective (Window > Open Perspective > Other...)
3. Create a small MPI C project.
4. Open the PTP Runtime perspective (Window>Open Perspective>Other...).
5. Create a new resource manager (e.g. Open MPI) and start it. You should see a machine and one or more nodes appear in the 'Machines View'.
6. Create a launch configuration (Run > Run Configurations...) and create a launch configuration for a Parallel Application.
   • Fill in the information for the particular type of resource manager.
   • For example, for an Open MPI application this includes (but may not be limited to):
     • On Resources tab, select Number of processes
     • On Application tab, select your Project for Parallel Project, and the executable for Application Program.
     • On Debugger tab, find the sdm executable that you build from the command line during the installation process as the "Path to debugger executable"
7. Launch the application.

More details on using PTP can be found in Help > Help Contents. Click on "Parallel Tools Platform User Guide".

The PTP help is also available online.

Known Issues

Resource Manager

1. After restarting Eclipse, a resource manager that uses "Remote Tools" is no longer able to connect.

Reproduce By: Create a new resource manager. Select "Remote Tools" as the remote service provider, and create a new location. Set the hostname, username and password. The resource manager should connect successfully. Exit from and then restart Eclipse. In some circumstances the subsequent connection will fail.

Workaround: Remote Tools sometimes forgets the password you supplied. Open the "Remote Environments" view (go to "Window > Show View > Other..." and choose "Remote Environments" from the "Remote Tools" category). In this view, locate the "Target Environment" that was created when you configured the resource manager. Double-click on this and reset the password to the correct value.

2. There are synchronization issues when two (or more) remote resource managers attempt to start simultaneously.

This only happens when PTP is first launched, and attempts to reconnect to resource managers that were running when Eclipse was closed.

Reproduce By: Start two remote resource managers that use remote tools connections. Exit from Eclipse. Restart Eclipse.

Workaround: Shutdown resource managers before exiting Eclipse

3. If two resource managers share the same remote connection, stopping one will also stop the other.

Reproduce By: Create two resource managers that use the same remote connection. Start both resource managers. Stop one resource manager.

Workaround: None. This is expected behavior since shutting down a resource manager also closes the connection.

Debugger

4. Accelerator keys (F5, F6) operate on the current process selected in Debug view.

There is currently no way to use these keys for the Parallel Debug view.

Launch configurations

5. Certain sequences can result in a blank Resources page in the Resources tab.

Reproduce By: Start a resource manager. Open launch configuration and choose resource manager from Main tab. Select Resources tab and fill in some resources. Close launch configuration and shutdown resource manager. Start a different resource manager. Open the same launch configuration and switch to Resources tab, then switch back to main tab. Set the launch config to new the new resource manager, then switch to resources.

Workaround: None.

Remote Development Tools (RDT)

6. The Remote C/C++ Editor is used by default for local files

RDT currently provides a framework for developing C/C++ programs on remote machines. Although it is our intention that RDT work seamlessly with CDT and other Eclipse plug-ins, there is a problem with the Eclipse platform which hinders interoperabilty between base CDT and RDT, namely that the Remote C/C++ Editor provided by RDT is always used by default for editing C/C++ files.

Workaround: Although this issue can be worked around by manually selecting the CDT editor (Context Menu->Open With->C/C++ Editor) when you are working with a local CDT project, it can be tedious having to do this for every single file you open. As such, we recommend that you install RDT into an Eclipse installation which you do not intend to do local C/C++ development with. Future versions of Eclipse and RDT should better support local development.

7. Remote projects are always closed when Eclipse restarts

When you exit and restart Eclipse, all your remote projects are closed. This happens because the .project file is on the remote server and when eclipse starts there is no connection yet, so eclipse thinks the .project file is missing and closes the project.

Workaround: To work around this issue, simply reopen your projects after the connection to the remote machine has been re-established.

8. Remote editors are not properly restored on Eclipse restart

If you leave remote editors open upon closing down the workbench, when you restart, the editors will not be properly initialized, showing a "resource not found" error. Closing and reopening the editor will not alleviate the problem. This problem occurs because on startup of the workbench, the remote connection is not yet established, and hence opening the editors fails.

Workaround: To work around this problem, close all editors opened on remote resources before shutting down the workbench. If the problem occurs, you can alleviate it by closing the affected editors and restarting the workbench.

9. Cannot cancel builds when using the RSE provider

There is an RSE bug which prevents RSE from forcibly terminating remote processes. As a result, if you use RSE to build, you won't be able to cancel a build while it's in progress.

Workaround: Use the Remote Tools provider if possible. Otherwise there is no workaround.

10. Quick Type Hierarchy does nothing

Invoking Quick Type Hierarchy from the editor context menu does not currently do anything.

Workaround: Use "Open Type Hierarchy" instead to open the Type Hierarchy View for an alternate method of viewing type hierarchy information.

11. Creating a remote project that uses Remote Tools as the filesystem does not seem to work.

There is a problem with that causes the ssh connection used by Remote Tools to become disconnected.

Workaround: Use RSE for the filesystem. This will be fixed in the next release of PTP.

Parallel Language Development Tools (PLDT)

12. Analysis of parallel applications located on a remote server doesn't seem to work

Note that not all PLDT features are tested and verified for running with RDT at this time.

Workaround: To use PLDT analysis features, use a local CDT project, instead of a remote project.

Troubleshooting

1. I restarted Eclipse and now my existing resource manager is stuck in the "Starting" (blue) state. If I right click on it, all the options are grayed out (except for "Add Resource Manager...").

Solution: The resource manager is trying to connect to the remote system (see below for possible causes of connection failure). You can stop the resource manager by opening the "Progress" view ("Window > Show View > Other...", then select the "Progress" view from the "General" category) and clicking on the "Cancel Operation" button (red square).

2. I created a resource manager using the "Remote Tools" remote services. It connected successfully, but after restarting Eclipse, it no longer connects.

Solution: See Known Issue #1 above.

See also the older 2.0 Release notes troubleshooting section

FAQ

1. How do I simulate multiple nodes on a single machine with OpenMPI?
   • Edit <openmpi_install>/etc/openmpi-default-hostfile (default is /usr/local/etc/openmpi-default-hostfile) and add lines containing 'node0', 'node1', etc. Edit /etc/hosts and add a localhost entry for each name you added to the openmpi-default-hostfile. So, for 'node0', add '127.0.0.1 node0' to /etc/hosts. This will simulate a multi-node machine.
2. How do I launch eclipse?
   • Normally just run the eclipse executable - however, you may want to do this from a command line instead of from a "shortcut" - some OS's or windowing systems do not properly send the environment information.