Revision as of 17:32, 14 April 2006

Evaluation of Related Works for Remote Service Support

Background

The Need for Remote Compile, Launch and Debug

The current implementation of PTP (v1.0) provides basic support for the coding, compiling, launching and debugging of parallel jobs locally (the parallel machine). In a practical configuration, the development of parallel programs often involves three environments:

We need to provide support for such configurations in PTP, which includes remote build, remote launch and foreseenably remote debug.

Configuration Alternatives

Actually, there are however three different possibilities in the actual configuration:

Local Project

In the case that the user desktop is the central location where the whole project is physically located, the following capabilities are required to support remote build, launch and debug:

Remote Build

1. The code must be first transferred to the build machine (possibly through project export).
2. The build command is invoked on the build machine.
3. The build log is transferred back to the local machine and update local models and views (problem view, marker view etc).
4. The binary is transferred back to the local machine.

Remote Launch & Debug

1. The runtime (e.g. ORTE) is transferred to the parallel machine, possibly also includes the debug runtime.
2. The binary is transferred to the parallel machine.
3. The command is invoked on the parallel machine to start the runtime.
4. The launch command is invoked on the parallel machine.
5. The user desktop can then control the execution and debugging of the
6. The stdio and stderr is trapped and send back to the user desktop.
7. Any output is transferred back to the user desktop.

Remote Project

In the case that a remote machine physically holds the project code, the following capabilities are required to support remote build, launch and debug:

Code Edit

1. The user environment access the project through remote file system.

Remote Build

1. The code should be transferred to the build machine (possibly through project export), if the build machine is different from the machine that physically holds the project.
2. The build command is invoked on the build machine.
3. The build log is transferred back to the local machine and update local models and views (problem view, marker view etc).
4. The binary is transferred to the remote machine that holds the project.

Remote Launch & Debug

1. The runtime (e.g. ORTE) is transferred to the parallel machine, possibly also includes the debug runtime.
2. The binary is transferred to the parallel machine from where the project is located, if different from build machine.
3. The command is invoked on the parallel machine to start the runtime.
4. The launch command is invoked on the parallel machine.
5. The user desktop can then control the execution and debugging of the
6. The stdio and stderr is trapped and send back to the user desktop.
7. Any output is transferred back to the machine where the project is located, if different from the build machine.

Hybrid Project

There's only quite minor differences between this case and that of a remote project.

Required Functionalities

Depending on the actual configuration, different functionalities are required:

Local Project

1. File transfer between local machine and remote machine (Export project to remote machine and update project with remote changes)
2. Execute commands on remote machines
3. Access services on the remote machine (proxy via standard TCP/IP)

Remote Project

1. Create project on remote file system
2. Transfer files between remote machines
3. Execute commands on remote machines
4. Access services on the remote machine (proxy via standard TCP/IP)

Hybrid Project

1. Create project folder or files on remote file system
2. Transfer files between remote machines
3. Execute commands on remote machines
4. Access services on the remote machine (proxy via standard TCP/IP)

Related Works

There are some on going works from both the Eclipse platform and the DSDP project that are relevant to us. Almost all of these works are still primitive, and we summarize below both their current status and proposed future development targets.

Alternate File System

The effort to enable alternative file systems, including non local file system, non physical file system etc have been strongly requested by many people from different aspects.

Status

Starting from Eclipse 3.2 M5, alternative file system support has been partially added. There are two extension points involved:

• org.eclipse.core.filesystem (defined in org.eclipse.core.filesystem plugin) - core support for file system
• org.eclipse.ui.filesystemSupport (defined in org.eclipes.ui plugin) - ui support for file system

Existing implementations provided by the Eclipse core for file system include: local, zip. UI supports for local and zip file systems are available. The work on FTP file system seems to be going but the current code in the CVS is broken. There is currently no UI extension in the CVS.

RSE has also provided an extension to the org.eclipse.core.filesystem extension point, which is a wrapper of its remote file subsystem. However, no extension to org.eclipse.ui.filesystemSupport has been provided and thus it is not possible to use it directly.

Projects and Linked Resources can now be created with alternative file systems (tried with zip file system). The 3.2 M5 release notes say that JDT has support non local file systems. However, to which extend such support is provided remains to be investigated.

Future

I can expect that the final release of Eclipse 3.2 can provide better support for alternative file systems in both the platform level and the JDT level. However, other projects might need some time to catch up with such a change. Fundamental efforts might even be necessary to provide full support for some of the projects, like CDT, Photran etc.

Remote Service Explorer

RSE provides UI and API support for browsing and manipulating files on remote systems, execute commands on remote systems, monitoring processes on remote systems and searching

Status

The code currently in the CVS are solely those contributed by IBM. Many of the IDs are still prefixed by "com.ibm.rse". An experiment of the RSE shows that it basically works as it claimed. However, it doesnot meet our needs and possibly also does not meet the need of DSDP project:

1. The datastore feature is not interesting to us, as it requires to start on the target machine a specific datastore service.
2. The support for remote file and remote shell based on SSH is not currently available.
3. The current code in the CVS twicks core model and UI implementation together in org.eclipse.rse.ui, and thus is difficult to read.
4. Although a wrapper is implemented that provides RSE file system, it has not been integrated to the UI and further level of integration to the Eclipse platform is still needed.

Another possibility to improve RSE is to separate its dependency on IDE into another plug-in. This will be helpful to use it within RCP programs.

Future

RSE is now part of the Target Management subproject of DSDP. Because of the similar challenge for both the PTP and the DSDP, I'm expecting that RSE will be adapted and extended to better support the need of target management for both embedded devices and parallel machines. We will have to collaborate with the developers from DSDP to speed up this process.

Roadmap to PTP Remote Service Support

Based on the current status of Alternative File System and RSE, the following development tasks has been identified:

Local Project

1. Extending the RSE to support remote file and remote shell on SSH.
2. Implement UI support for RSE file system. This can be done by extending org.eclipse.ui.filesystemSupport.
3. Extend Eclipse platform to support project export to remote file system.
4. Extend Eclipse platform to support project update from remote file system.
5. Make sure the runtime and debug proxy implementations are appropriate for remote invocation.
6. Modify CDT and Photran project build to support remote build [to be detailed]
7. Modify PTP launch to support remote launch [to be detailed]
8. Modify PTP debug to support remote debug [to be detailed]

Remote & Hybrid Project

1. Extending the RSE to support remote file and remote shell on SSH.
2. Implement UI support for RSE file system. This can be done by extending org.eclipse.ui.filesystemSupport.
3. Make sure the runtime and debug proxy implementations are appropriate for remote invocation.
4. Modify CDT and Photran project build to support remote build [to be detailed]
5. Modify PTP launch to support remote launch [to be detailed]
6. Modify PTP debug to support remote debug [to be detailed]

Summary

RSE can be a potential platform to enable remote service support for PTP. However, huge efforts are still demanded. Collaborative development is also necessary to get the CDT and Photran prepared for remote building and launching.

Open Issues

Resource management is being designed and developed in PTP, how does remote services support fits into the design remains to be discssed.