Jump to: navigation, search

Difference between revisions of "PTP/planning/remote"

< PTP‎ | planning
(Evaluation of Related Works for Remote Service Support)
(Meeting Minutes)
 
(32 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= Evaluation of Related Works for Remote Service Support =
+
= Remote Development Tools =
  
Author: Tianchao Li (lit@in.tum.de), Greg Watson (gwatson@lanl.gov)
+
List of Authors:
 +
    Chris Recoskie (recoskie@ca.ibm.com)
 +
    Greg Watson (grw@us.ibm.com)
  
 
== Background ==
 
== Background ==
  
=== The Need for Remote Compile, Launch and Debug ===
+
This page is for organizing efforts surrounding remote development tools for PTP. Initially this will encompass efforts surrounding a remote development solution that utilizes CDT to provide a suite of remote development tools for C/C++ development. Given that the Photran (Fortran) project is built on top of CDT, we foresee that in the future the remote support we build for CDT could form the basis for remote support for Fortran as well.
The current implementation of PTP (v1.0) provides basic support for the coding, compiling, launching and debugging of parallel jobs on a local parallel machine. However, a more typical situation involves one or more of these activities ocurring on a remote machine. In a practical configuration, the development of parallel programs often involves four environments:
+
  
; User desktop : the user environemnt that supports coding and provids UI for compiling, launching and debugging
+
== Interested Parties ==
; File server : the machine on which the project files reside
+
; Build environment : the machine on which the program is to be build, which provides an environment with the same libraries and services available on the parallel machine
+
; Parallel machine : the machine on which the program is executed and debugged
+
  
It is possible that any combination of these machines may be physically remote, however it is normal for the user's desktop to be remote from the other machines. PTP must be able to support this range of configurations if it is to meet the needs of application developers.
+
IBM
 +
* Greg Watson (PTP)
 +
* Chris Recoskie (CDT)
 +
* Jason Montojo (CDT)
  
=== Project Configuration Alternatives ===
+
HP
 +
* Jacob Augustine
  
There are three different possible ways that projects may be hosted in Eclipse:
+
Intel
 +
* Leo Treggiari (CDT)
  
; Local Project : the project and all code of the parallel program is stored in the local file system of the user's desktop.
+
Cril Technology
; Remote Project : the project and all code is physically located on a remote machine  (either a dedicated machine or one of the build machine or parallel machine).
+
* Benoit Souyri
; Hybrid Project : the user desktop physically holds the project (in particular the project configuration information) but some or all of the code is physically located on a remote machine.
+
  
The process of building, launching and debugging applications differs for each combinations of local and remote resources.
+
[[ECF | ECF Project]]
 +
* Scott Lewis
  
==== Local Project ====
+
== Presentations ==
  
In the case where the user's desktop is the central location where the whole project is physically located, the following capabilities are required to support remote build, launch and debug:
+
* [http://download.eclipse.org/tools/cdt/docs/summit2006/Remote_Indexing_&_Searches.ppt Remote Indexing & Searches] Presented at the CDT Summit 2006 detailing the need for remote indexing.
  
'''Remote Build'''
+
* [http://wiki.eclipse.org/images/1/16/Remote_development.pdf Remote Development] Presented at the CDT Summit 2007 detailing the motivation behind the need for remote development tools and the status of current efforts.
  
# The code and build information (e.g. Makefiles) must be first transferred to the build machine (possibly through project export).
+
== Meeting Minutes ==
# The build command is invoked on the build machine.
+
# The build log is transferred back to the local machine and used to update the local models and views (problem view, marker view etc).
+
# The binary is transferred back to the local machine.
+
  
'''Remote Launch & Debug '''
+
Monthly RDT calls are no longer being held.  Instead, we are discussing RDT issues during the regular PTP monthly call.  Archived meeting minutes from RDT calls can be found below.
  
# The control/monitoring/resource management system is invoked on the parallel machine.
+
[[PTP/planning/minutes/January_11_2008 | RDT conference call, January 11, 2008]]
# The binary is transferred to the parallel machine.
+
# Input data files are transferred to the parallel machine.
+
# The launch command is invoked on the parallel machine.
+
# The user desktop can then control the execution and debugging of the program
+
# The stdio and stderr is trapped and send back to the user desktop.
+
# Any output data files are transferred back to the user desktop.
+
  
==== Remote Project ====
+
[[PTP/planning/minutes/March_14_2008 | RDT conference call, March 14, 2008]]
  
In the case that a remote machine physically holds the project code, the following capabilities are required to support remote build, launch and debug:
+
[[PTP/planning/minutes/June_13_2008 | RDT conference call, June 13, 2008]]
  
'''Code Edit'''
+
[[PTP/planning/minutes/August_22_2008 | RDT conference call, August 22, 2008]]
# The user environment accesses the project files through remote file system.
+
  
'''Remote Build'''
+
== Early builds ==
 +
 +
Early builds of RDT will be available on the [[PTP/builds/2.1 | PTP 2.1 builds page]]
  
# The code and build information (e.g. Makefiles) should be transferred to the build machine (possibly through project export), if the build machine is different from the machine that physically holds the project.
+
== Planning ==
# The build command is invoked on the build machine.
+
# The build log is transferred back to the local machine and update local models and views (problem view, marker view etc).
+
# The binary is transferred to the remote machine that holds the project.
+
  
'''Remote Launch & Debug '''
+
Here are are some areas that are currently being worked on and/or investigated.  We welcome contributions in all areas, so if you are interested in a particular area, do not hesitate to offer your input and assistance.
  
# The control/monitoring/resource management system is invoked on the parallel machine.
+
Here is the current state of affairs, as of June, 2008.
# The binary is transferred to the parallel machine, if different from build machine.
+
# Input data files are transferred to the parallel machine, if different from build machine.
+
# The launch command is invoked on the parallel machine.
+
# The user desktop can then control the execution and debugging of the program
+
# The stdio and stderr is trapped and send back to the user desktop.
+
# Any output data files are transferred back to the build machine, if necessary.
+
  
==== Hybrid Project ====
+
* General Remote Services
 +
** PTP has a preliminary implementation for remote services abstraction.  "Remote services" means essentially file operations, as well as launching and control of remote processes.
 +
*** There are several supported implementations of these services, the two most prominent of which are an implementation based on the [http://wiki.eclipse.org/TM_and_RSE_FAQ#What_is_RSE.3F Remote Systems Explorer (RSE)], and a lightweight SSH-only provider that is a part of PTP proper.
 +
** Greg Watson and Chris Recoskie have designed a [http://wiki.eclipse.org/PTP/designs/remote#Service_Model service model] for remote projects.  This model has been implemented in PTP in the org.eclipse.ptp.rdt.services plugin., which allows:
 +
*** The user to configure which services in their system are mapped to which providers and locations.
 +
*** ISVs and tool implementers to define new services, and contribute providers to any defined services.
  
If some project files reside on the user's desktop and some on a remote fileserver, then both sets of files need to be reconciled prior to building and/or launching. Once the files have been reconcilde, the hybrid project can be treated as a remote project.
+
* EFS Support in CDT [[http://wiki.eclipse.org/PTP/designs/remote#Eclipse_File_System_Support Designs and notes]]
 +
** Chris Recoskie (IBM) has made some changes in CDT 5.0 to allow for one to successfully use CDT to create EFS hosted projects and edit the files.  There are some other additional enabling API changes that were made to allow for CModel elements to be tied to EFS URIs.
 +
** Currently creating an EFS project does not magically allow all parts of CDT to work remotely.  Most of CDT is still not EFS aware.
  
=== Required Functionality ===
+
* Remote Indexing for C/C++ (using CDT) [[http://wiki.eclipse.org/PTP/designs/remote#Parsing_and_Indexing overview]]
 +
** Jason Montojo and Chris Recoskie have have implemented an initial set of C/C++ index based services using RSE's dstore protocol as the communication mechanism.  An initial patch has been created and [https://bugs.eclipse.org/bugs/show_bug.cgi?id=236944 submitted] to the PTP Bugzilla and is currently waiting to be committed.  Currently implemented features include:
 +
*** A New Remote C/C++ Project wizard that allows you to create remote projects and configure your service model
 +
*** Integrated index lifecycle management
 +
*** Automatic source code delta handling (the index is automatically updated when files in your project are added/removed/changed)
 +
*** Remote Search
 +
*** Remote Call Hierarchy
 +
*** Remote Navigation (e.g. Go To Declaration)
 +
** Other planned services include:
 +
*** Content Assist
 +
*** Type Hierarchy
 +
*** Include browser
 +
*** Model builder
 +
** Current Work Items
 +
*** Jason is working on content assist.
 +
*** Mike Kucera is working on remote scanner info support.
  
Additional functionality is required to handle each of the project configuration cases:
+
* Remote Standard Make (using CDT)
 
+
** Chris Recoskie has started working on a Remote Make Builder and hopes to have a prototype working sometime in the summer of 2008. This builder will use PTP's Remote Services API in order to launch the builder (make) on the remote system.
==== Local Project ====
+
 
+
# File transfer between local machine and remote machine (replicte project to remote machine and update project with remote changes, using import/export or team synchronization)
+
# Execute commands on remote machines
+
# Access services on the remote machine (proxy via standard TCP/IP)
+
 
+
==== Remote Project ====
+
 
+
# Create project on remote file system
+
# Initiate transfer of files between remote machines from the local machine
+
# Execute commands on remote machines
+
# Access services on the remote machine (proxy via standard TCP/IP)
+
 
+
==== Hybrid Project ====
+
 
+
# Transfer files between local and remote machine and initiate transfer between remote machines
+
# Otherwise, same as remote project
+
 
+
== Related Work ==
+
 
+
There is already some work in both the Eclipse platform and the DSDP project that is relevant to PTP. Almost all of this work is still at a relatively primitive stage however, and we summarize below both their current status and proposed future development targets.
+
 
+
=== Alternate File System ===
+
The ability to support alternative file systems within Eclipse, including non local file systems, has already been requested by a number of people from different areas. Some work has already been undertaken to provide this support.
+
 
+
==== Status ====
+
Starting from Eclipse 3.2 M5, alternative file system support has been partially added. There are two extension points involved:
+
 
+
* org.eclipse.core.filesystem (defined in org.eclipse.core.filesystem plugin) - core support for file system
+
* org.eclipse.ui.ide.filesystemSupport (defined in org.eclipes.ui.ide plugin) - ui support for file system
+
 
+
Existing implementations provided by the Eclipse core for file system include: local, zip, memory. UI supports are also available.
+
The work on FTP file system seems to be going but the current code in the CVS is broken. There is currently no UI extension in the CVS. For more information about the alternate file system, please reference http://www.eclipse.org/eclipse/platform-core/documents/3.2/flexible_workspaces/ and bug item 106176 and 109194.
+
 
+
RSE has also provided an extension to the org.eclipse.core.filesystem extension point, which is a wrapper of its remote file subsystem. However, no extension to org.eclipse.ui.ide.filesystemSupport has been provided and thus it is not possible to use it directly.
+
 
+
Projects and Linked Resources can now be created with alternative file systems (tried with zip file system).
+
The 3.2 M5 release notes say that JDT has support non local file systems. However, no actual support is seen in the distribution.
+
 
+
==== Future ====
+
I expect that the final release of Eclipse 3.2 will provide better support for alternative file systems in both the platform level and the JDT level. However, other projects might need some time to catch up with such a change. Fundamental efforts might even be necessary to provide full support for some of the projects, like CDT, Photran etc.
+
 
+
=== Remote System Explorer ===
+
More specifically, RSE provides UI and API support for browsing and manipulating files on remote systems, execute commands on remote systems, monitoring processes on remote systems and searching
+
 
+
==== Status ====
+
The code currently in the CVS are solely those contributed by IBM. Many of the IDs are still prefixed by "com.ibm.rse". An experiment of the RSE shows that it basically works as it claimed. However, it does not meet our needs and possibly also does not meet the need of DSDP project:
+
 
+
# As the name suggests, the RSE is intended to be an explorer (or shell). The whole infrastructure is designed toward this aim. So, instead of trying to integrate and contribute to the underlying Eclipse platform, the RSE sets up a closed environment that are divided into a core and different subsystems. The design of UI components also towards the same target.
+
# The file subsystem and search subsystem of RSE are actually a duplicate functionality with the emerge of alternate file system support from the Eclipse core. Although a wrapper is implemented that exposes RSE file system, its implementation is not complete. Also, no UI integration has been provided.
+
 
+
Besides, there are also other drawbacks in the current implementation of RSE:
+
# The datastore feature is unlikely to be of interest to us, as it requires a specific datastore service to be started on the remote machine.
+
# The support for remote file and remote shell based on SSH is not currently available.
+
# The current code in the CVS combines the core model and UI implementation together in org.eclipse.rse.ui, and thus is difficult to read and understand.
+
 
+
Another possibility to improve RSE is to separate its dependency on the IDE into another plug-in. This will be helpful to use it within RCP programs.
+
 
+
==== Future ====
+
I'm expecting that the file subsystem and search subsystem of RSE should be faded out. The shell subsystem of RSE for remote execution is something also useful for remote file system support in the Eclipse core. So, I'll expect that part be adapted and finally become part of the Eclipse platform. And, only the UI part remains in RSE.
+
RSE is now part of the Target Management subproject of DSDP. Because of the similar challenge for both the PTP and the DSDP, I'm expecting that RSE will be adapted and extended to better support the need of target management for both embedded devices and parallel machines. We will have to collaborate with the developers from DSDP to speed up this process.
+
 
+
== Roadmap to PTP Remote Service Support ==
+
 
+
Based on the current status of Alternative File System and RSE, the following development tasks have been identified:
+
 
+
# Extend RSE to support remote execution via SSH [on hold while the status of similar projects is determined]
+
# Provide a remote file system implementation for core.filesystem. Initial implementation will use the 9P2000 protocol.
+
# Extend Eclipse platform to support project export to remote file system.
+
# Extend Eclipse platform to support project update from remote file system.
+
# Make sure the runtime and debug proxy implementations are appropriate for remote invocation.
+
# Modify CDT and Photran project build to support remote build [to be detailed]
+
# Modify PTP launch to support remote launch [to be detailed]
+
# Modify PTP debug to support remote debug [to be detailed]
+
 
+
== Summary ==
+
 
+
RSE can be a potential platform to enable remote service support for PTP. However, huge efforts are still demanded.
+
Collaborative development is also necessary to get the CDT and Photran prepared for remote building and launching.
+
 
+
== Open Issues ==
+
 
+
Resource management is being designed and developed in PTP, how does remote services support fits into the design remains to be discssed.
+

Latest revision as of 12:41, 9 October 2008

Remote Development Tools

List of Authors:

   Chris Recoskie (recoskie@ca.ibm.com)
   Greg Watson (grw@us.ibm.com)

Background

This page is for organizing efforts surrounding remote development tools for PTP. Initially this will encompass efforts surrounding a remote development solution that utilizes CDT to provide a suite of remote development tools for C/C++ development. Given that the Photran (Fortran) project is built on top of CDT, we foresee that in the future the remote support we build for CDT could form the basis for remote support for Fortran as well.

Interested Parties

IBM

  • Greg Watson (PTP)
  • Chris Recoskie (CDT)
  • Jason Montojo (CDT)

HP

  • Jacob Augustine

Intel

  • Leo Treggiari (CDT)

Cril Technology

  • Benoit Souyri

ECF Project

  • Scott Lewis

Presentations

  • Remote Development Presented at the CDT Summit 2007 detailing the motivation behind the need for remote development tools and the status of current efforts.

Meeting Minutes

Monthly RDT calls are no longer being held. Instead, we are discussing RDT issues during the regular PTP monthly call. Archived meeting minutes from RDT calls can be found below.

RDT conference call, January 11, 2008

RDT conference call, March 14, 2008

RDT conference call, June 13, 2008

RDT conference call, August 22, 2008

Early builds

Early builds of RDT will be available on the PTP 2.1 builds page

Planning

Here are are some areas that are currently being worked on and/or investigated. We welcome contributions in all areas, so if you are interested in a particular area, do not hesitate to offer your input and assistance.

Here is the current state of affairs, as of June, 2008.

  • General Remote Services
    • PTP has a preliminary implementation for remote services abstraction. "Remote services" means essentially file operations, as well as launching and control of remote processes.
      • There are several supported implementations of these services, the two most prominent of which are an implementation based on the Remote Systems Explorer (RSE), and a lightweight SSH-only provider that is a part of PTP proper.
    • Greg Watson and Chris Recoskie have designed a service model for remote projects. This model has been implemented in PTP in the org.eclipse.ptp.rdt.services plugin., which allows:
      • The user to configure which services in their system are mapped to which providers and locations.
      • ISVs and tool implementers to define new services, and contribute providers to any defined services.
  • EFS Support in CDT [Designs and notes]
    • Chris Recoskie (IBM) has made some changes in CDT 5.0 to allow for one to successfully use CDT to create EFS hosted projects and edit the files. There are some other additional enabling API changes that were made to allow for CModel elements to be tied to EFS URIs.
    • Currently creating an EFS project does not magically allow all parts of CDT to work remotely. Most of CDT is still not EFS aware.
  • Remote Indexing for C/C++ (using CDT) [overview]
    • Jason Montojo and Chris Recoskie have have implemented an initial set of C/C++ index based services using RSE's dstore protocol as the communication mechanism. An initial patch has been created and submitted to the PTP Bugzilla and is currently waiting to be committed. Currently implemented features include:
      • A New Remote C/C++ Project wizard that allows you to create remote projects and configure your service model
      • Integrated index lifecycle management
      • Automatic source code delta handling (the index is automatically updated when files in your project are added/removed/changed)
      • Remote Search
      • Remote Call Hierarchy
      • Remote Navigation (e.g. Go To Declaration)
    • Other planned services include:
      • Content Assist
      • Type Hierarchy
      • Include browser
      • Model builder
    • Current Work Items
      • Jason is working on content assist.
      • Mike Kucera is working on remote scanner info support.
  • Remote Standard Make (using CDT)
    • Chris Recoskie has started working on a Remote Make Builder and hopes to have a prototype working sometime in the summer of 2008. This builder will use PTP's Remote Services API in order to launch the builder (make) on the remote system.