PTP/photran/rsync remote projects

Overview

One item that has been discussed several times at NCSA/Blue Waters meetings is the desire to have Eclipse support for a new type of C/C++/Fortran project, which we're calling a "rsync-style remote project." In this type of project, the code is stored and edited locally, e.g., on a user's laptop. When it is time to build, however, the code is copied (rsync'd) to a remote machine (e.g., a cluster) and compiled there. The user then runs and debugs the executable on the remote machine (using PTP).

This type of remote support allows the user to work offline until it is time to build/run/debug. It also avoids the need for a separate remote editor and remote indexing, and it does not require any additional components to be installed on the server. However, it complicates the handling of included files, which may exist on the remote machine but not on the local machine.

Matt Fotzler (UIUC) will be prototyping support for rsync-style remote projects in Photran. The final code will be contributed as Bug 313194; this wiki page has been set up to track requirements/issues in the interim.

Implementation

Strawman Prototype

This is an extremely primitive proof-of-concept, intended mainly to facilitate requirements gathering.

Components

• New project wizard: Essentially the same as a C Makefile project, except the builder is changed from make to a custom script, build.sh, which is stored in the project root.  This script rsyncs the project to the remote machine and runs make there.
• Remote include path support: Changes Photran to use URIs, rather than local paths, to specify INCLUDE paths, and allows the user to set such paths by changing the project properties dialog to allow the selection of folders on any filesystem, including RSE filesystems. (Bug 305525)

Technical Issues

• Remote include path support only affects Photran, not CDT.
• The C build console will not allow SSH to ask for a password (gives an error); the user must have password-less SSH set up.
• Make may give a "clock skew detected" error if the local and remote machine do not have their clocks synchronized.
• The prototype builder establishes multiple connections per build, which slows it down noticeably.
• Include files are not cached locally.  Repeatedly loading them from the server is slow; also, one advantage of rsync-style remote projects is supposed to be that the code can be edited locally, without a persistent connection to the server. So a full implementation must perform local include caching.
• There is no notification mechanism for Photran to determine when remote include files are modified; it must poll for changes.
• It may be useful for a full implementation to include an "SSH error parser" so that connection timeouts, etc. can be displayed in the Problems view.
• The prototype only supports remote paths for Fortran INCLUDE lines.  A full implementation must also allow remote module paths and remote C preprocessor include paths.
• Attempting to set a remote breakpoint in PTP is giving the error "Error completing debug job launch: Cannot create routing file: unable to determine process location"
• CDT's scanner discovery would also need to be run on the remote machine (scanner discovery runs GCC -- and probably does some other things -- to determine the compiler's predefined macros and include paths)

Usability Issues

• Windows users do not have make and rsync available by default
• The user may want to exclude some folders (e.g., documentation) from the rsync (for efficiency)
• There should probably be a nicer user interface than forcing the user to manually edit the build script :)

Considerations for Full Implementation

• FYI: Breakdown of functionality provided by the Platform, Remote Tools, and RDT:
  • The Eclipse Platform includes JSch (Java equivalent of SSH); it is used by the CVS client. Both RSE and Remote Tools use it as well.
  • RSE provides various ways to connect to a remote machine, including SSH, dstore, telnet, etc. (N.B. dstore requires a dstore daemon running on the remote machine.)
  • Remote Tools is effectively a cut-down version of RSE that only uses SSH connections (not telnet, etc.). It includes APIs for browsing the filesystem (implemented using sftp), reading/writing remote files, launching remote processes, etc.
  • RDT just provides remote indexing and build.
  • Therefore, rsync-style projects may still benefit from using Remote Tools, even if they do not use RDT.
• PTP's service configuration mechanism could be used to store remote connection information.
• Remote building could use the remote builder from RDT.
• Remote tools could issue the rsync command on resource change or later.