PTP/designs/remote/using

Remote-Enabling the Rest of PTP

Notes on enabling plug-ins (include those within PTP) to work on remote resources as well as local resources

First Conference Call - August 24, 2010

Host: Chris Recoskie - Notes: Jeff Overbey

Attendees:

• Chris Recoskie, John Liu, Michael Lindo, Vivian Kong (IBM)
• Beth Tibbitts (IBM)
• Jeff Overbey (Illinois)
• Wyatt Spear (Oregon)

Chris gave us overview/history/motivation of the development of RDT (Remote Development Tools) and we discussed how to enable other parts of PTP to be able to work on remote projects.

History/motivation

1. History/motivation
   1. RDT's current remote model is based on other remote work and requirements developed at IBM
   2. Remote support for CDT
      1. Needed access to system headers, libraries to preprocess code/parse correctly
         1. These headers are often under license agreements that prevent them from being transferred to another machine
         2. Experimented with accessing via SMB or NFS; 4x slower than parsing locally
         3. Parsing/indexing is slow for large projects even when done locally; 4x slowdown was unacceptable
         4. Solution: perform all work involving the index or AST on the remote machine, and return the results in a lightweight form (e.g., as ICElements - see Call Hierarchy example below)
      2. CDT needed better separation between core and UI to support remote indexing
         1. Some things were easy; e.g., there was already an extension point for adding a new content assist provider
         2. Call hierarchy, type hierarchy, search talked directly to the parser and indexer APIs.  These had to be subclassed to override behavior; occasionally code had to be copied and pasted.
      3. Interaction between client and server -- example: Call Hierarchy view
         1. Client wants to find all callers of a particular function
         2. Client UI determines selection region: filename, offset, length; sends this to the server
         3. Server invokes the parser, maps this to an ASTName.
         4. Server queries the index for that ASTName (translates it to a binding, finds references)
         5. Reference bindings are resolved to ICElements
         6. Server returns a list of ICElements
            1. Outline, call hierarchy, etc. are essentially views of the CModel.  The elements of the CModel are ICElements.
            2. The ICElement hierarchy was duplicated to model remote elements
            3. Not all methods work on remote elements since they do not correspond to things on the local filesystem
         7. View is populated from this list of ICElements
      4. Users may have different projects on different machines.
         1. Service provider: e.g., remote builder, remote indexer
            1. Each project has a service configuration, which is a list of the services it's using
         2. Service model: Description of services that are applicable to a given project; they may or may not actually be configured
      5. RDT has separate managed build toolchains definitions for remote toolchains (e.g., Remote XLC)

Converting other tools

1. Converting other PTP tools for remote support
   1. Generalities
      1. C/C++ ASTs obtained locally will probably be incorrect since headers not available; anything that uses C/C++ ASTs should be done remotely
         • However, see PLDT's initial lightweight approach below
      2. Services, subsystems, and miners
         1. RSE Terminology: A miner runs on a remote server; it accepts commands and returns results (e.g., accept search keywords, return search results); in implementation terms, it is a Java class on the server that implements an RSE API
         2. Miners are always tied to a subsystem on the local machine. The subsystem knows how to send a command to a miner.
            
      3. A CDT miner was created; it accepts commands to reindex a project, get content assist for a string, etc.
      4. Other miners (e.g., for PLDT) could talk to the CDT miner
      5. Service providers are not necessarily tied to miners/RSE, but often it's useful to use RSE subsystems/miners to implement a service
      6. Example: Open Declaration
         1. RDT asks for a service provider; this provider provides an Open Declaration API (e.g., which accepts a filename and offset indicating the cursor location and returns the filename, offset, and length of the declaration)
         2. Service provider talks to the remote indexing subsystem on the local machine
         3. Subsystem uses an RSE API to talk to the CDT miner on the remote machine
         4. Miner returns the filename, offset, length of the declaration
   2. Specifics
      1. PLDT - Gets AST, constructs control flow graph, runs analysis
         1. Biggest problem is probably the preprocessor since headers aren't necessarily available; this means the ASTs constructed locally probably aren't complete
         2. Would need to convert analysis to run on the server, then create markers based on its results
         3. This means creating a PLDT miner, subsystem
         • Currently PLDT employs an initial lightweight approach to remote-abling some of the analysis:
           • Use getLocationURI() instead of getLocation() on IResource, and don't use java.io.File objects
           • This makes basic artifact identification work. However caveats are:
             • Work is done on local machine and file is moved to local machine for the analysis
             • Other files must be pulled to the local machine
             • header files won't be the remote ones? will be the possible inaccurate local header files? So includes may provide some inaccuracies
             • There is a current bug? with RemoteTools where translationUnit.getAST() may hang, so sometimes this doesn't work.
      2. ETFw
      3. Photran
         1. Builder
            1. RDT has a remote make builder (i.e., the one for standard make projects) and a generated makefile builder which uses remote toolchains
               1. Remote command launcher is responsible for running the build command on the remote machine
            2. Standard make should be reusable directly from CDT
            3. Create remote GNU toolchain
            4. Scanner discovery caused complications for C/C++; ignoring this issue in Photran right now
         2. Indexer
            1. Need to implement a service provider, subsystem, miner
            2. Greg implemented a service that can deploy a payload
               1. Copies server to the remote system, executes it, and tunnels that traffic through SSH
               2. We would need two implementations of our service: one talks to RSE directly, one knows how to tunnel the traffic over SSH

Next Meeting

Next conference call discussing this ("Remote-enabling 'the rest of' PTP") is scheduled for Tues Sept 7 at 3:00 PM ET; Toll-free USA: 877-421-0030 Passcode: 754803. Contact Beth Tibbitts tibbitts@us.ibm.com if you would like to attend (to be informed of any schedule changes if they happen)

Second Conference Call - Sept 7, 2010

Attendees: Wyatt, Beth, Greg, Chris, Alan

Alan has done some implementation for GEM. Replaced java.io.File with IResource (using EFS). Most other operations are the same, except getLocation(0 needs to be replaced with getLocationURI(). Also using RemoteProcessBuilder.

Remote editing: the Remote Editor is the default if RDT is installed. "Open With" context menu will show which editor it is. There is no visual difference between the Remote and local CDT editors. (Current bug in remote editor breakpoints, Chris needs to sign off on bug fix. Bug fix is in a patch awaiting approval/checkin.)

For an example piece of code, pick an RDT command or feature, such as Open Declaration, and follow it from client to server in the code.

ETFW Feedback code - Wyatt noticed some use of java.io.File. Beth will investigate and fix.

File.separator - EFS doesn't tell what separator character is. URI always uses "/" (forward slash)

ETFw will need to launch remote processes (will need to use IRemoteProcessBuilder). Alan and Wyatt have the same problems so can share experiences in implementing.

Remote Execution (Launch Configurations and Resource Manager code) is separate from Remote Projects (Remote projects are specified in the new project wizard).

Greg can assist with questions about remote processes.

Next meeting Tues Sept 28 3:30 pm ET

Third Conference Call - Sept 28, 2010

How to tell if a resource is remote? From Chris: get project nature. remoteNature will be included. Save nature ID.

• A failed AST fetch (NullPointerException) can be for other reasons besides resource being remote.

Wyatt is fixing ETFw to use EFS (Eclipse File System) and hopes to have chagnes done by SC10 (Nov. 2010). We'll discuss on Oct. PTP monthly call.

Reference

EFS Eclipse File System - which abstracts away the implementation details about where files/resources are located.

Working with remote projects includes Setting up the EFS as the major part. (not using platform-specific APIs when referencing files etc, such as java.io.File. ) These two articles can probably help:

• http://www.eclipsezone.com/eclipse/forums/t83786.html
• http://www.eclipsezone.com/articles/efs/

Practical Experience and Notes

(From discussions on ptp-dev August 2010 including between Alan Humphrey and Greg Watson)

The first step is to convert the plugin to the EFS (Eclipse File System) e.g. replacing all references to java.io.File and using the IResource Interface ( getLocationURI() will become your best friend here ). Then use the abstract interfaces in org.eclipse.ptp.remote.core for remote file manipulation, etc.

There are two main approaches to remote enabling your plugin, depending on what you need to do:

a) If you need to run remote processes, manipulate remote files directly, etc., then you should start by looking at the APIs provided by org.eclipse.ptp.remote.core and org.eclipse.ptp.remote.ui (described below).

b) If you need to access files in a remote project (i.e. one set up using RDT), then you should always use the IResource interfaces for file manipulation and never use java.io.File (you should never use java.io.File inside Eclipse anyway). If you use IResource then you can treat local and remote files in exactly the same manner as the EFS layer will hide everything from you. See the org.eclipse.core.resource documentation for more information.

You may also use a combination of both (a) and (b).

in PTP, the org.eclipse.ptp.remote.core plugin provides a range of abstract interfaces for manipulating remote "things":

• IRemoteConnection - an abstract remote connection. Provides methods to open/close the connection, forward ports (aka tunneling), get the environment, get/set the working directory, etc.
• IRemoteProcess - an abstract remote process created by IRemoteProcessBuilder. Provides methods for getting the stdin, stdout and stderr streams. An IRemoteProcess is created using the IRemoteProcessBuilder interface
• IFileStore - an abstract remote file (provided by EFS). Has methods for file manipulation and obtaining input and output streams. An IFileStore is created using the IRemoteFileManager interface, or can be obtained directly from EFS if you know the URI.

There are also a range of interfaces and classes for UI related activities in org.eclipse.ptp.remote.ui. For example, there's a file browser for accessing files on the remote system.

These APIs work with either Remote Tools (which is a light-weight SSH provider in PTP) or with RSE (heavy weight provider that supports other protocols such as dstore and ftp).