Adding Remote Build and Remote File System Support to CDT
List of Authors:
Greg Watson (firstname.lastname@example.org) Tianchao Li (email@example.com) (future authors)
Our objective is to provide remote support for the Parallel Tools Project in a way that is natural to the user. The approach we take is to extend the new emerging Eclipse File System (EFS) with support for remote file systems, and use RSE to run commands on the remote machine (see the evaluation). Most parallel application development uses either the CDT and Photran components, however neither of these components currently support EFS or remote building.
The following diagram shows a typical scenario in which a separate file server, build server and compute server are employed. Projects are physically located on the file server. Using EFS and a remote filesystem protocol, project files appear local to Eclipse users. The build server mounts the project from the file server using a standard remote filesystem protocol. When Eclipse needs to index or build the project, it uses RSE to launch a remote command onto the build server. The result of the indexing or building is saved in the project directory and so is then accessible to Eclipse to update it's project view. Once the project has been built it can be launched or debugged on the compute server using RSE to send remote commands. The compute server also mounts the project directory from the file server, so no executable files actually need to be transmitted. Other scenarios in which the build, file and compute servers are combined are also possible.
The Current Status
CDT has no support for EFS at all. Even support for locally linked resources, as a special example of EFS, is quite limited. For example, the standard make C project does not support linked files. The following bugs are also identified:
- 136136 Build directory should not be restricted to workspace
- 70491 Invalid Include Path when "Link to Folder in the Filesystem"
Also see http://dev.eclipse.org/mhonarc/lists/cdt-dev/msg03059.html for a discussion of linked resources.
CDT also assumes that the source files are available locally and uses this fact when running the indexer. The build command is also assumed to run locally and access a local filesystem.
What is Needed?
The above process abstracts and covers many different cases. Some typical scenarios include, but not limited to:
- whole project on remote server: if the build machine is the same as the file server, then no replication is necessary; otherwise replicate.
- project with locally linked directories and files (linked to local file system): the files need to be replicated to a local build directory before invoking the build command.
- project with virtual file system (zip, for example): the files has to be replicated to a local build directory.
This will affect several other related components of CDT. Such components include the Indexer, Managed Make etc.
Remote Filesystem Support for EFS
With EFS, projects or its component directories and files can be linked to different file systems, which can be local, remote, or virtual. The files and directories can be physically located at completely diverse places. However, there is currently no support for true remote filesystem protocols (e.g. NFS or V9FS). The only implementation provided is for FTP. Support for these remote filesystem protocols needs to be added, along with UI support for specifying remote hostname, protocol and mount directories.
Adding EFS Support to CDT
The CDT project implementation currently makes the assumption that the project is locally resident on the machine running Eclipse. Changes will need to be made to the way CDT accesses and uses files so that projects hosted on remote machines using EFS are possible.
Adding Remote Indexer and Build Support to CDT
CDT currently assumes that the local machine contains the necessary envrionment to build the project, and that the project source files and this environment will be available to external build commands such as 'make'. However, employing a build server breaks this assumption because the local machine almost certainly has a different environment from the build server. In some cases, this may extended to architectural or operating system differences.
To ensure that the Eclipse user sees a development environment that is as close as possible to that on which the project will be built, activities such as indexing, which make reference to the environment outside the project, need to be run on the build server. The build command also needs to be run on this machine.
We propose to extend the existing Build Directory property of the C/C++ Make Project preference page for this purpose. This extension will add an additional property to specify the Build Server (see the picture). This server will be used to run the indexer and build commands. The Build Directory will specify the directory path of the project relative to this server.
Supporting Managed Make Projects
CDT managed make has many of the same requirements as standard make projects for the purposes of remote building. However, there are two additional requirements. The first is that the 'makefiles' are generated automatically by the system, and these files reside in the project directory. In the case where the project directory resides on a remote server, Eclipse has access to this directory using EFS, so as long as the managed make system uses EFS it should work as normal. The second requirement is the calculation of dependency information that is used to generate the 'makefiles'. This dependency information must reflect the environment on the build server, not the local machine, so the dependency calculation must take place remotely. It is envisioned that this will use the same mechanism as will be used for remote indexing.
Roadmap to EFS and Remote Build for CDT
- Modify the code to allow build directory outside of workspace (fix to bug 136136)
- Modify CDT Project Creation/Conversion Wizard for EFS support
- Add build server property to the build property tab and modify the code to launch build command on build server
- Modify the affected components like Indexer and ManagedMake and fix those components [to be investigated]
- Enable file replication before build to support other scenarios