PTP/designs/3.x/rm proxy

Overview

This is a preliminary design for the PTP Resource Management proxy communication protocol. This protocol is used to communicate between the Resource Manager System in Eclipse, and a lightweight proxy agent running on a target system. The primary purpose of the protocol is for system monitoring, process launch, and process control activities.

Resource Manager System

The Resorce Manager System (RMS) is an Eclipse plugin that manages interaction with arbitrary resource managers. A resource manager in this context, is anything that provides program launch and monitoring services on a target system. Typically, a resource manager will be a job scheduler (e.g. LSF, LoadLeveler, PBS, etc.) running on a large multi-user system. Other types of resource managers include the Open Runtime Environment (ORTE) which is part of the OpenMPI distribution, or the MPICH2 runtime system. The RMS is responsible for populating an internal model in Eclipse which provides a cached representation of the system and program state. Various user interface views are available to inspect and interact with this model. Details of the RMS are provided in a separate document PTP/designs/rms.

Resource Manager Proxy

The RMS communicates with proxy agents to gather information about the state of a target system. The proxy agent may be located on either a local or remote machine.

Proxy Agent Launch

The RMS is responsible for launching the proxy agent. On a local machine, this just involves executing a local process. To launch on a remote machine, the RMS must use an authenticated command service, such as ssh. Current plans are to utilize the Remote System Explorer (RSE) system to provide this remote proxy launch capability.

Proxy Protocol

The communication protocol used between the RMS and the proxy agent is a simple text-based asynchronous command/event protocol. The RMS sends one or more commands to the proxy agent, which in turn will generate events that are returned to the RMS. Some generic properties of the protocol include:

• One command may generate multiple events.
• Commands and events are matched using a transaction ID (tid). The tid in an event will always match a corresponding command.
• Completion of a command is indicated by either an ERROR or OK event with matching tid.
• Any events received after an ERROR or OK event with matching tid can be discarded.

Protocol for commands

1. The RM creates a transaction id for the command.
2. The RM sends the command plus the transaction id to the proxy.
(A version number should be maintained for the protocol.)

Wire protocol for a command

Commands are currently sent as transformed (tohex) ascii strings.
They command message contains: {MESSAGE_LENGTH (8 byte hex integer), TRANSACTION_ID (8 byte integer), COMMAND (4 byte hex integer), COMMAND_ARGS}, where, MESSAGE_LENGTH is length of the message in bytes (excluding the MESSAGE_LENGTH item) and COMMAND_ARGS is a list of command arguments separated by spaces.

Initialize (CMD_INIT)

Command to initialize the proxy. It has one argument, the wire protocol version number. After this command has been received, the proxy is ready to receive and process other commands from the RM. Initialization data may be passed on the command line when the proxy is run. The proxy asynchronously returns an OK/FAIL event completion of the command.

Model definition (CMD_MODEL_DEF)

Command to start the model definition sequence. The proxy responds by a series of ATTR_DEF, LAUNCH_DEF, and ELEMENT_DEF events. Attributes (see below) are meta-data describing data from the proxy that the RM is expected to receive and possibly display in the UI. The sequence is terminated by an OK/FAIL event.

Start events (CMD_START_EVENTS)

Command to start sending events back to the RM. Initially the proxy sends back the full machine state, but sends only state changes as diffs thereafter.

Stop events (CMD_STOP_EVENTS)

Command to halt the event stream to the RM. The proxy responds by stoping the event stream, sending an OK event for the start event transaction, and finally sending an OK event for the stop event transaction.

List event filters (CMD_LIST_FILTERS)

Command to list the set of filters used to limit the event stream. The proxy responds by returning a filter list event and then an OK/FAIL event.

Set event filters (CMD_SET_FILTERS)

Command to set a set of filters used to limit the event stream. The proxy resonds with an OK/FAIL event.

Finish (CMD_QUIT)

Command for the proxy to cleanup and exit. Proxy responds with an OK/FAIL event.

Submit job (CMD_SUBMIT_JOB)

Command to submit a job.

Others

Kill job, ...

Suggestions for resource manager commands (from Dave Wootton)

Additional commands that might be useful include commands to query node availability, query resource pools, query job classes, query job queues, query what's running on the machine, what's running on nodes, change job priority, cancel pending jobs (vs killing a running job), disconnect from and reconnect to proxy. If the proxy is expected to inform the resource manager of the complete state of the machine, queues, etc, at startup, some of these queries might not be required. NOTE that some of these will be including in the event stream.

Proxy Events

The proxy responds to commands from the RM by sending events back to the RM.

Protocol for events

1. The proxy completes (or at least initiates) the command.
2. The proxy sends an event containing the transaction id back to the RM. Any data are returned with the event after the transaction id.

Wire protocol for an event

Events are currently sent as transformed (tohex) ascii strings.
They contain: LENGTH(8 hex digits) TRANSACTION_ID(8 hex digits) CODE(4 hex digits) " " DATA.

FAILURE_EVENT

Sent if a failure occurs while processing a command.

ATTR_DEF_EVENT

A list of attributes that the proxy may send as data.

LAUNCH_CONFIG_EVENT

A list of attributes needed to launch a job, e.g., every queue needs to send a list of attribute ids needed to submitted a job

JOB_SUBMISSION_EVENT

Notification that a job has been submitted for execution. The job id will be returned as part of the event.

Others

Machine change, host change, queue change, job status change, ...

Attributes

Attributes are used so that data sent to the RM is self describing. Attributes are meta-data describing actual data. Attribute ids must be unique and are generated by the proxy. The attribute name must persist across instances of the proxy. An attribute has a:

   ATTR_ID:
   ATTR_NAME:
   ATTR_TYPE:
   ATTR_SNAME:
   ATTR_LNAME:
   ATTR_MIN_VALUE:
   ATTR_MAX_VALUE:
   ATTR_DEF_VALUE:
   ATTR_VALSl: