PTP/designs/3.x/rm proxy

Overview

This is a preliminary design for the PTP Resource Management proxy communication protocol. This protocol is used to communicate between the Resource Manager System in Eclipse, and a lightweight proxy agent running on a target system. The primary purpose of the protocol is for system monitoring, process launch, and process control activities.

Terminiology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119]

Resource Manager System

The Resorce Manager System (RMS) is an Eclipse plugin that manages interaction with arbitrary resource managers. A resource manager in this context, is anything that provides program launch and monitoring services on a target system. Typically, a resource manager will be a job scheduler (e.g. LSF, LoadLeveler, PBS, etc.) running on a large multi-user system. Other types of resource managers include the Open Runtime Environment (ORTE) which is part of the OpenMPI distribution, or the MPICH2 runtime system. The RMS is responsible for populating an internal model in Eclipse which provides a cached representation of the system and program state. Various user interface views are available to inspect and interact with this model. Details of the RMS are provided in a separate document PTP/designs/rms.

Resource Manager Proxy

The RMS communicates with proxy agents to gather information about the state of a target system. The proxy agent may be located on either a local or remote machine.

Proxy Agent Launch

The RMS is responsible for launching the proxy agent. On a local machine, this just involves executing a local process. To launch on a remote machine, the RMS must use an authenticated command service, such as ssh. Current plans are to utilize the Remote System Explorer (RSE) system to provide this remote proxy launch capability.

Proxy Session

One instance of a communcation channel between the RMS and a proxy agent is known as a session. A session only supports communication to a single proxy agent at a time. The mechanism used to effect communcation between the RMS and a proxy agent is not defined in this document, but can be any bi-directional communications channel (e.g. TCP/IP sockets, etc.)

Proxy Protocol

The communication protocol used between the RMS and the proxy agent is a simple text-based asynchronous command/event protocol. The RMS sends one or more commands to the proxy agent, which in turn will generate events that are returned to the RMS.

Some generic properties of the protocol include:

• One command MAY generate multiple events.
• Commands and events are matched using a transaction ID (tid). The tid in an event MUST match a corresponding command.
• Completion of a command is indicated by either an ERROR or OK event with matching tid.
• Tids need only be unique for uncompleted commands. Once a command is completed, it's tid can be reused.
• Any events received with an invalid tid SHOULD be discarded.

Message Format

Commands and events consist of sequences of ASCII characters formatted into a message. A message is transmitted in the following format:

LENGTH COMMAND_OR_EVENT

LENGTH and COMMAND_OR_EVENT are separated by a space (hex 20). The LENGTH is the length of the COMMAND_OR_EVENT portion of the message including the space. COMMAND_OR_EVENT is the actual text of the command or event.

The COMMAND_OR_EVENT portion of the message will contain a sequence of numbers and strings, formatted according to the rules described in the #Commands or #Events sections below.

Numbers are always formatted as fixed length sequences of hexadecimal characters. Strings are formatted as follows:

LENGTH:CHARACTERS

where LENGTH is the number of characters in the string (formatted as 8 hexadecimal characters), ':' is a colon character (hex 3A), and CHARACTERS are the actual ASCII characters in the string.

For example, the string "A String" would be formatted as:

00000008:A String

A zero length string would be formatted as:

00000000:

Protocol Phases

The proxy protocol is divided into a number of phases. A phase determines the legal commands that can be sent to the proxy agent. During a particular phase, illegal commands SHOULD be discarded. Note: this may be changed to SHOULD generate an ERROR event.

Phases follow a strict ordering. Transition from one phase to the next occur when an OK event is received in response to a phase initiation command. A phase initiation command is a command that must be sent to initiate a particular phase. Once a phase has been initiated, any legal commands for that phase may be sent. The phase ordering is defined as follows:

INITIALIZE -> MODEL_DEF -> {START_EVENTS -> STOP_EVENTS}

The phases are defined in more detail in the following sections.

INITIALIZE

This is the first phase, and is used to initiate a communication session between the RMS and proxy agent, and agree on any protocol parameters that apply to this session.

Phase initiation command:

• INIT

Legal commands:

• none

DISCOVERY

The discovery phase is used to allow the proxy agent to inform the RMS of any dynamic property information. This information currently consists of attribute definitions and filter definitions which are described in more detail below.

Phase initiation command:

• MODEL_DEF

Legal commands:

• none

NORMAL

The normal phase is entered once the initialize and discovery phases are completed. This is the normal command/event processing phase.

Phase initiation command:

• START_EVENTS

Legal commands:

• SUBMIT_JOB
• TERMINATE_JOB
• STOP_EVENTS
• QUIT

SUSPENDED

The suspended phase is used when the RMS needs to prevent the proxy agent from sending additional events.

Phase initiation command:

• STOP_EVENTS

Legal commands:

• START_EVENTS
• QUIT

Phase Example

The following provides a simple example of phase transitions. Commands go from left to right, events from right to left. The command tid is shown in ()'s after the command or event name.

-- intialize phase --
INIT(1)         ->
                <- OK(1)
-- definition phase --
MODEL_DEF(2)    ->
                <- ATTR_DEF(2)
                <- ATTR_DEF(2)
                <- OK(2)
-- normal phase --
START_EVENTS(3) ->
                <- NEW_MACHINE(3)
                <- NEW_NODE(3)
STOP_EVENTS(4)  ->
                <- OK(3)
-- suspended phase --
START_EVENTS(5) ->
                <- OK(4)
-- normal phase --
                <- NEW_QUEUE(5)
QUIT(6)         ->
                <- OK(5)
                <- OK(6)

Note that the first suspended phase is not entered until the OK event corresponding to the START_EVENTS command (tid 3) is received. Similarly, the second normal phase is not entered until after the OK event corresponding to the STOP_EVENTS command (tid 4) is received.

Commands

Commands are formatted as simple ASCII text strings. A proxy command consists of a header and a body, separated by a space (hex 20), as follows:

COMMAND_HEADER COMMAND_BODY

The command header consists of three fixed length strings separated by colons (hex 3A), so it is itself fixed length. The format of the header is:

COMMAND_ID : TRANSACTION_ID : NUM_ATTRS

where

COMMAND_ID is a 4 digit number representing the command to be performed

TRANSACTION_ID is the 8 digit transaction ID assigned to this command

NUM_ATTRS is an 8 digit number of space separated elements in the command body

The following sections describe the currently defined commands.

QUIT

Message Format:

0000:TID:00000000

Description 

Terminate the proxy agent. This command will cause the proxy agent to terminate as soon as possible.

Events 

OK

INIT

Message Format:

0001:TID:00000002 VERSION BASE_ID

Description 

Initialize proxy communication. VERSION is the wire protocol version number. BASE_ID" is the base ID used by the proxy agent when allocating new element IDs. After this command has been received, the proxy is ready to receive and process other commands from the RM. Initialization data may be passed on the command line when the proxy is run.

Events 

OK, ERROR

MODEL_DEF

Message Format:

0002:TID:00000000

Description 

Start the proxy discovery phase. The proxy responds with a series of ATTR_DEF and FILTER_DEF events. Attributes (see below) are meta-data describing data from the proxy that the RMS is expected to receive and possibly display in the UI.

Events 

ATTR_DEF, FILTER_DEF, OK, ERROR

START_EVENTS

Message Format:

0003:TID:00000000

Description 

Initiate normal event processing phase.

Events 

CHANGE_JOB, CHANGE_MACHINE, CHANGE_NODE, CHANGE_PROCESS, CHANGE_QUEUE, NEW_JOB, NEW_MACHINE, NEW_NODE, NEW_PROCESS, NEW_QUEUE, REMOVE_JOB, REMOVE_MACHINE, REMOVE_NODE, REMOVE_PROCESS, REMOVE_QUEUE, OK, ERROR

STOP_EVENTS

Message Format:

0004:TID:00000000

Description 

Suspend normal event processing phase.

Events 

OK, ERROR

SUBMIT_JOB

Message Format:

0005:TID:NUM_ATTRS ATTR_1 ATTR_2 ... ATTR_N

Description 

Submit a job to the resource manager for execution.

Events 

OK, ERROR

TERMINATE_JOB

Message Format:

0006:TID:00000001 JOB_ID

Description 

Request the terminaton of an existing job. The meaning of 'termination' depends on the state of the job.

Events 

OK, ERROR

MOVE_JOB

Message Format:

0007:TID:00000002 JOB_ID QUEUE_ID

Description 

Not yet implemented. This command is intended to allow jobs to be moved between queues.

Events 

OK, ERROR

CHANGE_JOB

Message Format:

0008:TID:NUM_ATTRS JOB_ID ATTR_1 ATTR_2 ... ATTR_N - 1

Description 

Not yet implemented. This command is intended to allow a job's status to be changed (e.g. place a hold on a job)

Events 

OK, ERROR

LIST_FILTERS

Message Format:

0009:TID:00000000

Description 

Not yet implemented. This command lists the filters that are currently enabled in the proxy agent.

Events 

OK, ERROR

SET_FILTERS

Message Format:

000A:TID:NUM_ATTRS ATTR_1 ATTR_2 ... ATTR_N

Description 

Not yet implemented. This command sets the filters in the proxy agent.

Events 

OK, ERROR

Proxy Events

The proxy responds to commands from the RM by sending events back to the RM.

Protocol for events

1. The proxy completes (or at least initiates) the command.
2. The proxy sends an event containing the transaction id back to the RM. Any data are returned with the event after the transaction id.

Wire protocol for an event

Events are currently sent as transformed (tohex) ascii strings.
They contain: LENGTH(8 hex digits) TRANSACTION_ID(8 hex digits) CODE(4 hex digits) " " DATA.

FAILURE_EVENT

Sent if a failure occurs while processing a command.

ATTR_DEF_EVENT

A list of attributes that the proxy may send as data.

LAUNCH_CONFIG_EVENT

A list of attributes needed to launch a job, e.g., every queue needs to send a list of attribute ids needed to submitted a job

JOB_SUBMISSION_EVENT

Notification that a job has been submitted for execution. The job id will be returned as part of the event.

Others

Machine change, host change, queue change, job status change, ...

Attributes

Attributes are used so that data sent to the RM is self describing. Attributes are meta-data describing actual data. Attribute ids must be unique and are generated by the proxy. The attribute name must persist across instances of the proxy. An attribute has a:

   ATTR_ID:
   ATTR_NAME:
   ATTR_TYPE:
   ATTR_SNAME:
   ATTR_LNAME:
   ATTR_MIN_VALUE:
   ATTR_MAX_VALUE:
   ATTR_DEF_VALUE:
   ATTR_VALSl: