Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/JobManager"

(Common Behaviour of JobManager definition APIs)
(Common Behaviour of JobManager definition APIs)
Line 28: Line 28:
  
 
SMILA provides APIs to read and write the JobManager configuration elements (currently you can write only buckets, workflows and job definitions). The pages linked below describe the specific APIs to do this. However, they have some common properties:
 
SMILA provides APIs to read and write the JobManager configuration elements (currently you can write only buckets, workflows and job definitions). The pages linked below describe the specific APIs to do this. However, they have some common properties:
* Elements can be defined either in the system configuration, or using the APIs. System-defined elements cannot be changed by API calls. Therefore, when reading such system-defined elements using the API, they will contain a <code>readOnly</code> flag set to <code>true</code>. Requests to update these elements will result in an error.  
+
* Elements can be defined either in the system configuration, or using the APIs. System-defined elements cannot be changed by API calls. Therefore, when reading such system-defined elements using the API, they will contain a <code>readOnly</code> flag set to <code>true</code>. Requests to update these elements will result in an error. You cannot set this flag when you create own elements to protect them from being overwritten. The API will remove them.
* User-defined elements, on the other hand, will contain a <code>timestamp</code> attribute describing the it was last changed. This can be used by modelling tools to ensure that they do not overwrite changes made by other users.
+
* User-defined elements, on the other hand, will contain a <code>timestamp</code> attribute describing the it was last changed. This can be used by modelling tools to ensure that they do not overwrite changes made by other users. You cannot set this timestamp yourself in an update request, it will be overwritten by the API.
 
* Additionally, when a update request for an element is performed successfully, the response object will also contain the <code>timestamp</code> attribute generated for this update action.
 
* Additionally, when a update request for an element is performed successfully, the response object will also contain the <code>timestamp</code> attribute generated for this update action.
 
* Apart from the required and optional structure and content of the job manager elements as specified in the pages linked below, each element can contain additional information as needed by the user. This makes it possible to add comments, descriptions, author information, etc. However, the read APIs show this additional information in the result objects only, if invoked with a <code>...?returnDetails=true</code>. Else the results will show only the basic information.
 
* Apart from the required and optional structure and content of the job manager elements as specified in the pages linked below, each element can contain additional information as needed by the user. This makes it possible to add comments, descriptions, author information, etc. However, the read APIs show this additional information in the result objects only, if invoked with a <code>...?returnDetails=true</code>. Else the results will show only the basic information.

Revision as of 06:56, 15 August 2011

Note.png
Available since SMILA 0.9!


Job Manager

This page is work in progress.

The Job Manager controls the processing logic of asynchronous workflows in SMILA by regulating the Task Manager, which in turn generates tasks and decides which task should be processed by which worker and when.

Understanding the Entities of the Job Manager

The Job Manager handles the following entities:

  • Workflows
  • Workers
  • Buckets
  • Data object types
  • Jobs and job runs

The definition of an asynchronous workflow in SMILA consists of one or multiple steps, called actions. Each action specifies the worker that does the actual processing and connects the slots of the worker to actual buckets. A slot is a description of the worker's input and output behavior. There are input slots that describe the type of data objects that the worker is able to process and also output slots that define the type of data objects that the worker will produce. Hence, to be able to use a worker in a workflow, you will have to assign buckets of the correct type. A bucket is simply a logical data container grouping data objects of the same type. In addition to that, parameters may be set at different levels in a workflow to configure its behavior. Global parameters are valid for all actions in a workflow; specific parameters at the action level are only valid for the respective worker.

To run a certain workflow in SMILA, you will have to create a job definition that references the desired workflow. Like in workflows, it is possible to set further parameters, e.g. to adapt the used workflow for a certain application. For example, when using the same workflow in two jobs you could set different values for the store parameter in each to make sure that data is written to different places.

With a job definition alone, the system is not yet doing anything. First, the job must be started to get a so called job run. Job runs in turn consist of one or multiple workflow runs. A workflow run refers to one traversal of the respective workflow, e.g. one traversal for each object in an input bucket.

Job runs provide so called job run data which can be used to monitor the current job processing. Also, job runs can be canceled in case of any problems. Except for the so-called "runOnce" jobs, which are finished automatically, job runs must be stopped manually when they should no longer react on changes.

Common Behaviour of JobManager definition APIs

SMILA provides APIs to read and write the JobManager configuration elements (currently you can write only buckets, workflows and job definitions). The pages linked below describe the specific APIs to do this. However, they have some common properties:

  • Elements can be defined either in the system configuration, or using the APIs. System-defined elements cannot be changed by API calls. Therefore, when reading such system-defined elements using the API, they will contain a readOnly flag set to true. Requests to update these elements will result in an error. You cannot set this flag when you create own elements to protect them from being overwritten. The API will remove them.
  • User-defined elements, on the other hand, will contain a timestamp attribute describing the it was last changed. This can be used by modelling tools to ensure that they do not overwrite changes made by other users. You cannot set this timestamp yourself in an update request, it will be overwritten by the API.
  • Additionally, when a update request for an element is performed successfully, the response object will also contain the timestamp attribute generated for this update action.
  • Apart from the required and optional structure and content of the job manager elements as specified in the pages linked below, each element can contain additional information as needed by the user. This makes it possible to add comments, descriptions, author information, etc. However, the read APIs show this additional information in the result objects only, if invoked with a ...?returnDetails=true. Else the results will show only the basic information.

See the following pages for examples of all this behaviour.

Using the Job Manager

Back to the top