Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/JobManagerConfiguration"

(→‎Configuring the Job Manager)
(→‎Predefined JobManager entities)
 
(12 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
{{note| Available since SMILA 0.9!}}  
 
{{note| Available since SMILA 0.9!}}  
  
−
== Configuring the Job Manager ==
+
== Configuring the JobManager ==
  
−
The Job Manager comes with a simple configuration file located in <tt>SMILA/configuration/org.eclipse.smila.jobmanager/jobmanager.properties</tt>. By default it looks like this:
+
The JobManager is configured via the [[SMILA/Documentation/Bundle_org.eclipse.smila.clusterconfig.simple|ClusterConfig service]]. With the "simple" ClusterConfig service, it uses one properties in the "taskmanager" section of <code>clusterconfig.json</code>:
  
−
<pre>
+
<source lang="javascript">
−
jobmanager.task.max.retries.recoverable.error=10
+
{
−
</pre>
+
  ...
 +
  "taskmanager": {
 +
    ...
 +
    "maxRetries": 10,
 +
    ...
 +
  },
 +
  ...
 +
}
 +
</source>
  
−
;jobmanager.task.max.retries.recoverable.error: Defines the maximum number of retries for tasks finished with a RECOVERABLE_ERROR result, either explicity sent by the worker or by the task monitoring of the Task Manager due to the worker not having sent keepAlive signals anymore for a configured period of time. As long as this retry limit is not reached, the Job Manager will recreate the task with a new task ID but the same settings, so that it can be processed by another worker. If the retry count is reached, however, the RECOVERABLE_ERROR will be handled as a FATAL_ERROR. Default: 10.
+
* maxRetries: Used to decide how often a task should be retried that has failed with an RECOVERABLE_ERROR, either because the "timeToLive" was exceeded or the worker itself reported such an error. If the retry limit is reached, the task will finally fail with a FATAL_ERROR.
 +
 
 +
See [[SMILA/Documentation/TaskManager#Configuration]] for more details.
 +
 
 +
== Predefined JobManager entities ==
 +
 
 +
The configuration directory <tt>org.eclipse.smila.jobmanager</tt> can contain 5 files:
 +
 
 +
* <tt>dataObjectTypes.json</tt>: DataObject types for buckets
 +
* <tt>workers.json</tt>: Description of available workers
 +
* <tt>buckets.json</tt> Predefined persistent buckets.
 +
* <tt>workflows.json</tt>: Predefined workflow definitions.
 +
* <tt>jobs.json</tt>: Predefined job definitions.
 +
 
 +
These files contain JobManager entity definitions that are available immediately after system startup. We call them "predefined" elements. DataObject types and workers can currently be only predefined, for the other entities additional definitions can be added and updated via the ReST API.
 +
 
 +
The structure of these files is similar, e.g. the standard <tt>dataObjectTypes.json</tt> looks like this:
 +
 
 +
<source lang="javascript">
 +
{
 +
  "dataObjectTypes":
 +
  [
 +
      {
 +
      "name": "recordBulks",  
 +
        "persistent": {
 +
          "store": "${store}",
 +
          "object": "${_bucketName}/${_uuid}"
 +
        },
 +
        "transient": {
 +
          "store": "${tempStore}",
 +
          "object": "${_bucketName}/${_uuid}"
 +
        }
 +
      }
 +
  ]
 +
}
 +
</source>
 +
 
 +
Each file contains a single JSON object with a single key identical to the filename without suffix ("dataObjectTypes", "workers", "buckets", "workflows", "jobs"). The value for this key is a sequence of appropriate entity definitions, see the respective page for details:
 +
* [[SMILA/Documentation/DataObjectTypesAndBuckets]]
 +
* [[SMILA/Documentation/WorkerAndWorkflows]]
 +
* [[SMILA/Documentation/JobDefinitions]]
 +
 
 +
Predefined entities cannot be changed or deleted using the ReST API, but only by changing the configuration files and restarting SMILA. Therefore they are marked with a <tt>"readOnly": true</tt> property when retrieved from the JobManager. You can of course make copies of them with different names to create variants.
 +
 
 +
If an entity cannot be parsed or is not valid (e.g. a job definition using an undefined workflow) you will find error messages in the SMILA log file.

Latest revision as of 08:46, 11 November 2011

Note.png
Available since SMILA 0.9!


Configuring the JobManager

The JobManager is configured via the ClusterConfig service. With the "simple" ClusterConfig service, it uses one properties in the "taskmanager" section of clusterconfig.json: 

{
  ...
  "taskmanager": {
    ...
    "maxRetries": 10,
    ...
  },
  ...
}
  • maxRetries: Used to decide how often a task should be retried that has failed with an RECOVERABLE_ERROR, either because the "timeToLive" was exceeded or the worker itself reported such an error. If the retry limit is reached, the task will finally fail with a FATAL_ERROR.

See SMILA/Documentation/TaskManager#Configuration for more details.

Predefined JobManager entities

The configuration directory org.eclipse.smila.jobmanager can contain 5 files:

  • dataObjectTypes.json: DataObject types for buckets
  • workers.json: Description of available workers
  • buckets.json Predefined persistent buckets.
  • workflows.json: Predefined workflow definitions.
  • jobs.json: Predefined job definitions.

These files contain JobManager entity definitions that are available immediately after system startup. We call them "predefined" elements. DataObject types and workers can currently be only predefined, for the other entities additional definitions can be added and updated via the ReST API.

The structure of these files is similar, e.g. the standard dataObjectTypes.json looks like this: 

{
   "dataObjectTypes": 
   [
      {
      	"name": "recordBulks", 
        "persistent": {
           "store": "${store}",
           "object": "${_bucketName}/${_uuid}"
        },
        "transient": {
           "store": "${tempStore}",
           "object": "${_bucketName}/${_uuid}"
        }
      }
   ]
}

Each file contains a single JSON object with a single key identical to the filename without suffix ("dataObjectTypes", "workers", "buckets", "workflows", "jobs"). The value for this key is a sequence of appropriate entity definitions, see the respective page for details:

Predefined entities cannot be changed or deleted using the ReST API, but only by changing the configuration files and restarting SMILA. Therefore they are marked with a "readOnly": true property when retrieved from the JobManager. You can of course make copies of them with different names to create variants.

If an entity cannot be parsed or is not valid (e.g. a job definition using an undefined workflow) you will find error messages in the SMILA log file.

Retrieved from "https://wiki.eclipse.org/index.php?title=SMILA/Documentation/JobManagerConfiguration&oldid=278088"

This page was last modified 08:46, 11 November 2011 by Eclipsepedia anonymous user Unnamed Poltroon.

Back to the top