Skip to main content
Jump to: navigation, search

Difference between revisions of "SMILA/Documentation/JobDefinitions"

m (Job definitions)
 
(33 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Jobs =
+
= Job definitions =
== Job Definitions ==  
+
  
The user can create a job by choosing a workflow and setting all its parameters. All parameter variables of the used data object types and workers that are not set as bucket or workflow parameters have to be set as job parameters.
+
To run a certain workflow in SMILA, you will have to create a job definition first that references a workflow and sets the desired parameters. With a job definition alone, the system is not yet doing anything. First, the job must be started to get a so called [[SMILA/Documentation/JobRuns|job run]]. For jobs running in "standard" mode, the actual job processing is triggered with every new object that is dropped into the bucket connected to the start action of the respective workflow. The triggering continues with new objects until the job run is finished manually. Jobs running in "runOnce" mode, by contrast, do not react on new objects but process all objects currently contained in the respective input bucket and then finish automatically.
 +
Jobs can also be predefined in the system, in which case they are located at <tt>configuration/org.eclipse.smila.jobmanager/jobs.json</tt> and are marked with <tt>"readOnly": true</tt>. Predefined jobs cannot be modified and in contrast to user-defined jobs do not contain a <tt>timestamp</tt> element.
  
Jobs are linked by sharing <tt>persistent</tt> buckets: A workflow having a start action worker with an input section is started when another workflow adds objects to the <tt>persistent</tt> bucket associated to this worker's input slot. (Note: this only works with <tt>persistent</tt> input buckets, not transient ones!)
+
Please note that job names must conform to the job manager naming convention:
 +
* names must inly consist of the following characters: <b>a-zA-Z._-</b>
  
The job definition is provided by the user. It defines the workflow to be used along with the parameters to configure the workflow.
+
If they do not conform, they won't be accessible in SMILA.
* <tt>name</tt> is mandatory and defines the name of the job.
+
* Pushing elements with invalid names will result in a 400 Bad Request,
** If <tt>name</tt> is missing an exception occurs.
+
* predefined elements with invalid names won't be loaded, a warning will be logged in the SMILA.log file.
** There will be an error when trying to add a job with the same name as an existing job.
+
E.g.
* <tt>parameters</tt>
+
<source lang="text">
** This is optional and defines the job parameters that will be resolved in the workflow to configure the participating workers and instantiate the buckets.
+
... WARN  ...  internal.DefinitionPersistenceImpl            - Error parsing predefined jobs from configuration area
** All parameter (variables) that are still not set in the worker, bucket and data object type definitions used by the job workflow must be set here. Otherwise an error will occur when trying to add the job.
+
  org.eclipse.smila.common.exceptions.InvalidDefinitionException: Value 'indexInsert/Update' in field 'name'
* <tt>workflow</tt>
+
  is not valid: A name must match pattern ^[a-zA-Z0-9-_\.]+$.
** mandatory field. If the field is missing an exception occurs when trying to add the job.
+
</source>
** references an existing workflow in the jobmanager. If the workflow with the given name does not exist, there will be an error when trying to add the job.
+
  
Sample:
+
== Job properties in detail ==
 +
 
 +
* <tt>name</tt>: Required. Defines the name of the job.
 +
* <tt>workflow</tt>: Required. Gives the name of the desired workflow.
 +
* <tt>modes</tt>: Optional. Restricts the modes the job can be started in and defines the default mode. Possible modes are {{code|standard}} and {{code|runOnce}}.
 +
** the first mode in this list will be used as the default job run mode (i.e. if no mode is provided during job start).
 +
** If a <tt>modes</tt> section in the workflow is already set, the <tt>modes</tt> section in the job definition can not add modes to the list of available modes defined in the workflow definition. Adding a job definition that violates the workflow definition concerning modes will result in a <tt>400 Bad Request</tt> error. See [[SMILA/Documentation/JobRuns#Job_modes|Job modes]] for more information.
 +
* <tt>parameters</tt>: Optional. Defines the [[SMILA/Documentation/JobParameters|job parameters]] that will be resolved in the workflow to configure the participating workers and to instantiate the buckets. All parameter (variables) that are declared in the used data object types and workers and that have not yet been set in the workflow or bucket definitions must be set here at the latest. Otherwise an error will occur when trying to create the job.
 +
* <tt>taskControl</tt>: Optional, additional configuration of task delivery to the workers that are part of this job's workflow. Currently, this section can describe that for some workers tasks should not be delivered as fast as possible but with a delay: A delayed worker will only receive a new task if no other task for this worker in this job has been started in the configured time period. E.g., if the delay is set to 10 seconds, the worker will not get a new task if another task for this worker in the same job has been started during the last 10 seconds. This feature can be useful for example during crawling, when you want to reduce the number requests to a data source (typically web sources) to avoid denial of service.
 +
** <tt>workers</tt>: A list of names of workers that should be delayed.
 +
** <tt>delay</tt>: a long value specifying the delay in milliseconds.
 +
Job definitions can include additional information (e.g. comments or additional for external tools, etc.), but a GET request will return only relevant information (i.e. the above attributes). If you want to retrieve the additional info that is present in the json file or has been posted with the definition, add <tt>returnDetails=true</tt> as request parameter.
 +
 
 +
== Example ==
 +
An exemplary job definition:
 +
 
 +
<pre>
 +
{
 +
  "name":"myJob",
 +
  "workflow":"myWorkflow",
 +
  "modes": ["standard"],
 +
  "parameters":{
 +
    "index": "wikipedia",
 +
    "store": "wikidocs"
 +
  }
 +
}
 +
</pre>
 +
 
 +
An exemplary job definition including task control section:  
  
 
<pre>
 
<pre>
 
{
 
{
   "name":"indexUpdateTestJob",
+
   "name":"crawlSmilaWiki",
 +
  "workflow":"crawlWeb",
 
   "parameters":{
 
   "parameters":{
     "index":"wikipedia",
+
     "startUrl":"http://wiki.eclipse.org/SMILA",
     "store":"wikidocs"
+
     "dataSource": "web",
 +
    "tempStore": "wikidocs",
 +
    "jobToPushTo":"indexUpdate"
 
   },
 
   },
  "workflow":"indexUpdate"
+
  "taskControl":{
 +
    "workers": ["webCrawler"],
 +
    "delay": 2000
 +
  }
 
}
 
}
 
</pre>
 
</pre>
  
=== Monitor jobs ===
+
== List, create, modify jobs ==
==== All jobs ====
+
=== All jobs ===
  
Use a GET request to retrieve monitoring information for all jobs. Use POST for adding or updating a job.
+
Use a GET request to retrieve a list of all job definitions. Use POST for adding or updating a job definition.
  
 
'''Supported operations:'''  
 
'''Supported operations:'''  
*GET: Get a list of all defined job definitions and details about latest job run. Switch off details with <tt>returnDetails=false</tt> parameter. If there are no jobs defined, you will get an empty list.
+
*GET: Get a list of all job definitions and details about latest job run. Switch off details with <tt>returnDetails=false</tt> as a URL parameter. If there are no jobs defined, you will get an empty list.
*POST: Put one new job or update existing one. If an already existing name is used, the existing job definition will be updated if validation was successful. If you update during a running job, the running job will not be influenced, but after the next start of the job the updated job definition will be used.
+
*POST: Create a new job definition or update an existing one. If the job already exists, it will be updated after successful validation. However, the changes will not apply until the next job run, i.e. the current job run is not influenced by the changes.
 
+
<pre> If you POST more than one workflow only the first job will be added, all following workflows will be ignored.</pre>
+
  
 
'''Usage:'''  
 
'''Usage:'''  
  
*URL: <tt><nowiki>http://<hostname>:8080/smila/jobmanager/jobs</nowiki></tt>.
+
*URL: <tt><nowiki>http://<hostname>:8080/smila/jobmanager/jobs/</nowiki></tt>
 
*Allowed methods:  
 
*Allowed methods:  
 
**GET
 
**GET
Line 50: Line 82:
 
**200 OK: Upon successful execution (GET).  
 
**200 OK: Upon successful execution (GET).  
 
**201 CREATED: Upon successful execution (POST).
 
**201 CREATED: Upon successful execution (POST).
**400 Bad Request: If you reference undefined workflows, not all parameters are resolved, mandatory fields are missing or validation finds errors you get a Bad Request (POST).  
+
**400 Bad Request: If you reference undefined workflows, if not all parameters were resolved, if mandatory fields are missing, the job name is invalid or if validation finds errors (POST).
  
==== Specific job ====
+
'''Examples:'''
  
Use a GET request to retrieve monitoring information for a specific job. Use DELETE for deleting a job.
+
To get all job definitions:
 +
 
 +
<pre>
 +
GET /smila/jobmanager/jobs/
 +
</pre>
 +
 
 +
The result would be:
 +
 
 +
<pre>
 +
HTTP/1.x 200 OK
 +
 
 +
{
 +
  "jobs" : [ {
 +
    "name" : "myJob",
 +
    "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/"
 +
  } ]
 +
}
 +
</pre>
 +
 
 +
To create a job:
 +
 
 +
<pre>
 +
POST /smila/jobmanager/jobs/
 +
 
 +
{
 +
  "name":"myJob",
 +
  "parameters":{
 +
    "index": "wikipedia",
 +
    "store": "wikidocs"
 +
  },
 +
  "workflow":"myWorkflow"
 +
}
 +
</pre>
 +
 
 +
The result would be:
 +
 
 +
<pre>
 +
HTTP/1.x 201 CREATED
 +
 
 +
{
 +
  "name" : "myJob",
 +
  "timestamp": "2011-08-12T14:49:48.862+0200",
 +
  "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/"
 +
}
 +
</pre>
 +
 
 +
=== Specific job ===
 +
 
 +
Use a GET request to retrieve the definition of a specific job and an overview of job runs (current job run, if running, and a history of any previous job runs). Use DELETE to delete a job.
  
 
'''Supported operations:'''  
 
'''Supported operations:'''  
*GET: retrieve information about a job definition with a given name.
+
*GET: get the definition of the given job.
*DELETE: delete a job with a given name.
+
** You can set the URL parameter <tt>returnDetails</tt> to <tt>true</tt> to return additional information that might have been provided when creating the job. If the parameter is ommitted or set to <tt>false</tt> only the relevant information (see above) is gathered.
 +
** You can set the URL parameter <tt>returnRuns</tt> to <tt>false</tt> to hide the job run lists. This should make the request faster, if you don't need this information. If the parameter is ommitted or set to <tt>true</tt>, the job run list is included.
 +
*DELETE: delete the given job definition.
 +
*POST: start a job run, see [[SMILA/Documentation/JobRuns#Start_job_run]]
  
 
'''Usage:'''  
 
'''Usage:'''  
  
*URL: <tt><nowiki>http://<hostname>:8080/smila/jobmanager/jobs/<job-name></nowiki></tt>.
+
*URL: <tt><nowiki>http://<hostname>:8080/smila/jobmanager/jobs/<job-name>/</nowiki></tt>
 
*Allowed methods:  
 
*Allowed methods:  
 
**GET
 
**GET
 
**DELETE
 
**DELETE
 +
**POST: see [[SMILA/Documentation/JobRuns#Start_job_run]]
 
*Response status codes:  
 
*Response status codes:  
**200 OK: Upon successful execution (GET, DELETE). If the to be deleted job does not exist you will get a 200 anyway.  
+
**200 OK: Upon successful execution (GET, DELETE). If the job definition to be deleted does not exist you will get a 200 anyway.
**404 Server Error: If a wrong name is used, a HTTP 404 Server Error is followed by an error in json format (GET).
+
**400 Bad Request: If a job run mode has been requested that is not allowed by workflow or job definition.
 +
**404 Server Error: If an undefined name is used, an HTTP 404 Server Error including an error message in the response body will be returned.
 +
 
 +
'''Examples:'''
 +
 
 +
To get a specific job definition:
 +
 
 +
<pre>
 +
GET /smila/jobmanager/jobs/myJob/
 +
</pre>
 +
 
 +
The result would be:
 +
 
 +
<pre>
 +
HTTP/1.x 200 OK
 +
 
 +
{
 +
  "definition": {
 +
    "name":"myJob",
 +
    "timestamp": "2011-08-12T14:49:48.862+0200",
 +
    "parameters":{
 +
      "index": "wikipedia",
 +
      "store": "wikidocs"
 +
    },
 +
    "workflow":"myWorkflow"
 +
  },
 +
  "runs": {
 +
    "current" : {
 +
      "jobId" : "20110930-121108155498",
 +
      "state" : "RUNNING",
 +
      "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/20110930-121108155498/"
 +
    },
 +
    "history" : [ {
 +
      "jobId" : "20110930-121054946584",
 +
      "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/20110930-121054946584/"
 +
    }, {
 +
      "jobId" : "20110930-121042898813",
 +
      "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/20110930-121042898813/"
 +
    } ]
 +
  }
 +
}
 +
</pre>
 +
 
 +
If the job is currently not running, the "current" map will be empty, if there are no finished job runs, the "history" sequence will be empty. If "returnRuns=false" has been set in the URL, the complete "runs" part will be missing:
 +
 
 +
<pre>
 +
GET /smila/jobmanager/jobs/myJob/?returnRuns=false
 +
 
 +
-->
 +
 
 +
HTTP/1.x 200 OK
 +
{
 +
  "definition": {
 +
    "name":"myJob",
 +
    "timestamp": "2011-08-12T14:49:48.862+0200",
 +
    "parameters":{
 +
      "index": "wikipedia",
 +
      "store": "wikidocs"
 +
    },
 +
    "workflow":"myWorkflow"
 +
  }
 +
}
 +
</pre>

Latest revision as of 08:19, 8 January 2013

Job definitions

To run a certain workflow in SMILA, you will have to create a job definition first that references a workflow and sets the desired parameters. With a job definition alone, the system is not yet doing anything. First, the job must be started to get a so called job run. For jobs running in "standard" mode, the actual job processing is triggered with every new object that is dropped into the bucket connected to the start action of the respective workflow. The triggering continues with new objects until the job run is finished manually. Jobs running in "runOnce" mode, by contrast, do not react on new objects but process all objects currently contained in the respective input bucket and then finish automatically. Jobs can also be predefined in the system, in which case they are located at configuration/org.eclipse.smila.jobmanager/jobs.json and are marked with "readOnly": true. Predefined jobs cannot be modified and in contrast to user-defined jobs do not contain a timestamp element.

Please note that job names must conform to the job manager naming convention:

  • names must inly consist of the following characters: a-zA-Z._-

If they do not conform, they won't be accessible in SMILA.

  • Pushing elements with invalid names will result in a 400 Bad Request,
  • predefined elements with invalid names won't be loaded, a warning will be logged in the SMILA.log file.

E.g.

... WARN  ...  internal.DefinitionPersistenceImpl            - Error parsing predefined jobs from configuration area
  org.eclipse.smila.common.exceptions.InvalidDefinitionException: Value 'indexInsert/Update' in field 'name'
  is not valid: A name must match pattern ^[a-zA-Z0-9-_\.]+$.

Job properties in detail

  • name: Required. Defines the name of the job.
  • workflow: Required. Gives the name of the desired workflow.
  • modes: Optional. Restricts the modes the job can be started in and defines the default mode. Possible modes are standard and runOnce.
    • the first mode in this list will be used as the default job run mode (i.e. if no mode is provided during job start).
    • If a modes section in the workflow is already set, the modes section in the job definition can not add modes to the list of available modes defined in the workflow definition. Adding a job definition that violates the workflow definition concerning modes will result in a 400 Bad Request error. See Job modes for more information.
  • parameters: Optional. Defines the job parameters that will be resolved in the workflow to configure the participating workers and to instantiate the buckets. All parameter (variables) that are declared in the used data object types and workers and that have not yet been set in the workflow or bucket definitions must be set here at the latest. Otherwise an error will occur when trying to create the job.
  • taskControl: Optional, additional configuration of task delivery to the workers that are part of this job's workflow. Currently, this section can describe that for some workers tasks should not be delivered as fast as possible but with a delay: A delayed worker will only receive a new task if no other task for this worker in this job has been started in the configured time period. E.g., if the delay is set to 10 seconds, the worker will not get a new task if another task for this worker in the same job has been started during the last 10 seconds. This feature can be useful for example during crawling, when you want to reduce the number requests to a data source (typically web sources) to avoid denial of service.
    • workers: A list of names of workers that should be delayed.
    • delay: a long value specifying the delay in milliseconds.

Job definitions can include additional information (e.g. comments or additional for external tools, etc.), but a GET request will return only relevant information (i.e. the above attributes). If you want to retrieve the additional info that is present in the json file or has been posted with the definition, add returnDetails=true as request parameter.

Example

An exemplary job definition:

{
  "name":"myJob",
  "workflow":"myWorkflow",
  "modes": ["standard"],
  "parameters":{
    "index": "wikipedia",
    "store": "wikidocs"
   }
}

An exemplary job definition including task control section:

{
  "name":"crawlSmilaWiki",
  "workflow":"crawlWeb",
  "parameters":{
    "startUrl":"http://wiki.eclipse.org/SMILA",
    "dataSource": "web",
    "tempStore": "wikidocs",
    "jobToPushTo":"indexUpdate"
   },
   "taskControl":{
    "workers": ["webCrawler"],
    "delay": 2000
   }
}

List, create, modify jobs

All jobs

Use a GET request to retrieve a list of all job definitions. Use POST for adding or updating a job definition.

Supported operations:

  • GET: Get a list of all job definitions and details about latest job run. Switch off details with returnDetails=false as a URL parameter. If there are no jobs defined, you will get an empty list.
  • POST: Create a new job definition or update an existing one. If the job already exists, it will be updated after successful validation. However, the changes will not apply until the next job run, i.e. the current job run is not influenced by the changes.

Usage:

  • URL: http://<hostname>:8080/smila/jobmanager/jobs/
  • Allowed methods:
    • GET
    • POST
  • Response status codes:
    • 200 OK: Upon successful execution (GET).
    • 201 CREATED: Upon successful execution (POST).
    • 400 Bad Request: If you reference undefined workflows, if not all parameters were resolved, if mandatory fields are missing, the job name is invalid or if validation finds errors (POST).

Examples:

To get all job definitions:

GET /smila/jobmanager/jobs/

The result would be:

HTTP/1.x 200 OK

{
  "jobs" : [ {
    "name" : "myJob",
    "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/"
  } ]
}

To create a job:

POST /smila/jobmanager/jobs/

{
  "name":"myJob",
  "parameters":{
    "index": "wikipedia",
    "store": "wikidocs"
   },
  "workflow":"myWorkflow"
}

The result would be:

HTTP/1.x 201 CREATED

{
  "name" : "myJob",
  "timestamp": "2011-08-12T14:49:48.862+0200",
  "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/"
}

Specific job

Use a GET request to retrieve the definition of a specific job and an overview of job runs (current job run, if running, and a history of any previous job runs). Use DELETE to delete a job.

Supported operations:

  • GET: get the definition of the given job.
    • You can set the URL parameter returnDetails to true to return additional information that might have been provided when creating the job. If the parameter is ommitted or set to false only the relevant information (see above) is gathered.
    • You can set the URL parameter returnRuns to false to hide the job run lists. This should make the request faster, if you don't need this information. If the parameter is ommitted or set to true, the job run list is included.
  • DELETE: delete the given job definition.
  • POST: start a job run, see SMILA/Documentation/JobRuns#Start_job_run

Usage:

  • URL: http://<hostname>:8080/smila/jobmanager/jobs/<job-name>/
  • Allowed methods:
  • Response status codes:
    • 200 OK: Upon successful execution (GET, DELETE). If the job definition to be deleted does not exist you will get a 200 anyway.
    • 400 Bad Request: If a job run mode has been requested that is not allowed by workflow or job definition.
    • 404 Server Error: If an undefined name is used, an HTTP 404 Server Error including an error message in the response body will be returned.

Examples:

To get a specific job definition:

GET /smila/jobmanager/jobs/myJob/

The result would be:

HTTP/1.x 200 OK

{
  "definition": {
    "name":"myJob",
    "timestamp": "2011-08-12T14:49:48.862+0200",
    "parameters":{
      "index": "wikipedia",
      "store": "wikidocs"
     },
    "workflow":"myWorkflow"
  }, 
  "runs": {
    "current" : {
      "jobId" : "20110930-121108155498",
      "state" : "RUNNING",
      "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/20110930-121108155498/"
    },
    "history" : [ {
      "jobId" : "20110930-121054946584",
      "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/20110930-121054946584/"
    }, {
      "jobId" : "20110930-121042898813",
      "url" : "http://localhost:8080/smila/jobmanager/jobs/myJob/20110930-121042898813/"
    } ]
  }
}

If the job is currently not running, the "current" map will be empty, if there are no finished job runs, the "history" sequence will be empty. If "returnRuns=false" has been set in the URL, the complete "runs" part will be missing:

GET /smila/jobmanager/jobs/myJob/?returnRuns=false

--> 

HTTP/1.x 200 OK
{
  "definition": {
    "name":"myJob",
    "timestamp": "2011-08-12T14:49:48.862+0200",
    "parameters":{
      "index": "wikipedia",
      "store": "wikidocs"
     },
    "workflow":"myWorkflow"
  }
}

Back to the top