SMILA/Documentation/JobParameters

Available since SMILA 0.9!

Job Parameters

The Job Manager entities can be parameterized to be configured for an actual use case.

Entities that declare parameter variables are:

• Data object types
• Worker definitions

Entities that initialize or set parameters are:

• Job definitions
• Workflow definitions
  • Global workflow parameters
  • Local action parameters
• Bucket definitions

The parameter priority increases from top to bottom, meaning that the parameters in job definitions, for example, have a lower priority than those in workflow definitions.

Workflow definitions are special entities in as far as they allow you to define general parameters in the global workflow section and worker-specific parameters in local action sections. It is important to know that the latter do not affect the buckets that the respective worker is using. As a consequence, parameters declared in a data object type can only be initialized using global workflow parameters or job parameters alternatively.

Example

Job:

  {
    "name":"...",
    "workflow":"...",
    "parameters":{
      "p1":"a"   // job parameter
    }
  }

Workflow:

  {
    "name":"...",  
    "parameters":{
      "p1":"b"   // global workflow param, overwrites "a" from job def
    },
    ...
    "actions":[
       ...
      {
        "worker":"...",
        "parameters":{
          "p1":"c"  // lokal workflow param, overwrites "b" from global
        },
     ...
  }

Setting parameters for specific workers in jobs

Available since SMILA 1.3!

To set different values for the same parameter in different workers in one workflow, use the worker name (plus a dot ".") as a prefix to the parameter in the job parameter definition. For example:

  {
    "name":"...",
    "workflow":"...",
    "parameters":{
      "worker1.p1":"a",
      "worker2.p1":"b"
    }
  }

In this job, worker1 would see value "a" for parameter "p1", while worker2 would see value "b". Other workers would not see a parameter "p1" at all. Of course, all workers would see also parameters "worker1.p1" and "worker2.p1" as the prefixed versions are not removed.

Another example:

  {
    "name":"...",
    "workflow":"...",
    "parameters":{
      "p1":"a",
      "worker2.p1":"b"
    }
  }

In this case, only worker2 would see "p1" and "worker2.p1" both with value "b", while all other workers would see parameter "p1" with value "a" and parameter "worker2.p1" with value "b".

The same transformation is applied to the general workflow parameters. It is not done for parameters in workflow actions.

The priorities are the same as for normal priorities: workflow parameters override job parameter, and action parameter override them all. An example:

• Job parameters:

"parameters": {
  "name1": "v1",
  "worker2.name1" : "v2"
}

• Workflow parameters:

"parameters": {
  "worker1.name1" : "v3"
}

• In an action for worker2:

"parameters": {
  "name1" : "v4"
}

The workers will get these values for parameter "name1":

• worker1 sees value "v3": the workflow parameter will be copied to "name1" and override the value from the job.
• worker2 in the action with the parameter sees value "v4", because the action parameter overrides everything
• worker2 in an action without the parameters sees value "v2": the job parameter "worker2.name1" is copied to "name1" and not overridden anymore
• any other worker will see "v1": the value from the job parameter will not be overridden for other workers.

If you use complex parameters (i.e. the parameter values are maps or arrays), the copying will be done only for the top-level parameter names and the parameters will not be merged:

"parameters": {
  "name": {
    "k1":"v1"
    "k2":"v2"
  }
  "worker2.name" : {
    "k1":"v3"
    "worker.k2":"v4"
  }
}

• worker1 sees "name": { "k1":"v1", "k2":"v2" } as usual.
• worker2 sees "name": { "k1":"v3", "worker.k2":"v4" }: the sub-parameter "worker2.k2" is not copied to "k2", also the sub-parameter "k2" from parameter "name" is not merged into the map for worker2.name.