SMILA/Specifications/Service Runtime Parameters

Service runtime parameters as SMILA record annotations

We have planned to add somthing similar to the empolis IAS general/pipelet parameter mechanism in SMILA using record annotations. However, we have not yet really specified how to do this which leads to some ad-hoc definitions (e.g. in LuceneIndexService) which may not be sufficient in the general case.

Analysis, Requirements

Example: Parameters in empolis IAS

In empolis IAS we have 4 different places where parameters can be specified:

• OSC pipelet parameters: part of the definiton of a pipelet instance in the OSC file.
• OSC pipeline parameters: part of the definiton of a pipeline in OSC file.
• Query general parameters: part of <GeneralParameters> section in query - possibly relevant for all pipelets in a pipeline.
• Query pipelet parameters: part of <PipeletParameters id="pipelet-name"> section in query - relevant only for the named pipelet.

During pipelet initialization, only the pipelet parameters from the OSC are available. During pipelet execution, a parameter requested by a pipelet can be set in all 4 places, so there is a precedence rule to decide from which location a parameter must be actually read.

• Starting with reading from query pipelet parameters
• if none is found there, look in query general parameters
• if none is found there, look in OSC pipelet parameters
• if none is found there, look in OSC pipeline parameters

All in all, this is a quite powerful, but complex scheme. We would like to come up with something easier for SMILA.

Current state in SMILA

• Each record in the pipelet can have annotations that could be used to represent parameters
• There is currently no place to attach parameters to the complete request.
• The BPEL invocation activities for SMILA pipelets and services can contain record annotations that are written to each record in the input variable just before the invocation of the pipelet or service.
• SMILA services have a lifecycle independent from the BPEL/pipeline engine so they cannot get init parameters from the pipeline configuration.
• SMILA pipelets are configured at instance creation time by a configuration element in the invocation activity. They can also have annotations that are added to the record at invocation, but this does not make much sense for pipelets as there is already a single instance for each invocation activity that can be configured differently using the configuration properties.
• There are no rules yet how to name annotations and in which order to evaluate them (and also no utility classes to make reading parameter annotations easier for service implementors).

Requirements

• Define naming of parameter annotations.
• Define precedence for query annotations, BPEL annotations, service/pipelet configurations.
• Define interface for a helper class that pipelet/service implenentors can use to read parameter annotations.

Proposal for SMILA

Parameter types

There could be two different kinds of parameters:

• General parameters: These parameters are set for all pipelets in a pipeline at once. A good example may be a parameter "language" that specifies the query language. There are possibly a number of pipelets that are interested in such a parameter to adjust their operation language-specifically. A client does not want to set this parameter for each pipelet seperately, and having to do it would make the client too depenendent on details of the called pipeline. Hence, we definitifely need a way to set parameters without knowing about specific pipelet names.
• Pipelet/Service parameters: These parameters are set only for a single pipelet in the called pipeline. Other pipelets do not see it. A pipelet parameter overrides a general parameter of the same name for this specific pipelet. So there would need to be a way to address a parameter to a single pipelet. This means that each pipelet that wants to receive specific paramters must have a unique name wrt. to the pipeline.

Do we really need pipelet specific parameters? They make the parameter handling significantly more complicated, and pipelets become a bit more complicated, e.g. because they must know about their names. On the other hand: They are used by a client only in very special situations, when two pipelets need two different values for the same parameter (or a single one needs a different value than the general value set for all others), and both must be controlled by the client. Even such cases could be simulated with only general parameters by setting the second value to a different parameter name and putting another pipelet between the pipelets in questions that moves the second value to the correct parameter name. Therefore, in this first version, we do not specify pipelet parameters. Note that nevertheless it is still possible to add annotations outside the definitions in this specification (and without special support by the Search API or parameter helper class (see below)) to do more advanced things if necessary.

Parameter values

Usually, parameters will have only a single value, like the requested size of the search result, or the language of the query. Sometimes lists of values will be necessary, e.g. a list of attribute names to process. More complex structures are of course possible (see, e.g., the OrderBy parameter), too, but not often used, so we limit the generic support to single values and simple lists (this corresponds to what is currently supported by the generic PipeletConfiguration object).

All parameter values must be represented as String because this is the only value type provided by record annotations. To transport other types of values, the helper classes offer functions that convert the stored string value to a requested value type, if possible. For a start, we will support integers, floating point numbers (double precision), and booleans.

Structure of parameter annotation

All parameters are part of a top-level annotation "parameters" of the query record. This annotation is called the "parameter annotation" or "parameters" for short.

Example: Parameterization of a search query - see [[SMILA/Specifications/Search API] for an explanation of the parameters:

<Record>
  <Id/>
  <A ...>                                        <!-- attributes -->
  <A ...>
  <A ...>
  <An n="parameters">                            <!-- the parameter annotation -->
    <V n="resultSize">12</V>
    <V n="resultOffset">24</V>
    <V n="threshold">0.2</V>
    <V n="language">de</V>
    <An n="orderBy">                           <!-- order by modificaton date first, newest come first -->
      <V v="attribute">modificationDate</V>
      <V v="mode">DESC</V>
    </An>
    <An n="orderBy">                           <!-- results with same mod date are ordered by their size, smallest first -->
      <V v="attribute">size</V>
      <V v="mode">ASC</V>
    </An>
  </An>
</Record>

Precedence rules for deciding which parameter value to use

The precedence rule for a service/pipelet named "foo" looking for a single-valued parameter named "bar" would be

• if it exists: use named value "bar" of parameter annotation of the processed record.
• else: SearchServices/Pipelets could additionally try to use the annotations of the effective record here
• else: if standard pipelet config format is used: use property "bar" from the pipelet/service configuration
• else: use related setting from service specific config or hardcoded default values

For a multi-valued parameter the sequence would be:

• if it exists: use anon values of sub-annotation "bar" of parameter annotation of the processed record
• else: SearchServices/Pipelets could additionally try to use the annotations of the effective record here
• else: if standard pipelet config format is used: use property "bar" from the pipelet/service configuration
• else: use related setting from service specific config or hardcoded default values

For a description of SearchServices/Pipelets see SMILA/Specifications/Search Processing

TO DECIDE: There are two possibilities for the handling of <SetAnnotations> in the BPEL <invokeService>/<invokePipelet> extension activities:

• it could overwrite existing annotation values and so ensure that a parameter value is set correctly for special pipelet, regardles of what is defined in the query (this would be the current implementation).
• it could be changed NOT to overwrite existing annotation values. This way it would be possible to use them to define default values for required parameters that are not set in the query.

The second alternative is probably more understandable as it describes a more clean precendence: single query > pipeline > service instance/configuration > hardcoded defaults. Alternatively it would be possible to add an attribute to <SetAnnotations> allowing to configure its behaviour.

Helper class interface

The following class should help services/pipelets to access the parameters they need without having to know about the basic parameter structure and precedence rules and in most cases even without having to know with the annotation interface at all. At the beginning of each invocation the service/pipelet creates such an access helper and uses it during this invocation to access parameters for each record it processes.

class MissingParameterException extends ProcessingException {
    // insert the usual stuff ...
}
 
class ParameterAccess {
    // create accessor for given record 
    ParameterAccess(Blackboard blackboard);
 
    // for search pipelets/services: create accessor for given record, effective query record for fallback
    ParameterAccess(Blackboard blackboard, Id queryRecord);
 
    // set the ID of the record to read parameters from.
    // returns "this", make it easier to use this method immediately after constructor.
    ParameterAccess setCurrentRecord(Id currentRecord);
 
    // set the configuration of the current pipelet, to read fallback values for missing parameters.
    // returns "this", make it easier to use this method immediately after constructor
    ParameterAccess setPipeletConfiguration(PipeletConfiguration config)
 
    // access to predefined parameters as supported in the search API
    String getQuery();
    int getResultSize(); // hardcoded default = 10
    int setResultOffset(); // hardcoded default = 0
    double getThreshold(); // hardcoded default = 0.0
    String getLanguage(); // hardcoded default = null
    String getUserID(); // hardcoded default = null
    Iterator<String> getOrderByAttributeNames(); // hardcoded default is empty iterator
    OrderMode getOrderByMode(String attributeName); // return null for unordered attributes
 
    // get named value of parameter annotation, according to precedence rules.
    // return default value if no matching parameter value can be found
    String getParameter(String name, String defaultValue);
 
    // get named value of parameter annotation, according to precedence rules.
    // throw exception if no matching parameter value can be found.
    String getRequriedParameter(String name) throws MissingParameterException;
 
    // get anonymous values of first (according to precendence rules) sub annotation
    // return empty list if no matching parameter value can be found
    List<String> getParameters(String name);
 
    // get anonymous values of first (according to precendence rules) sub annotation
    //  throw exception if no matching parameter value can be found.
    List<String> getParameters(String name) throws MissingParameterException;
 
    // type-aware convenience methods: convert result of getParameter() to desired type.
    // needed for lists, too? Don't think so, currently.
    Integer getIntParameter(String name, Integer defaultValue) throws NumberFormatException;
    Double getFloatParameter(String name, Double defaultValue) throws NumberFormatException;
    Boolean getBooleanParameter(String name, Boolean defaultValue);
 
    // type-aware convenience methods: convert result of getRequiredParameter() to desired type.
    // needed for lists, too? Don't think so, currently.
    Integer getRequiredIntParameter(String name) throws NumberFormatException, MissingParameterException;
    Double getRequiredFloatParameter(String name) throws NumberFormatException, MissingParameterException;
    Boolean getRequiredBooleanParameter(String name) throws MissingParameterException;
 
    // access to first sub annotation, according to precedence rules.
    Annotation getAnnotation(String name);
 
    // access to a multiple sub annotations, according to precedence rules
    // the annotations are not merged from multiple sources, e.g. if both service and general parameters have sub annotations
    // of this name, only the service parameter annotations are returned.
    List<Annotation> getAnnotations(String name);
}

QUESTION: do we also need methods to set parameters in this helper?