Difference between revisions of "SMILA/Documentation/Usage of Blackboard Service"

From Eclipsepedia

Jump to: navigation, search
(added "Creation and Lifecycle of Blackboard")
Line 1: Line 1:
== What is Blackboard ==
+
== What is the blackboard ==
Purpose of the Blackboard is management of SMILA record data during processing in SMILA component (Connectivity, Workflow Processor). Complete record data is stored only on a Blackboard which is not pushed through the workflow engine itself. The Blackboard hides handling of record persistence from the services.
+
The purpose of the blackboard is the management of SMILA record data during processing in SMILA components (Connectivity, Workflow Processor). Complete record data is stored only on a blackboard which is not pushed through the workflow engine itself. The blackboard hides the handling of record persistence from the services.
Clients should generally manipulate records using Blackboard API methods in most cases so records will be completely under control of the Blackboard.
+
Clients should generally manipulate records using blackboard API methods in most cases, so records will be completely under the control of the blackboard.
  
== Creation and Lifecycle of Blackboard ==
+
== Creation and lifecycle of blackboard ==
  
For Blackboard creation we use a BlackboardFactory service running as a declarative service.
+
For blackboard creation we use a BlackboardFactory service running as a declarative service.
* The factory can create Blackboard instances which are either "transient" (pure in-memory implementation, not using any storages) or "persisting" (linked to binary storage and optionally to record storage). The client selects which kind of blackboard it wants to use. A persisting blackboard can only be created successfully, if at least a binary storage is known. Creation of transient blackboards is always possible.
+
* The factory can create blackboard instances which are either "transient" (pure in-memory implementation, not using any storages) or "persisting" (linked to binary storage and optionally to record storage). The client selects which kind of blackboard it wants to use. A persisting blackboard can only be created successfully, if at least a binary storage is known. Creation of transient blackboards is always possible.
 
* For each "session" an own new blackboard instance is created that manages only those records worked on by this request. A session is for example:
 
* For each "session" an own new blackboard instance is created that manages only those records worked on by this request. A session is for example:
 
** a single task list execution of a QueueWorker router or listener (i.e. add/delete one record in Connectivity, or processing one input record from a queue message and manage all additional records created by the invoked workflows)
 
** a single task list execution of a QueueWorker router or listener (i.e. add/delete one record in Connectivity, or processing one input record from a queue message and manage all additional records created by the invoked workflows)
Line 31: Line 31:
 
</source>
 
</source>
  
== Usage of Blackboard ==
+
== Usage of blackboard ==
 +
{{Note|Discussion| happens @ {{bug|336818}} }}
  
=== Record lifecycle on the Blackboard ===
+
=== Record lifecycle on the blackboard ===
  
Record can be put onto Blackboard with one of the following operations:
+
A record may be put on the blackboard using one of the following operations:
* create(Id);
+
* <tt>create(String id);</tt>
*: Creates a new record with a given Id. No data is loaded from persistence. If record with this Id already exists in the storages it will be overwritten when the created record will be committed. E.g. used by Connectivity to initialize the record from incoming data.
+
*: Creates a new record with the given ''id''. No data is loaded from persistence. If a record with this ID already exists in the storages it will be overwritten when the created record will be committed. E.g. used by Connectivity to initialize the record from incoming data.
* load(Id);
+
* <tt>load(String id);</tt>
 
*: Loads record data for the given Id from persistence. Used by a client to indicate that it wants to process this record.
 
*: Loads record data for the given Id from persistence. Used by a client to indicate that it wants to process this record.
* split(Id, String);
+
* <tt>copyRecord(String id, String copyId);
*: Creates a fragment of a given record, i.e. the record content is copied to a new Id derived from the given by adding a frament name (see Id Concept for details).
+
*: Creates a new record using ''copyId'' and copies all metadata from the record with 'id' to it. Attachments are not copied.
* setRecord(Record);
+
* <tt>setRecord(Record);</tt>
*: Puts record on the Blackboard, saves record attachments to BinStorage and replaces actual record attachments values with null.
+
*: Puts the record on the blackboard, saves record attachments to BinStorage, and replaces actual record attachments values by ''null''.
* synchronize(Record);
+
* <tt>synchronizeRecord(Record);</tt>
*: Assumes that record with the same Id as of given record already exists on Blackboard or in storage. Loads record from the storage if needed and updates it's properties with properties of the given record.
+
*: Assumes that a record with the same ID as the given one already exists on the blackboard or in the storage. Loads the record from the storage if needed and updates its properties with the properties of the given record.
  
Record is removed from the blackboard with one of these operations:
+
A record may be removed from the blackboard using one of these operations:
* commit(Id);
+
* <tt>commit(String id);</tt>
*: Saves record and attachments to storages and removes record from the Blackboard.
+
*: Saves the record including its attachments to storages and removes the record from the blackboard.
* invalidate(Id);
+
* <tt>invalidate(String id);</tt>
*: Record is removed from the Blackboard. If the record was created new (not overwritten) on the Blackboard it will be removed completely.
+
*: Removes the record from the blackboard. If the record was created newly (not overwritten) on the blackboard it will be removed completely. The record is not removed from persistence, though.
 +
* <tt>removeRecord(String id);</tt>
 +
*: Removes a record completely from blackboard and all persistence layer.
 +
 
 +
=== Record access ===
 +
 
 +
* <tt>Record getRecord(String id);</tt>
 +
*: Gets the record with its complete metadata. You should not rely on the record having all attachment values attached, only the names will be available. Use the blackboard attachments access methods to access attachments. If the record is not yet loaded in the blackboard, the PersistingBlackboard will return null, if it has a RecordStorage set and the record does not exist in record storage. If it doesn't have a RecordStorage a new record is created automatically. The TransientBlackboard creates a new record, too, if necessary. If you change the record metadata, you change the record stored on the blackboard, too, so you don't have to call setRecord() to write back the changes explicitly.
 +
* <tt>AnyMap getMetadata(String id);</tt>
 +
*: A shortcut for <tt>getRecord(id).getMetadata()</tt>
 +
* <tt>Record getRecord(String id, String filterName);</tt>
 +
*: Creates a copy of the record with only those metadata elements allowed by the named record filter. Changes done to the filtered metadata are not applied to the original record automatically.
 +
* <tt>Record filterRecord(Record record, String filterName);</tt>
 +
*: Applies a record filter known to the blackboard to a record which does not have to be loaded on the blackboard.
  
 
=== Attachments management ===
 
=== Attachments management ===
There are following methods for working with Record attachments in the Blackboard:
+
There are following methods for working with Record attachments in the blackboard:
* setAttachment(id, name, byte[]);
+
* <tt>setAttachment(id, name, byte[]);</tt>
* setAttachmentFromStream(id, name, InputStream);
+
* <tt>setAttachmentFromStream(id, name, InputStream);</tt>
* byte[] getAttachment(id, name);
+
* <tt>byte[] getAttachment(id, name);</tt>
* InputStream getAttachmentAsStream(id, name);
+
* <tt>InputStream getAttachmentAsStream(id, name);</tt>
* boolean hasAttachment(id, name);
+
* <tt>boolean hasAttachment(id, name);</tt>
  
Attachments are not stored anywhere in the Blackboard, they are saved to BinStorage directly and the actual attachment value in the corresponding Record is replaced with {{null}}. It is highly recommended to use only Stream methods to manage attachments because loading the whole attachments in memory will cause great memory consumption and can be cause for application crash.
+
Attachments are not stored anywhere in the blackboard, they are saved to BinStorage directly and the actual attachment value in the corresponding record is replaced by ''null''. It is highly recommended to use only <tt>Stream</tt> methods to manage attachments because loading the whole attachments in memory will cause great memory consumption and may cause application crashes.
  
=== Usage of Blackboard Notes ===
+
=== Usage of blackboard notes ===
Notes is additional temporary data created by pipelets to be used in later pipelets in the same workflow, but not to be persisted in the storages. Notes can be either global or record specific (associated with a record Id). Record specific notes are copied on record splits and removed when the associated record is removed from the Blackboard. Each Note has a String name and Serialaizable value.
+
Notes are  additional temporary data created by pipelets to be used in later pipelets in the same workflow, but not to be persisted in the storages. Notes can be either global or record specific (associated with a record ID). Record specific notes are copied on record splits and removed when the associated record is removed from the blackboard. Each Note has a String name and Serialaizable value.
 
There are following methods for working with Notes:
 
There are following methods for working with Notes:
* boolean hasGlobalNote(name);
+
* <tt>boolean hasGlobalNote(name);</tt>
* Serializable getGlobalNote(name);
+
* <tt>Serializable getGlobalNote(name);</tt>
* setGlobalNote(name, value);
+
* <tt>setGlobalNote(name, value);</tt>
* boolean hasRecordNote(id, name);
+
* <tt>boolean hasRecordNote(id, name);</tt>
* getRecordNote(id, name);
+
* <tt>getRecordNote(id, name);</tt>
* setRecordNote(id, name, value);
+
* <tt>setRecordNote(id, name, value);</tt>
  
 
[[Category:SMILA]]
 
[[Category:SMILA]]
 
=== Usage of Path with Blackboard methods ===
 
Some methods of Blackboard accept Path as an argument, for example ''getAttributeNames(Id, Path)''. Path represents the attribute path in the Record. String format of Path looks like ''attributeName1[index1]/attributeName2[index2]/...''. The specification of index is optional and defaults to 0. Index can refer to a literal or a sub-object that depends on methods getting the argument.
 
 
Consider the following example Record structure:
 
 
<source lang="xml">
 
<Record>
 
  <A n="AccessTreeExpanded">
 
    <O>
 
      <A n="account">
 
        <O>
 
          <A n="sub">
 
            <O>
 
              <A n="sid">
 
                <L>
 
                  <V>Value1</V>
 
                </L>
 
                <L>
 
                  <V>Value2</V>
 
                </L>
 
              </A>
 
            </O>
 
            <O>
 
              <A n="sid">
 
                <L>
 
                  <V>Value3</V>
 
                </L>
 
              </A>
 
            </O>
 
        </A>
 
        </O>
 
      </A>
 
    </O>
 
  </A>
 
</Record>
 
</source>
 
 
The path to access first MObject (<O>) of the ''sub'' attribute is "''AccessTreeExpanded[0]/account[0]/sub[0]/''". Index in each step means the number of MObject inside the attribute. That is, to access second MObject of the ''sub'' attribute the path will be "''AccessTreeExpanded[0]/account[0]/sub[1]/''".
 
 
There are some cases when index of last step has a different meaning:
 
 
- in the ''getLiteral(Id, Path)'' method the index of last step means the number of literal inside the attribute. That is, path for accessing literal from ''sid'' attribute of second ''sub'' MObject (literal with value "Value3") will be "''AccessTreeExpanded[0]/account[0]/sub[1]/sid[0]''" and path for accessing second literal of ''sid'' attribute of first ''sub'' MObject (literal with value "Value2") will be: "'' AccessTreeExpanded[0]/account[0]/sub[0]/sid[1]''".
 
 
- in the ''getLiterals(Id, Path)'' method index of last step is irrelevant, that means this method will return all literals of the attribute found at the given path;
 
 
- in the ''setLiteral(Id, Path, Value)'' and ''addLiteral(Id, Path, Value)'' methods index of last step is irrelevant, that means that literal will be set or added to the attribute found on specified path
 
 
- in the methods that modify annotations to access root annotations of the record path should be null, "" (empty string), or empty Path
 

Revision as of 10:22, 20 April 2011

Contents

What is the blackboard

The purpose of the blackboard is the management of SMILA record data during processing in SMILA components (Connectivity, Workflow Processor). Complete record data is stored only on a blackboard which is not pushed through the workflow engine itself. The blackboard hides the handling of record persistence from the services. Clients should generally manipulate records using blackboard API methods in most cases, so records will be completely under the control of the blackboard.

Creation and lifecycle of blackboard

For blackboard creation we use a BlackboardFactory service running as a declarative service.

  • The factory can create blackboard instances which are either "transient" (pure in-memory implementation, not using any storages) or "persisting" (linked to binary storage and optionally to record storage). The client selects which kind of blackboard it wants to use. A persisting blackboard can only be created successfully, if at least a binary storage is known. Creation of transient blackboards is always possible.
  • For each "session" an own new blackboard instance is created that manages only those records worked on by this request. A session is for example:
    • a single task list execution of a QueueWorker router or listener (i.e. add/delete one record in Connectivity, or processing one input record from a queue message and manage all additional records created by the invoked workflows)
    • a single search request in the search service.
  • After the session the blackboard instance is released completely, thus freeing any memory resources automatically without interfering with other blackboard sessions.

BlackboardFactory interfaces

interface BlackboardFactory {
  /**
   * create a new non-persisting blackboard instance. 
   * This method must always return a valid empty blackboard instance.
   */
  Blackboard createTransientBlackboard();
 
  /**
   * create a blackboard able to persist records in storages
   * @throws BlackboardAccessException no persisting blackboard can be created, because 
   * not even a  binary storage service is available (record storage remains optional)
   */
  Blackboard createPersistingBlackboard() throws BlackboardAccessException;
}

Usage of blackboard

Note.png
Discussion
happens @ bug 336818


Record lifecycle on the blackboard

A record may be put on the blackboard using one of the following operations:

  • create(String id);
    Creates a new record with the given id. No data is loaded from persistence. If a record with this ID already exists in the storages it will be overwritten when the created record will be committed. E.g. used by Connectivity to initialize the record from incoming data.
  • load(String id);
    Loads record data for the given Id from persistence. Used by a client to indicate that it wants to process this record.
  • copyRecord(String id, String copyId);
    Creates a new record using copyId and copies all metadata from the record with 'id' to it. Attachments are not copied.
  • <tt>setRecord(Record);
    Puts the record on the blackboard, saves record attachments to BinStorage, and replaces actual record attachments values by null.
  • synchronizeRecord(Record);
    Assumes that a record with the same ID as the given one already exists on the blackboard or in the storage. Loads the record from the storage if needed and updates its properties with the properties of the given record.

A record may be removed from the blackboard using one of these operations:

  • commit(String id);
    Saves the record including its attachments to storages and removes the record from the blackboard.
  • invalidate(String id);
    Removes the record from the blackboard. If the record was created newly (not overwritten) on the blackboard it will be removed completely. The record is not removed from persistence, though.
  • removeRecord(String id);
    Removes a record completely from blackboard and all persistence layer.

Record access

  • Record getRecord(String id);
    Gets the record with its complete metadata. You should not rely on the record having all attachment values attached, only the names will be available. Use the blackboard attachments access methods to access attachments. If the record is not yet loaded in the blackboard, the PersistingBlackboard will return null, if it has a RecordStorage set and the record does not exist in record storage. If it doesn't have a RecordStorage a new record is created automatically. The TransientBlackboard creates a new record, too, if necessary. If you change the record metadata, you change the record stored on the blackboard, too, so you don't have to call setRecord() to write back the changes explicitly.
  • AnyMap getMetadata(String id);
    A shortcut for getRecord(id).getMetadata()
  • Record getRecord(String id, String filterName);
    Creates a copy of the record with only those metadata elements allowed by the named record filter. Changes done to the filtered metadata are not applied to the original record automatically.
  • Record filterRecord(Record record, String filterName);
    Applies a record filter known to the blackboard to a record which does not have to be loaded on the blackboard.

Attachments management

There are following methods for working with Record attachments in the blackboard:

  • setAttachment(id, name, byte[]);
  • setAttachmentFromStream(id, name, InputStream);
  • byte[] getAttachment(id, name);
  • InputStream getAttachmentAsStream(id, name);
  • boolean hasAttachment(id, name);

Attachments are not stored anywhere in the blackboard, they are saved to BinStorage directly and the actual attachment value in the corresponding record is replaced by null. It is highly recommended to use only Stream methods to manage attachments because loading the whole attachments in memory will cause great memory consumption and may cause application crashes.

Usage of blackboard notes

Notes are additional temporary data created by pipelets to be used in later pipelets in the same workflow, but not to be persisted in the storages. Notes can be either global or record specific (associated with a record ID). Record specific notes are copied on record splits and removed when the associated record is removed from the blackboard. Each Note has a String name and Serialaizable value. There are following methods for working with Notes:

  • boolean hasGlobalNote(name);
  • Serializable getGlobalNote(name);
  • setGlobalNote(name, value);
  • boolean hasRecordNote(id, name);
  • getRecordNote(id, name);
  • setRecordNote(id, name, value);