Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "COSMOS Design 197867"

(→‎Data Broker Use Cases)
 
(44 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
== '''Data Broker for COSMOS DC framework''' ==
 
== '''Data Broker for COSMOS DC framework''' ==
 +
 +
This design document addresses COSMOS Bugzilla enhancement request [https://bugs.eclipse.org/bugs/show_bug.cgi?id=197867 197867].
  
  
Line 11: Line 13:
 
Bill Muldoon 8/1/2007 Added content
 
Bill Muldoon 8/1/2007 Added content
  
−
== Overview ==
+
Bill Muldoon 8/17/2007 Updated content
−
The Data Broker is intended to be a simple component. 
+
  
 +
Jimmy Mohsin 8/17/2007 Please see my comments on the Talk page [http://wiki.eclipse.org/Talk:COSMOS_Design_197867 Talk for 197867]
  
−
== ''' Workload Estimation (required)''' ==
+
Hubert Leung    8/21/2007      Updated broker design as discussed in meeting on Aug 20: http://wiki.eclipse.org/COSMOSBrokerSpecification.  Need to consolidate the new design with this deisgn doc.
  
 +
Bill Muldoon 8/22/2007 Consolidated and reorganized the design
  
 +
Bill Muldoon 8/22/2007 Updated the schema diagram to migrate the classifications into a separate table so that a data manager can register many classification keywords
  
 +
== '''Overview''' ==
  
−
  Rough workload estimate in person weeks:  
+
The Broker is the component in the COSMOS data collection framework that holds a registry of Data Managers (ie: Management Data Repositories (MDRs)). The registry stores the addresses of the Data Managers in the form of end point references, indexed according to the data classification and the data source types of the data. Each Data Manager provides the data classification and data source type information upon registration with the Broker and maintains the relevant registry data over its life cycle. The Broker provides interfaces for clients to perform queries according to the classifications and data source types available in the registry. The result of the queries is the endpoint references of the the Data Managers which match the classifications and/or data source types of the query selection. 
−
  Process Sizing Names of people doing the work
+
−
  Design 1 MR 2 development team
+
The “Broker” component is a logically centralized component in the COSMOS data collection framework. In practice, there can be multiple brokers, each responsible for a different group of management resources, for example, the distinction between data brokers and service brokers. However, the role and responsibilities of different kinds of brokers are the same.
−
  Code .5
+
−
  Test 1
+
In the case where there are multiple Brokers, there is a higher level lookup registry (the Management Domain) to find the address of the Brokers. The Management Domain can be considered a "Broker of Brokers".
−
  Documentation .2
+
 
−
  Build and infrastructure .5 Bill Muldoon
+
 
−
  Code review & other committer work (e.g. check-in, contribution tracking)
+
== ''' Implementation Stages and Corporate Use Cases ''' ==
−
  if this is to be contributed by someone who is not a committer in the component 0
+
 
−
  N/A - will be done by committer
+
The Data Broker will evolve over time to accommodate the needs of the corporate use cases as follows:
−
  Total 3.2
+
 
 +
Version 1 will satisfy the initial CA use cases.
 +
 +
Version 2 will support the data source types requirements of the IBM use cases.
 +
 +
Version 3 will support the CMDBf use cases from CA and IBM.
 +
 
 +
Note that Compuware has no use cases which require the Broker at this time.
 +
  
−
== ''' Terminologies/Acronyms (required)''' ==
+
== ''' Terminologies/Acronyms ''' ==
  
  
Line 41: Line 54:
 
   Data Broker This is the component where all the web services that share data register themselves
 
   Data Broker This is the component where all the web services that share data register themselves
 
   Data Store This is a physical software artifact that stores data, e.g. Oracle, a File System, etc
 
   Data Store This is a physical software artifact that stores data, e.g. Oracle, a File System, etc
−
   Data Manager Largely corporate implementation component that implements the SOA API for data exchange
+
   Data Manager Largely corporate implementation component that implements the SOA API for data exchange. A Data Manager can be an MDR.
 
   Service Broker This is the component where all the web services that share functional behavior register themselves
 
   Service Broker This is the component where all the web services that share functional behavior register themselves
 +
  Management Domain    A "Broker of Brokers"
  
−
* [mdw] Can we add these to the glossary?
 
  
 +
==Data Broker Use Cases==
 +
''' [[DataBrokerInitialization|Use Case 1: Data Broker Initialization]] '''
 +
* ''' [[DataBrokerInitialization|Page for Data Broker Initialization]] '''
  
−
Function:
+
This is how the Data Broker registers with the Management Domain
−
* [Assembly] Retrieving records from the data set
+
----
−
* [Data Broker] Associating a data manager with a data set
+
'''Use Case 2. Data Manager (DM) initialization'''
−
* Storing and maintaining a list of data sources, as references to where we get the data from
+
   
−
* Introspect the keyset definition
+
NOTE: Should be moved to the data manager design page
−
* Provide a cross reference to services.  How does this differ from the first one?
+
−
* Registration of new keysets & datasets
+
−
* Client to chose which
+
  
  
−
* Data source is typing information for a data set
 
−
* Data set is the actual information, key set is the metadata
 
  
 +
DM contacts the Broker via well-known (pre-configured) EPR. DM may also get the Broker address from the Management Domain.
 +
 +
DM invokes the "registration" operation of the Broker, providing the following information:
 +
 +
* EPR for the DM
 +
* name of the DM
 +
* classification of the DM
 +
* dialect of the DM
 +
* optional data source type identifiers
 +
 +
Broker populates tables in registry database with the information about the DM
 +
 +
Broker changes the state to online and updates the timestamp
 +
 +
If the DM is already registered with the Broker, then the DM invokes the "ping" operation, which updates the timestamp in the Broker registry
 +
 +
----
 +
'''3. Data Manager shutdown'''
 +
 +
NOTE: Should be moved to the data manager design page
  
−
* A Data Manager is:  A query assembly for a specific type of data.  This includes its ability to be exposed as a WSDM endpoint
+
Data Manager invokes the deregister operation of the Broker, providing the DM name as a parameter
−
* A Data Manager can read many data sets, as long as they are all the same shape.
+
−
* A Data Set is an observation of data
+
Broker removes the registry entry for the Data Manager
−
* A Data Manager can deal with multiple instances of the same kind of data sets.
+
----
−
* Most of the function of the data manager is done as of today.
+
''' [[QueryForADataManager|Use case 4: Query for a Data Manager]] '''
−
* Need to add the ability to support multiple data sets
+
* ''' [[QueryForADataManager|Page for: Query for a Data Manager]] '''
−
* We think the Data Broker and the Provider are the same
+
----
 +
5. Deregister a data manager
  
 +
== ''' Data Broker External Interfaces ''' ==
 +
 +
 +
'''Data Broker Capabilities'''
 +
 +
The Data Broker exposes this capability definition:
  
−
* [Bill] For comparison purposes, the current COSMOS Provider Registry schema:
+
public interface DataBrokerCapability {
 +
 +
/**
 +
* Get the Data Managers
 +
*
 +
* @param String classification
 +
* @return IDataQueryResult containing an XML Element object
 +
*/
 +
@ManagedOperation
 +
public IDataQueryResult getDataManagers( String classification ) throws Exception;
 +
 +
/**
 +
* register the Data Manager
 +
*
 +
* @param String endpointAddress
 +
* @param String endpointResourceId
 +
* @param String dataManagerName
 +
* @param String classification
 +
* @param String dialect
 +
*/
 +
@ManagedOperation
 +
public boolean registerDataManager( String endpointAddress, String endpointResourceId,
 +
                            String dataManagerName, String  classification, String dialect ) throws Exception
 +
 +
/**
 +
* deregister the Data Manager
 +
*
 +
* @param String dataManagerName
 +
*/
 +
@ManagedOperation
 +
public boolean deregisterDataManager( String dataManagerName) throws Exception;
 +
 +
/**
 +
* ping the Data Broker
 +
*
 +
* @param String dataManagerName
 +
*/
 +
@ManagedOperation
 +
public boolean pingDataBroker( String dataManagerName) throws Exception;
 +
  
−
[[Image:ProviderRegistry2.jpg]]
+
'''Data Broker WSDL'''
  
 +
The DC runtime converts the capability into a WSDM Endpoint with operations that correspond to the methods. For example, the WSDL contains:
  
  
−
=== Stuff to do ===
+
  <wsdl:operation name="getDataManagers">
−
* Jimmy/Mark: Get a picture that we can all agree on
+
    <wsdl:input name="getDataManagersRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/getDataManagers"
−
* Jimmy/Mark: Clearly define the components that are in the picture
+
                message="dyn:getDataManagersRequest"/>
−
* Jimmy/Mark: Distill & cleanup this list of stuff and move to data broker what belongs there
+
    <wsdl:output name="getDataManagersResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/getDataManagersResponse"
−
* Joel/Hubert/Martin: Identify what needs to be reconciled in the code, e.g. packages, et..
+
                message="dyn:getDataManagersResponse"/>
−
* Joel/Martin: Publish the Data Manager Capability (WSDL/RMD/Schema)
+
      <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
−
** see org.eclipse.cosmos.dc.remote.wsdm (this is the pojo version)
+
      <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
 +
    </wsdl:operation>
 +
  <wsdl:operation name="registerDataManager">
 +
    <wsdl:input name="registerDataManagerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/registerDataManager"
 +
                message="dyn:registerDataManagerRequest"/>
 +
    <wsdl:output name="registerDataManagerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/registerDataManagerResponse"
 +
                message="dyn:registerDataManagerResponse"/>
 +
      <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
 +
      <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
 +
    </wsdl:operation>
 +
  <wsdl:operation name="deregisterDataManager">
 +
    <wsdl:input name="deregisterDataManagerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/deregisterDataManager"
 +
                message="dyn:deregisterDataManagerRequest"/>
 +
    <wsdl:output name="deregisterDataManagerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/deregisterDataManagerResponse"
 +
                message="dyn:deregisterDataManagerResponse"/>
 +
      <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
 +
      <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
 +
    </wsdl:operation>
 +
    <wsdl:operation name="pingDataBroker">
 +
      <wsdl:input name="pingDataBrokerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/pingDataBroker"
 +
                  message="dyn:pingDataBrokerRequest"/>
 +
      <wsdl:output name="pingDataBrokerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/pingDataBrokerResponse"
 +
                  message="dyn:pingDataBrokerResponse"/>
 +
      <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
 +
      <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
 +
    </wsdl:operation>
  
−
== ''' Purpose (required)''' ==
 
  
 +
Client applications can invoke the Broker operations using the WSDM Endpoint operations.
  
−
This design document addresses COSMOS Bugzilla enhancement request [https://bugs.eclipse.org/bugs/show_bug.cgi?id=197867 197867].
 
  
−
The Data Broker maintains a catalog of information about Data Managers according to a classification type.
+
'''Data Broker Client API'''
  
−
The Data Managers can register and deregister themselves with the Data Broker.  
+
Alternatively, client application can use the Broker Client API to invoke the Broker operations. The Broker Client API contains a Java class that provides a convenience mechanism to invoke the Broker WSDM endpoint operations.
 +
 +
Broker Client API class:
  
−
The Client applications can query the Data Broker to locate the Data Managers.
 
  
−
== ''' Requirements (required)''' ==
+
public class DataBrokerClient extends WsResourceClient{
 +
 +
/**
 +
* get the Data Managers
 +
*
 +
* @param String classification
 +
*/
 +
 +
public Element getDataManagers( String classification ) throws Exception {
 +
 
 +
 +
/**
 +
* register the Data Manager
 +
*
 +
* @param String endpointAddress
 +
* @param String endpointResourceId
 +
* @param String dataManagerName
 +
* @param String classification
 +
* @param String dialect
 +
*/
 +
 +
public Element registerDataManager( String endpointAddress, String endpointResourceId,
 +
                  String dataManagerName, String classification, String dialect ) throws Exception
 +
 +
/**
 +
* deregister the Data Manager
 +
*
 +
* @param String name
 +
*/
 +
 +
public Element deregisterDataManager( String name) throws Exception
 +
 +
 +
/**
 +
* ping the Data Broker
 +
*
 +
* @param String name
 +
*/
 +
 +
public Element pingDataBroker( String name) throws Exception
 +
  
  
 +
sample code for the Broker Client API:
  
−
The “Data Broker” does the following:
+
//
 +
// get the Data Broker client from its endpointReference
 +
//
 +
DataBrokerClient brokerClient = new DataBrokerClient( brokerEPR );
 +
 +
//
 +
// get the Data Managers from the Data Broker for the "Performance" classification
 +
//
 +
Element result = brokerClient.getDataManagers( "Performance" );
 +
 +
System.out.println(XmlUtils.toString(result));
  
−
1. Processes registration and unregistration requests from Data Managers
+
Sample Output (containing the EPRs):
  
−
2. Processes queries from the client to locate the Data Managers
+
<?xml version="1.0" encoding="UTF-8"?>
 +
<DataManagers xmlns="http://cosmos.eclipse.org/dc/databroker">
 +
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="NSM">
 +
        <Classification>Performance</Classification>
 +
        <Dialect>SDMX</Dialect>
 +
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
 +
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.nsmpdg.NSMDataManager</Address>
 +
            <ReferenceParameters>
 +
                <ResourceId>NSMIdentifierValue</ResourceId>
 +
            </ReferenceParameters>
 +
        </EndpointReference>
 +
    </DataManager>
 +
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="eHealth">
 +
        <Classification>Performance</Classification>
 +
        <Dialect>SDMX</Dialect>
 +
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
 +
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.ehealth.eHealthDataManager</Address>
 +
            <ReferenceParameters>
 +
                <ResourceId>eHealthIdentifierValue</ResourceId>
 +
            </ReferenceParameters>
 +
        </EndpointReference>
 +
    </DataManager>
 +
  </DataManagers>
  
−
== ''' Use Cases (required)''' ==
+
== ''' Command extensions for the Data Broker.''' ==
  
 +
 +
Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\ConsoleExtension.java
  
−
<Include a set of common use cases that your feature will introduce or impact>
+
osgi> broker
−
 
+
Usage: broker query <classification>
−
Need to expand more details on these use cases, e.g. input/output/desired results
+
        broker register EPAddress EPResourceID DataManagerName classification dialect
−
 
+
        broker deregister DataManagerName
−
* Use case 1: Register a Data Manager
+
−
 
+
−
* Use case 2: Query for a Data Manager
+
−
 
+
−
* Use case 3: Deregister a Data Manager
+
−
 
+
−
== ''' Graphical Layout (optional)''' ==
+
−
 
+
  
−
Not applicable.
 
  
  
−
== ''' Class Diagram and Implementation Details''' ==
+
== ''' Data Broker Implementation Details''' ==
  
  
Line 149: Line 318:
 
3. Capability file, which defines the API and interface of the implementation class (see the next section). Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\capabilitiy\DataBrokerCapability.java
 
3. Capability file, which defines the API and interface of the implementation class (see the next section). Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\capabilitiy\DataBrokerCapability.java
  
−
4. The persistent store is implemented as a new table in the COSMOS DC runtime derby database. Refer to  org.eclipse.cosmos.dc.local.registry\persistence_setup\RegistrySchema.sql
+
4. The registry is internal to the Broker. It is implemented in the COSMOS DC runtime derby database. Refer to  org.eclipse.cosmos.dc.local.registry\persistence_setup\RegistrySchema.sql
  
−
  CREATE TABLE COSMOS.DATA_MANAGER (
+
 
 +
CREATE TABLE COSMOS.DATA_MANAGER (  
 +
        ENDPOINTADDRESS VARCHAR(255) NOT NULL ,
 +
        ENDPOINTRESOURCEID VARCHAR(50) NOT NULL ,
 
         NAME VARCHAR(50) NOT NULL ,
 
         NAME VARCHAR(50) NOT NULL ,
−
         ASSEMBLY VARCHAR(50) NOT NULL ,
+
         CLASSIFICATION VARCHAR(50) NOT NULL,
−
      CLASSIFICATION VARCHAR(50),
+
        DIALECT VARCHAR(50) NOT NULL,
−
      HOSTNAME VARCHAR(25) NOT NULL ,
+
        LASTMSGTIME TIMESTAMP NOT NULL );
−
      RUNTIMEPORT VARCHAR(10) NOT NULL );
+
 
        
 
        
−
  ALTER TABLE COSMOS.DATA_MANAGER
+
ALTER TABLE COSMOS.DATA_MANAGER
−
    ADD CONSTRAINT COSMOS.DATA_MANAGER_PK Primary Key (
+
  ADD CONSTRAINT COSMOS.DATA_MANAGER_PK Primary Key (
−
       ASSEMBLY, HOSTNAME, RUNTIMEPORT );
+
       ENDPOINTADDRESS, ENDPOINTRESOURCEID );
 +
 
 +
ALTER TABLE COSMOS.DATA_MANAGER
 +
  ADD CONSTRAINT COSMOS.DATA_MANAGER_UNQ_NM Unique (
 +
      NAME);
 +
 
 +
 
 +
Each Data Manager entry in the store contains the EPR (address and resource id), the name, the dialect, the classification keyword(s) and the timestamp of the last message received. The timestamp is updated automatically when the Data Manager is registered and also by the pingDataManager operation.
  
  
 
The Persistent Store is accessed using an ibatis sql map. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\persistence\sql\DataManager.xml  
 
The Persistent Store is accessed using an ibatis sql map. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\persistence\sql\DataManager.xml  
  
−
  <?xml version="1.0" encoding="UTF-8" ?>
+
<?xml version="1.0" encoding="UTF-8" ?>
−
  <!DOCTYPE sqlMap PUBLIC "-//ibatis.apache.org//DTD SQL Map 2.0//EN"
+
<!DOCTYPE sqlMap PUBLIC "-//ibatis.apache.org//DTD SQL Map 2.0//EN"
 
     "http://ibatis.apache.org/dtd/sql-map-2.dtd">
 
     "http://ibatis.apache.org/dtd/sql-map-2.dtd">
−
  <sqlMap namespace="StatisticalDataset">
+
<sqlMap namespace="StatisticalDataset">
−
    <typeAlias alias="DataManager" type="org.eclipse.cosmos.dc.sample.persistence.impl.DataManagerImpl"/>
+
  <typeAlias alias="DataManager" type="org.eclipse.cosmos.dc.sample.persistence.impl.DataManagerImpl"/>
−
    <statement id="getDataManagers" resultClass="DataManager" parameterClass="String">
+
  <statement id="getDataManagers" resultClass="DataManager" parameterClass="String">
−
      select
+
    select
−
        NAME as name,
+
      ENDPOINTADDRESS as endpointAddress,
−
        ASSEMBLY as assembly,
+
      ENDPOINTRESOURCEID as endpointResourceId,
−
        CLASSIFICATION as classification,
+
      NAME as name,
−
        HOSTNAME as hostname,
+
      CLASSIFICATION as classification,
−
        RUNTIMEPORT as runtimeport
+
      DIALECT as dialect,
−
      FROM COSMOS.DATA_MANAGER
+
      LASTMSGTIME as lastMsgTime
−
      WHERE CLASSIFICATION = #classification#
+
    FROM COSMOS.DATA_MANAGER
−
    </statement>
+
    WHERE CLASSIFICATION = #classification#
−
    <statement id="getAllDataManagers" resultClass="DataManager">
+
  </statement>
−
      select
+
  <statement id="getAllDataManagers" resultClass="DataManager">
−
        NAME as name,
+
    select
−
        ASSEMBLY as assembly,
+
      ENDPOINTADDRESS as endpointAddress,
−
        CLASSIFICATION as classification,
+
      ENDPOINTRESOURCEID as endpointResourceId,
−
        HOSTNAME as hostname,
+
      NAME as name,
−
        RUNTIMEPORT as runtimeport
+
      CLASSIFICATION as classification,
−
      FROM COSMOS.DATA_MANAGER
+
      DIALECT as dialect,
−
    </statement>
+
      LASTMSGTIME as lastMsgTime
−
    <insert id="addDataManager" parameterClass="DataManager">
+
    FROM COSMOS.DATA_MANAGER
−
      INSERT INTO COSMOS.DATA_MANAGER (NAME, ASSEMBLY, CLASSIFICATION, HOSTNAME, RUNTIMEPORT)  
+
  </statement>
−
      VALUES(#name#, #assembly#, #classification#, #hostname#, #runtimeport#)
+
  <insert id="addDataManager" parameterClass="DataManager">
−
    </insert>
+
    INSERT INTO COSMOS.DATA_MANAGER (ENDPOINTADDRESS, ENDPOINTRESOURCEID, NAME, CLASSIFICATION, DIALECT, LASTMSGTIME)  
−
    <delete id="deleteDataManager" parameterClass="String">
+
    VALUES(#endpointAddress#, #endpointResourceId#, #name#, #classification#, #dialect#, #lastMsgTime#)
−
      DELETE from COSMOS.DATA_MANAGER  
+
  </insert>
 +
  <delete id="deleteDataManager" parameterClass="String">
 +
    DELETE from COSMOS.DATA_MANAGER  
 
         WHERE NAME = #name#
 
         WHERE NAME = #name#
−
    </delete>
+
  </delete>
−
   </sqlMap>
+
   <update id="pingDataManager" parameterClass="DataManager">
−
 
+
    UPDATE COSMOS.DATA_MANAGER
−
== ''' Command extensions for the data broker.''' ==
+
        SET LASTMSGTIME = #lastMsgTime#
−
 
+
            WHERE NAME = #name#
−
+
   </update>
−
 
+
</sqlMap>
−
Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\ConsoleExtension.java
+
−
 
+
−
  osgi> broker
+
−
  Usage:
+
−
      broker query <classification>
+
−
      broker register DataManagerName assembly.xml classification hostname hostPort
+
−
      broker deregister DataManagerName
+
−
 
+
−
 
+
−
 
+
−
 
+
−
== ''' Data Broker Client API.''' ==
+
−
 
+
−
+
−
Refer to the Client API document for details.
+
−
 
+
−
 
+
−
 
+
−
 
+
−
== ''' Class Diagrams (optional)''' ==
+
−
 
+
−
 
+
−
 
+
−
<This section is only required for features that are introducing APIs that will be leveraged by other components>
+
−
 
+
−
The Data Broker exposes this API through its capability definition:
+
−
 
+
−
  ''        // Get the Data Managers''
+
−
  '' public IDataQueryResult getDataManagers( String classification ) throws Exception;''
+
−
  '' // register the Data Manager''
+
−
  '' public boolean registerDataManager( String name, String assembly, String classification, String hostname, String runtimeport ) throws Exception;''
+
−
   '' // deregister the Data Manager''
+
−
  '' public boolean deregisterDataManager( String name) throws Exception ''
+
−
 
+
−
 
+
−
 
+
−
== ''' Extension Points (optional)''' ==
+
−
 
+
−
 
+
−
<This section should only be included if new extension points are being introduced>
+
−
 
+
−
 
+
−
 
+
−
<Include any additional topics that will help in completing the implementation>
+
−
 
+
−
<Content related to the topic above>
+
−
 
+
−
 
+
−
 
+
−
 
+
−
== ''' Test Coverage (required)''' ==
+
−
 
+
−
 
+
−
<Include a description of the unit, functional, and system test cases that will be required to test the overall quality of the feature>
+
−
 
+
−
The following tests are performed by org.eclipse.cosmos.dc.tests\src\org\eclipse\cosmos\dc\tests\client\DataBrokerTests.java
+
−
 
+
−
1. Data Broker Client interface
+
−
 
+
−
2. Registration test
+
−
 
+
−
3. Query test
+
−
 
+
−
4. Deregistration test
+
−
 
+
−
 
+
−
 
+
−
== ''' Task Breakdown (required)''' ==
+
−
 
+
−
 
+
−
<Includes the individual tasks that need to be completed to meet the overall objective of the enhancement.  A PERT chart is included to indicate the dependency of each task.>
+
  
−
1. Query Assembly
 
  
−
2. Interface class
+
Version 2 of the Broker includes the Data Source Type information. The schema requires 2 additional tables to support the many to many relationship between the Data Managers and the Data Source Types as follows:
  
−
3. Implementation class
+
[[Image:BrokerSchema2.jpg]]
  
−
4. Derby database and iBatis sql map
 
  
−
5. Command extensions
 
  
−
6. Test application
 
  
  
−
== Open Issues/Questions ==
 
−
* How does the query work?  Need better example of how to query for data
 
−
* Need example of how to map existing data to SDMX structure
 
−
* Have we agreed that SDMX is the default model we support?
 
−
** What about other models, e.g. SQL?
 
−
** Does the user NEED to understand SDMX to use this?
 
−
* How does the design support other things that SDMX
 
−
* How do we reconcile the CMDBf query structure?
 
  
−
[[Image:Dcframework2.zip]]
 
  
 
----
 
----
 
[[Category:COSMOS_Bugzilla_Designs]]
 
[[Category:COSMOS_Bugzilla_Designs]]

Latest revision as of 11:40, 28 August 2007

Data Broker for COSMOS DC framework

This design document addresses COSMOS Bugzilla enhancement request 197867.


Change History:

Name: Date: Revised Sections:

Jimmy Mohsin 7/25/2007 Initial version

Bill Muldoon 8/1/2007 Added content

Bill Muldoon 8/17/2007 Updated content

Jimmy Mohsin 8/17/2007 Please see my comments on the Talk page Talk for 197867

Hubert Leung 8/21/2007 Updated broker design as discussed in meeting on Aug 20: http://wiki.eclipse.org/COSMOSBrokerSpecification. Need to consolidate the new design with this deisgn doc.

Bill Muldoon 8/22/2007 Consolidated and reorganized the design

Bill Muldoon 8/22/2007 Updated the schema diagram to migrate the classifications into a separate table so that a data manager can register many classification keywords

Overview

The Broker is the component in the COSMOS data collection framework that holds a registry of Data Managers (ie: Management Data Repositories (MDRs)). The registry stores the addresses of the Data Managers in the form of end point references, indexed according to the data classification and the data source types of the data. Each Data Manager provides the data classification and data source type information upon registration with the Broker and maintains the relevant registry data over its life cycle. The Broker provides interfaces for clients to perform queries according to the classifications and data source types available in the registry. The result of the queries is the endpoint references of the the Data Managers which match the classifications and/or data source types of the query selection.

The “Broker” component is a logically centralized component in the COSMOS data collection framework. In practice, there can be multiple brokers, each responsible for a different group of management resources, for example, the distinction between data brokers and service brokers. However, the role and responsibilities of different kinds of brokers are the same.

In the case where there are multiple Brokers, there is a higher level lookup registry (the Management Domain) to find the address of the Brokers. The Management Domain can be considered a "Broker of Brokers".


Implementation Stages and Corporate Use Cases

The Data Broker will evolve over time to accommodate the needs of the corporate use cases as follows:

Version 1 will satisfy the initial CA use cases.

Version 2 will support the data source types requirements of the IBM use cases.

Version 3 will support the CMDBf use cases from CA and IBM.

Note that Compuware has no use cases which require the Broker at this time.


Terminologies/Acronyms

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term:

Term Definition 

 Data Broker	This is the component where all the web services that share data register themselves
 Data Store	This is a physical software artifact that stores data, e.g. Oracle, a File System, etc
 Data Manager	Largely corporate implementation component that implements the SOA API for data exchange. A Data Manager can be an MDR.
 Service Broker	This is the component where all the web services that share functional behavior register themselves
 Management Domain    A "Broker of Brokers"


Data Broker Use Cases

Use Case 1: Data Broker Initialization

This is how the Data Broker registers with the Management Domain

Use Case 2. Data Manager (DM) initialization

NOTE: Should be moved to the data manager design page


DM contacts the Broker via well-known (pre-configured) EPR. DM may also get the Broker address from the Management Domain.

DM invokes the "registration" operation of the Broker, providing the following information:

  • EPR for the DM
  • name of the DM
  • classification of the DM
  • dialect of the DM
  • optional data source type identifiers

Broker populates tables in registry database with the information about the DM

Broker changes the state to online and updates the timestamp

If the DM is already registered with the Broker, then the DM invokes the "ping" operation, which updates the timestamp in the Broker registry

3. Data Manager shutdown

NOTE: Should be moved to the data manager design page

Data Manager invokes the deregister operation of the Broker, providing the DM name as a parameter

Broker removes the registry entry for the Data Manager

Use case 4: Query for a Data Manager

5. Deregister a data manager

Data Broker External Interfaces

Data Broker Capabilities

The Data Broker exposes this capability definition: 

public interface DataBrokerCapability {
	
	/**
	 * Get the Data Managers
	 * 
	 * @param String classification
	 * @return IDataQueryResult containing an XML Element object
	 */
	@ManagedOperation
	public IDataQueryResult getDataManagers( String classification ) throws Exception;

	/**
	 * register the Data Manager
	 * 
	 * @param String endpointAddress
	 * @param String endpointResourceId
	 * @param String dataManagerName
	 * @param String classification
	 * @param String dialect
	 */
	@ManagedOperation
	public boolean registerDataManager( String endpointAddress, String endpointResourceId, 
                            String dataManagerName, String  classification, String dialect ) throws Exception

	/**
	 * deregister the Data Manager
	 * 
	 * @param String dataManagerName
	 */
	@ManagedOperation
	public boolean deregisterDataManager( String dataManagerName) throws Exception;

	/**
	 * ping the Data Broker
	 * 
	 * @param String dataManagerName
	 */
	@ManagedOperation
	public boolean pingDataBroker( String dataManagerName) throws Exception;

Data Broker WSDL

The DC runtime converts the capability into a WSDM Endpoint with operations that correspond to the methods. For example, the WSDL contains: 


 <wsdl:operation name="getDataManagers">
   <wsdl:input name="getDataManagersRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/getDataManagers"
               message="dyn:getDataManagersRequest"/>
   <wsdl:output name="getDataManagersResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/getDataManagersResponse"
                message="dyn:getDataManagersResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>
 <wsdl:operation name="registerDataManager">
   <wsdl:input name="registerDataManagerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/registerDataManager"
               message="dyn:registerDataManagerRequest"/>
   <wsdl:output name="registerDataManagerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/registerDataManagerResponse" 
                message="dyn:registerDataManagerResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>
 <wsdl:operation name="deregisterDataManager">
   <wsdl:input name="deregisterDataManagerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/deregisterDataManager" 
               message="dyn:deregisterDataManagerRequest"/>
   <wsdl:output name="deregisterDataManagerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/deregisterDataManagerResponse" 
                message="dyn:deregisterDataManagerResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>
   <wsdl:operation name="pingDataBroker">
     <wsdl:input name="pingDataBrokerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/pingDataBroker" 
                 message="dyn:pingDataBrokerRequest"/>
     <wsdl:output name="pingDataBrokerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/pingDataBrokerResponse" 
                  message="dyn:pingDataBrokerResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>


Client applications can invoke the Broker operations using the WSDM Endpoint operations.


Data Broker Client API

Alternatively, client application can use the Broker Client API to invoke the Broker operations. The Broker Client API contains a Java class that provides a convenience mechanism to invoke the Broker WSDM endpoint operations.

Broker Client API class: 


public class DataBrokerClient extends WsResourceClient{

	/** 
	 * get the Data Managers 
	 *
	 * @param String classification
	 */
	
	public Element getDataManagers( String classification ) throws Exception {
 
	
	/**
	 * register the Data Manager 
	 *
	 * @param String endpointAddress
	 * @param String endpointResourceId
	 * @param String dataManagerName
	 * @param String classification
	 * @param String dialect
	 */
	
	public Element registerDataManager( String endpointAddress, String endpointResourceId, 
                  String dataManagerName, String classification, String dialect ) throws Exception

	/**
	 * deregister the Data Manager 
	 *
	 * @param String name
	 */
	
	public Element deregisterDataManager( String name) throws Exception


	/**
	 * ping the Data Broker 
	 *
	 * @param String name
	 */
	
	public Element pingDataBroker( String name) throws Exception


sample code for the Broker Client API: 

			//
			// get the Data Broker client from its endpointReference
			//
			DataBrokerClient brokerClient = new DataBrokerClient( brokerEPR ); 
			
			//
			// get the Data Managers from the Data Broker for the "Performance" classification
			//
			Element result = brokerClient.getDataManagers( "Performance" );
					
			System.out.println(XmlUtils.toString(result));

Sample Output (containing the EPRs): 

<?xml version="1.0" encoding="UTF-8"?>
<DataManagers xmlns="http://cosmos.eclipse.org/dc/databroker">
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="NSM">
        <Classification>Performance</Classification>
        <Dialect>SDMX</Dialect>
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.nsmpdg.NSMDataManager</Address>
            <ReferenceParameters>
                <ResourceId>NSMIdentifierValue</ResourceId>
            </ReferenceParameters>
        </EndpointReference>
    </DataManager>
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="eHealth">
        <Classification>Performance</Classification>
        <Dialect>SDMX</Dialect>
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.ehealth.eHealthDataManager</Address>
            <ReferenceParameters>
                <ResourceId>eHealthIdentifierValue</ResourceId>
            </ReferenceParameters>
        </EndpointReference>
    </DataManager>
</DataManagers>

Command extensions for the Data Broker.

Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\ConsoleExtension.java 

osgi> broker
Usage: broker query <classification>
       broker register EPAddress EPResourceID DataManagerName classification dialect
       broker deregister DataManagerName


Data Broker Implementation Details

The Data Broker is implemented as a COSMOS DC query assembly with the following components:

1. Query Assembly definition. Refer to org.eclipse.cosmos.dc.sample.configurations\META-INF\cosmos\DataBroker.xml 

 <?xml version="1.0" encoding="UTF-8"?>
 <context xmlns="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:cosmos="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:sample="http://www.eclipse.org/xmlns/sample/1.0"
 	cosmos:name="DataBroker" cosmos:direction="out">
 	<query cosmos:factory="org.eclipse.cosmos.dc.sample.components.query.DataBroker" cosmos:optimizable="true">
 		<sample:binding/>
 	</query>    
 </context>

2. Implementation Class, which is referenced by the Query Assembly definition. The implementation class maintains the persistent store. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\query\DataBroker.java

3. Capability file, which defines the API and interface of the implementation class (see the next section). Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\capabilitiy\DataBrokerCapability.java

4. The registry is internal to the Broker. It is implemented in the COSMOS DC runtime derby database. Refer to org.eclipse.cosmos.dc.local.registry\persistence_setup\RegistrySchema.sql 


CREATE TABLE COSMOS.DATA_MANAGER ( 
       ENDPOINTADDRESS VARCHAR(255) NOT NULL , 
       ENDPOINTRESOURCEID VARCHAR(50) NOT NULL , 
       NAME VARCHAR(50) NOT NULL ,
       CLASSIFICATION VARCHAR(50) NOT NULL,
       DIALECT VARCHAR(50) NOT NULL,
       LASTMSGTIME TIMESTAMP NOT NULL );
     
ALTER TABLE COSMOS.DATA_MANAGER
  ADD CONSTRAINT COSMOS.DATA_MANAGER_PK Primary Key (
     ENDPOINTADDRESS, ENDPOINTRESOURCEID );
  
ALTER TABLE COSMOS.DATA_MANAGER
  ADD CONSTRAINT COSMOS.DATA_MANAGER_UNQ_NM Unique (
     NAME);


Each Data Manager entry in the store contains the EPR (address and resource id), the name, the dialect, the classification keyword(s) and the timestamp of the last message received. The timestamp is updated automatically when the Data Manager is registered and also by the pingDataManager operation.


The Persistent Store is accessed using an ibatis sql map. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\persistence\sql\DataManager.xml 

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE sqlMap PUBLIC "-//ibatis.apache.org//DTD SQL Map 2.0//EN"
   "http://ibatis.apache.org/dtd/sql-map-2.dtd">
<sqlMap namespace="StatisticalDataset">
 <typeAlias alias="DataManager" type="org.eclipse.cosmos.dc.sample.persistence.impl.DataManagerImpl"/>
 <statement id="getDataManagers" resultClass="DataManager" parameterClass="String">
   select
     ENDPOINTADDRESS as endpointAddress,
     ENDPOINTRESOURCEID as endpointResourceId,
     NAME as name,
     CLASSIFICATION as classification,
     DIALECT as dialect,
     LASTMSGTIME as lastMsgTime
   FROM COSMOS.DATA_MANAGER
   WHERE CLASSIFICATION = #classification#
 </statement>
 <statement id="getAllDataManagers" resultClass="DataManager">
   select
     ENDPOINTADDRESS as endpointAddress,
     ENDPOINTRESOURCEID as endpointResourceId,
     NAME as name,
     CLASSIFICATION as classification,
     DIALECT as dialect,
     LASTMSGTIME as lastMsgTime
   FROM COSMOS.DATA_MANAGER
 </statement>
 <insert id="addDataManager" parameterClass="DataManager">
   INSERT INTO COSMOS.DATA_MANAGER (ENDPOINTADDRESS, ENDPOINTRESOURCEID, NAME, CLASSIFICATION, DIALECT, LASTMSGTIME) 
   VALUES(#endpointAddress#, #endpointResourceId#, #name#, #classification#, #dialect#, #lastMsgTime#)
 </insert>
 <delete id="deleteDataManager" parameterClass="String">
   DELETE from COSMOS.DATA_MANAGER 
       WHERE NAME = #name#
 </delete>
 <update id="pingDataManager" parameterClass="DataManager">
   UPDATE COSMOS.DATA_MANAGER
       SET LASTMSGTIME = #lastMsgTime# 
           WHERE NAME = #name#
 </update>
</sqlMap>


Version 2 of the Broker includes the Data Source Type information. The schema requires 2 additional tables to support the many to many relationship between the Data Managers and the Data Source Types as follows:

BrokerSchema2.jpg




Retrieved from "https://wiki.eclipse.org/index.php?title=COSMOS_Design_197867&oldid=48356"

This page was last modified 11:40, 28 August 2007 by Mark Weitzel. Based on work by Bill Muldoon, Hubert Leung and Jimmy Mohsin and others.

Back to the top