Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "COSMOS Design 197867"

('''Use Cases (required)''')
('''Use Cases (required)''')
Line 114: Line 114:
 
* Use case 2: Query for a Data Manager
 
* Use case 2: Query for a Data Manager
 
* Use case 3: Deregister a Data Manager
 
* Use case 3: Deregister a Data Manager
* Use case 4: Data Broker Initialization
+
* [[DataBrokerInitialization|Use case 4: Data Broker Initialization]]
  
  

Revision as of 14:07, 15 August 2007

Data Broker for COSMOS DC framework

Change History:

Name: Date: Revised Sections:

Jimmy Mohsin 7/25/2007 Initial version

Bill Muldoon 8/1/2007 Added content

Overview

The Data Broker is intended to be a simple component.

Workload Estimation (required)

 Rough workload estimate in person weeks: 
 Process	Sizing	Names of people doing the work
 Design	1	MR 2 development team
 Code	.5
 Test	1
 Documentation	.2
 Build and infrastructure	.5	Bill Muldoon
 Code review & other committer work (e.g. check-in, contribution tracking) 
 if this is to be contributed by someone who is not a committer in the component	0	
 N/A - will be done by committer
 Total	3.2

Terminologies/Acronyms (required)

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term:

Term Definition

 Data Broker	This is the component where all the web services that share data register themselves
 Data Store	This is a physical software artifact that stores data, e.g. Oracle, a File System, etc
 Data Manager	Largely corporate implementation component that implements the SOA API for data exchange
 Service Broker	This is the component where all the web services that share functional behavior register themselves
  • [mdw] Can we add these to the glossary?


Function:

  • [Assembly] Retrieving records from the data set
  • [Data Broker] Associating a data manager with a data set
  • Storing and maintaining a list of data sources, as references to where we get the data from
  • Introspect the keyset definition
  • Provide a cross reference to services. How does this differ from the first one?
  • Registration of new keysets & datasets
  • Client to chose which


  • Data source is typing information for a data set
  • Data set is the actual information, key set is the metadata


  • A Data Manager is: A query assembly for a specific type of data. This includes its ability to be exposed as a WSDM endpoint
  • A Data Manager can read many data sets, as long as they are all the same shape.
  • A Data Set is an observation of data
  • A Data Manager can deal with multiple instances of the same kind of data sets.
  • Most of the function of the data manager is done as of today.
  • Need to add the ability to support multiple data sets
  • We think the Data Broker and the Provider are the same


  • [Bill] For comparison purposes, the current COSMOS Provider Registry schema:

ProviderRegistry2.jpg


Stuff to do

  • Jimmy/Mark: Get a picture that we can all agree on
  • Jimmy/Mark: Clearly define the components that are in the picture
  • Jimmy/Mark: Distill & cleanup this list of stuff and move to data broker what belongs there
  • Joel/Hubert/Martin: Identify what needs to be reconciled in the code, e.g. packages, et..
  • Joel/Martin: Publish the Data Manager Capability (WSDL/RMD/Schema)
    • see org.eclipse.cosmos.dc.remote.wsdm (this is the pojo version)

Purpose (required)

This design document addresses COSMOS Bugzilla enhancement request 197867.

The Data Broker maintains a catalog of information about Data Managers according to a classification type.

The Data Managers can register and deregister themselves with the Data Broker.

The Client applications can query the Data Broker to locate the Data Managers.

Requirements (required)

The “Data Broker” does the following:

1. Processes registration and unregistration requests from Data Managers

2. Processes queries from the client to locate the Data Managers

Use Cases (required)

<Include a set of common use cases that your feature will introduce or impact>

Need to expand more details on these use cases, e.g. input/output/desired results


Need more definition on the following:

  • Use case 4: CMDBf considerations i7
  • Use case 5: Event handling i7
  • Use case 6: TTL for the Broker itself i7

Graphical Layout (optional)

Not applicable.


Class Diagram and Implementation Details

The Data Broker is implemented as a COSMOS DC query assembly with the following components:

1. Query Assembly definition. Refer to org.eclipse.cosmos.dc.sample.configurations\META-INF\cosmos\DataBroker.xml

 <?xml version="1.0" encoding="UTF-8"?>
 <context xmlns="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:cosmos="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:sample="http://www.eclipse.org/xmlns/sample/1.0"
 	cosmos:name="DataBroker" cosmos:direction="out">
 	<query cosmos:factory="org.eclipse.cosmos.dc.sample.components.query.DataBroker" cosmos:optimizable="true">
 		<sample:binding/>
 	</query>    
 </context>

2. Implementation Class, which is referenced by the Query Assembly definition. The implementation class maintains the persistent store. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\query\DataBroker.java

3. Capability file, which defines the API and interface of the implementation class (see the next section). Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\capabilitiy\DataBrokerCapability.java

4. The persistent store is implemented as a new table in the COSMOS DC runtime derby database. Refer to org.eclipse.cosmos.dc.local.registry\persistence_setup\RegistrySchema.sql

 CREATE TABLE COSMOS.DATA_MANAGER (  
       NAME VARCHAR(50) NOT NULL ,
       ASSEMBLY VARCHAR(50) NOT NULL ,
     	CLASSIFICATION VARCHAR(50),
     	HOSTNAME VARCHAR(25) NOT NULL ,
     	RUNTIMEPORT VARCHAR(10) NOT NULL );
     
 ALTER TABLE COSMOS.DATA_MANAGER
   ADD CONSTRAINT COSMOS.DATA_MANAGER_PK Primary Key (
     ASSEMBLY, HOSTNAME, RUNTIMEPORT );


The Persistent Store is accessed using an ibatis sql map. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\persistence\sql\DataManager.xml

 <?xml version="1.0" encoding="UTF-8" ?>
 <!DOCTYPE sqlMap PUBLIC "-//ibatis.apache.org//DTD SQL Map 2.0//EN"
   "http://ibatis.apache.org/dtd/sql-map-2.dtd">
 <sqlMap namespace="StatisticalDataset">
   <typeAlias alias="DataManager" type="org.eclipse.cosmos.dc.sample.persistence.impl.DataManagerImpl"/>
   <statement id="getDataManagers" resultClass="DataManager" parameterClass="String">
     select
       NAME as name,
       ASSEMBLY as assembly,
       CLASSIFICATION as classification,
       HOSTNAME as hostname,
       RUNTIMEPORT as runtimeport
     FROM COSMOS.DATA_MANAGER
     WHERE CLASSIFICATION = #classification#
   </statement>
   <statement id="getAllDataManagers" resultClass="DataManager">
     select
       NAME as name,
       ASSEMBLY as assembly,
       CLASSIFICATION as classification,
       HOSTNAME as hostname,
       RUNTIMEPORT as runtimeport
     FROM COSMOS.DATA_MANAGER
   </statement>
   <insert id="addDataManager" parameterClass="DataManager">
     INSERT INTO COSMOS.DATA_MANAGER (NAME, ASSEMBLY, CLASSIFICATION, HOSTNAME, RUNTIMEPORT) 
     VALUES(#name#, #assembly#, #classification#, #hostname#, #runtimeport#)
   </insert>
   <delete id="deleteDataManager" parameterClass="String">
     DELETE from COSMOS.DATA_MANAGER 
       WHERE NAME = #name#
   </delete>
 </sqlMap>

Command extensions for the data broker.

Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\ConsoleExtension.java

  osgi> broker
  Usage: 
      broker query <classification>
      broker register DataManagerName assembly.xml classification hostname hostPort
      broker deregister DataManagerName



Data Broker Client API.

Refer to the Client API document for details.

[comment] Please move broker APIs here from the design doc of enhancement 197870 so we can close this design independently of enhancement 197870.

Class Diagrams (optional)

<This section is only required for features that are introducing APIs that will be leveraged by other components>

The Data Broker exposes this API through its capability definition:

         // Get the Data Managers
 	public IDataQueryResult getDataManagers( String classification ) throws Exception;
 	// register the Data Manager	
 	public boolean registerDataManager( String name, String assembly, String classification, String hostname, String runtimeport ) throws Exception;
 	// deregister the Data Manager
 	public boolean deregisterDataManager( String name) throws Exception	


Extension Points (optional)

<This section should only be included if new extension points are being introduced>


<Include any additional topics that will help in completing the implementation>

<Content related to the topic above>



Test Coverage (required)

<Include a description of the unit, functional, and system test cases that will be required to test the overall quality of the feature>

The following tests are performed by org.eclipse.cosmos.dc.tests\src\org\eclipse\cosmos\dc\tests\client\DataBrokerTests.java

1. Data Broker Client interface

2. Registration test

3. Query test

4. Deregistration test


Task Breakdown (required)

<Includes the individual tasks that need to be completed to meet the overall objective of the enhancement. A PERT chart is included to indicate the dependency of each task.>

1. Query Assembly

2. Interface class

3. Implementation class

4. Derby database and iBatis sql map

5. Command extensions

6. Test application


Open Issues/Questions

  • How does the query work? Need better example of how to query for data
  • Need example of how to map existing data to SDMX structure
  • Have we agreed that SDMX is the default model we support?
    • What about other models, e.g. SQL?
    • Does the user NEED to understand SDMX to use this?
  • How does the design support other things that SDMX
  • How do we reconcile the CMDBf query structure?
  • Using operational status / life cycle / TTL (time to live) for appropriate components - i7 item

File:Dcframework2.zip


Back to the top