COSMOSBrokerSpecification

Overview

The broker is the component in the COSMOS data collection framework that holds a registry of management data repositories (MDRs)*. The registry stores the addresses of the MDRs in the form of end point references, indexed according to the data classification and the data source type of the data. Each MDR provides the data classification and data source type information upon registration with the broker and maintains the relevant registry data over its life cycle. The broker provides interfaces for clients to make queries for the classifications and data source types available in the registry and for addresses of MDRs based on classifications and/or data source types. The interfaces are provided in the form of WSDM capabilities and Java interfaces.

The “broker” component is a logically centralized component in the COSMOS data collection framework. In practice, there can be multiple brokers, each responsible for a different group of management resources, for example, the distinction between data brokers and service brokers. However, the role and responsibilities of different kinds of brokers are the same. In the case where there are multiple brokers, there is a higher level lookup registry (the management domain) to find the address of the brokers.

• MDR is a term defined by the Configuration Management Database Federation specification and is available at the CMDBf.org

Broker Registry Indexes

The registry is a lookup table of the addresses of MDRs that have registered with the broker. The addresses are indexed with two pieces of information:

1. Data Type Classification:
   • Data type classification (or simply "classification") is a well-known identifier that represents data with a predefined structure and meaning. It is analogous to a Java class definition.
   • Each classification identifier must be backed by a definition that defines the actual data structure or attributes of the classifier. The definition can be in the form of XML schema, database schema, or SML document.
   • Examples:
     • Common Base Events (CBE): CBE is a standard of event data which has a predetermined structure.
     • Hardware asset data: The attributes of hardware asset data (such as CPU speed, machine model number, hard drive space, etc) can be defined in standards like CML.
     • The classification does not have to be a public standard. A classification can be defined for a specific system, as long as the definition is a common knowledge among all MDRs. The definition must be available globally with the COSMOS data collection system, and there is a way to introspect the data structure of the classification. For example, we can define "statistical data" as the tuple (HeapMemoryUsed, NonHeapMemoryUsed).
     • The definition of the classification may be represented by using the SML facet concept.
2. Data source type:
   • The data source type value specifies the type of hardware or software that generated the data.
   • The data source of CBE data can be a web server or a network router.
   • Data source type identifies the type of data source, but not the instance of the data source.

Notes:

• Data classified by the same data type classification can originate from different data source types. (For example, CBE event data can be generated by web applications and network routers.)
• SDMX data model is not used in the broker registry. The classification and data source type concepts can be mapped roughly to the key family and data source type of SDMX. We try to keep the semantics of separating data structure from the source of data.
• The broker registry design is not standards-based. However, the design and implementation should allow the registry to be evolved to support CMDBf query and registration, and may alos use an implementation of WS-RC to store the registry data.

Broker Registry Structure

The registry can be composed of 3 database tables:

1. Classification(cid, name)
2. DataSourceType(did, name)
3. MDR(epr, cid_fk, did_fk, state)

Example data

Classification

cid	name
1	CBE
2	SDMX
3	SML

DataSourceType

did	name
10	Tomcat Server
20	Cisco router
30	SML Repository

MDR (values in brackets are referenced values)

epr	cid_fk	mid_fk	state
EPR 1	1 (CBE)	10 (Tomcat Server)	online
EPR 2	1 (CBE)	20 (Cisco Router)	online
EPR 3	2 (SDMX)	10 (Tomcat Server)	online
EPR 4	3 (SML)	30 (SML Repository)	online
EPR 5	1 (CBE)	10 (Tomcat Server)	online
EPR 6	1 (CBE)	10 (Tomcat Server)	offline

cid_fk and mid_fk are foreign keys to cid and mid in Classification and DataSourceType tables respectively.

User Cases

Broker initialization

• Broker wakes up.
• Broker undergoes initialization: Loops through the rows in the MDR table, ping each MDR for their status. If an MDR is reachable, set its state to "online"; otherwise, set it to "offline".
• Broker waits for requests from client or MDR.

MDR registers with broker

• MDR contacts broker via well-known (pre-configured) EPR. MDR may also get broker address from management domain.
• MDR invokes the "registration" capability of the broker. Provide the following information during registration:
  • EPR for the MDR
  • a pair of classification and data source type identifiers for each data source instance it is managing.
• Broker populates tables in registry database with the information about the MDR.

Data manager deregisters with broker

• Data manager invokes the deregister capability of the broker, providing ID as a parameter.
• Broker updates removes the entry that corresponds to this data manager from the registry

Client queries broker for MDRs

• Client invokes an API of the broker, providing a classification keyword (e.g. hardware configuration)
• Broker retrieves EPRs of data managers from the registry that are under the desired category and returns the EPRs to the client
• Example queries:
  1. "Give me MDRs that provide daa from SML Repositories." Using the example data above, the broker will return EPR4.
  2. "Give me MDRs that provide CBE data for Tomcat Servers." Using the example data above, the broker will return EPR1 and EPR5. EPR6 is not included because it is offline.
  3. "Give me MDRs that provide data about Tomcat Servers." EPR1, EPR3 and EPR5 are returned. Since classification is not specified, EPRs that provide CBE or statistical data (SDMX) are returned.
  4. "Give me MDRs that provide CBE data." EPR1, EPR2 and EPR5 are returned. Since data source type is not specified, CBE can orginate from either Tomcat Server or Cisco Router.

<Joel comment> What about the case where my MDRs are organized around a business-oriented topology? This current simplification of datasource type/ keset type doesn't allow for any instance-oriented organization. Are you sure that's what you want?

<Hubert> This is a simple registry structure, and I expect it to have limitations. Joel, can you elaborate on the "business-oriented topology" usecase? We can find a way to support it. </Hubert>

</Joel comment>

Register new classification or data source type

The classification and data source type data in the broker registry is a global setting for the system that is using the COSMOS data collection framework. We can expect users with adminitrative authroity to set up this list of identifiers and make sure there are no duplicates among the identifiers that actually refer to the same thing. MDRs registration will exiting identifiers during registration. We need a broker API for registering new classification and data source type entries in the registry. We have not discussed authentication/security in the design yet, so let's assume anyone can invoke the API for now.

Query for existing classifications and data source types available at the broker

This use case is to get existing classifications and data source types registered at the broker registry. Clients require this information to form queries and MDRs need this information when registering with the broker.

Broker API

The public API of the broker can be exposed via WSDM capabilities and Java interfaces. The API should provide operations required by the use cases described in the above section.

Notes

1. Use of iBatis: iBatis was initially introduced into the codebase as a means to access relational data via Java API. It was part of an investigation effort on query languages for heterogeneous data sources. I suggest that we review the use of iBatis in COSMOS. If it is only used to access data in the registry database, I don’t think it will bring much value. Using JDBC may be sufficient for our purpose. We should reduce dependency of 3rd party program as much as possible.