COSMOS Design 200222

Adding support for the CMDBf query service on top of the SML repository

Change History

Name:	Date:	Revised Sections:
David Whiteman	8/21/2007	Initial version
David Whiteman	8/27/2007	Added use cases
Ali Mehregani	8/28/2007	Modified purpose & requirements Added implementation details & task breakdown
David Whiteman	8/28/2007	Expanded use cases Added test coverage

Workload Estimation

Rough workload estimate in person weeks

Process	Sizing	Names of people doing the work
Design	1	David/Ali
Code	2	David/Ali
Test	1	David/Ali
Documentation	0.4
Build and infrastructure	0.2
Code review, etc.*	0.2
TOTAL	4.8

• - includes other committer work (e.g. check-in, contribution tracking)

Terminologies/Acronyms

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term regarding how it is used in this document:

Term	Definition
MDR	management data repository
CMDBf	specification for a CMDB that federates between multiple MDRs[1]
federating CMDB	the CMDB that federates between MDRs, offering a common access point to clients and correlating identifying record data
CMDB	configuration management database
SML Repository	repository of SML documents hosted in COSMOS with an established API (see bugzilla 179828)
Query service	MDRs make data available to Clients via a Query service. Queries may select and return items, relationships, and/or graphs containing items and relationships.

Purpose

According to the CMDBf specification, an MDR requires the implementation of the following services to be fully integrated with a federating CMDB:

1. Registration
2. Query

During the i6 time-frame a query service will be implemented for the SML repository. The service will allow clients to query the available data with the XML query language described in the CMDBf specification. The output of the query will also be in XML format, which the client is expected to understand.

Currently clients are expected to interact with the SML repository via a set of convenience APIs. This enhancement does not imply the removal of such APIs. They may need to slightly be modified to accompany the new changes but they can still be reused by clients who have no intentions of leveraging the new query service. Since the current consumers of the SML repository convenience APIs don't exploit the full range of query capabilities afforded by the query service, a programming interface will also be available to allow queries to be submitted using the XML schema described in the CMDBf standard.

There will likely be common set of code between the COSMOS client and the SML repository. It's expected that this common set will reside under the data collection subproject. One of the primary objectives of this document is to identify this reusable set of common code.

Requirements

This section lists the requirements of this enhancement:

1. Querying the SML repository using the XML query format specified in the CMDBf specification
2. Producing query results to clients using the XML response format specified in the CMDBf specification

Use Cases

For the following use cases, the SML repository acting as an MDR will contain the data described in the org.eclipse.cosmos.rm.example.datacenter COSMOS plug-in:

Use Case 1: Retrieve All Computers Using Red Hat Linux 8

1. Client connects to the SML repository
2. Client submits request for the CMDBf query service operation
3. Client sets the argument of the operation to a query that will retrieve: all computers with an installed OS with a brand name of "Red Hat 8"
4. Client runs the operation
5. Client retrieves the response from the operation's output

Use Case 2: Retrieve All Computers With At Least 2000 MB RAM

1. Client connects to the SML repository
2. Client submits client request for the CMDBf query service operation
3. Client sets the argument of the operation to a query that will retrieve: all computers with a RAM value greater than or equal to 2000 MB.
4. Client runs the operation
5. Client retrieves the response from the operation's output

Use Case 3: Retrieve All Server Software

1. Client connects to the SML repository
2. Client submits client request for the CMDBf query service operation
3. Client sets the argument of the operation to a query that will retrieve: all server software.
4. Client runs the operation
5. Client retrieves the response from the operation's output

Use Case 4: Retrieve Serial Numbers and Disk Drive Capacities

1. Client connects to the SML repository
2. Client submits client request for the CMDBf query service operation
3. Client sets the argument of the operation to a query that will retrieve: the serial numbers and disk drive capacities of all computers.
4. Client runs the operation
5. Client retrieves the response from the operation's output

Use Case 5: Retrieve All Non-database Software on Linux Machines

1. Client connects to the SML repository
2. Client submits client request for the CMDBf query service operation
3. Client sets the argument of the operation to a query that will retrieve: all software installed on Linux machines that is not a relational database.
4. Client runs the operation
5. Client retrieves the response from the operation's output

Implementation Detail

The following section details out the implementation of the CMDBf query service for the SML repository. As mentioned before, the SML repository has defined a set of convenience APIs that it is currently leveraged by clients. The implementation of this enhancement will need to understand queries based on the CMDBf specification and make use of the convenience APIs to fetch the required data.

Scope

Using the instanceIdSelector under the itemTemplate is equivalent to fetching documents with an org.eclipse.cosmos.rm.repository.provisional.resource.ISMLMetadata that has a specific ID. The property, xpath, and the record type selectors will all be supported under the itemTemplate element. Clients will also be able to use the propertySubsetDirective to return a set of desired properties, instead of full records.

The only valid relationship between items stored in an SML repository is the 'references' relationship. Item i1 references i2 if and only if there exists an inter-document reference of i2 in the SML document representation of i1. All relationshipTemplate elements should include a record type selector with:

• namespace = "http://cosmos.org"
• localName = "references"

No other selectors should be used in conjunction with this relationship.

The list below summarizes the scope of this enhancement:

• This enhancement will not provide an implementation of a federating CMDB
• The client will directly access the MDR and get data through the query service
• There will only be one MDR in COSMOS for i6. This will be the COSMOS SML repository
• The Registration service, as described by the CMDBf spec will not be implemented for i6 (Since there is no federating CMDB available)
• The SML Repository MDR can be referenced with the ID: http://cosmos.org/rm/sml/repository

Architecture

The flow diagram below depicts how the query service will process an input. The service will apply the selectors in the following order:

1. instanceIdSelector
2. recordTypeSelector
3. propertyValueSelector
4. xpath1Selector

The instance and record type selectors are faster and effective mechanisms of narrowing the result set for a query, whereas the property and xpath selectors are more processor intensive operations that are applied in the end.

The CMDBf query service will be implemented as an operation of the SML repository. Similar to the validation, import, and export operations, a client will first need to establish a connection to the repository to obtain the CMDBf query operation. The operation will take an input stream as argument and will produce an input stream as output. The argument is expected to be set to the XML query and the output will be the query response.

There will be a reusable component under the data collection subproject that will need to transform an XML query into a graph that is understandable by the query service. There also needs to be a common component to convert a graph representing a query response into an XML document that conforms to the CMDBf standard. The image below depicts the interaction between the query service provider, the SML repository, and the query service. The input and output transformers are reusable components that will reside under the data collection subproject. There needs to be a mirror operation provided for the output transformation to convert an XML query response into a graph representation. This is useful for clients that intend to traverse through a POJO representation of the response.

The diagram above does not include the sequence of steps needed for a client to access a query service end point. It only depicts the layer that will reside behind a WSDM end point that the COSMOS client will remotely access. The end-to-end interaction between the client and the query service should be described in the design of: https://bugs.eclipse.org/bugs/show_bug.cgi?id=200244. There needs to be a systematic method defined for a client to access the query service of MDRs.

Test Coverage

Unit tests will need to perform the following tests:

• Complex tests outlined in the Use Cases section of this document
• Tests using an instance selector, one with 0 results and one with exactly 1 result. More than one result will be logged as a failure.
• Tests using a property value selector, one for each of the following operators:
  • equal (caseSensitive attribute set to true, set to false, and omitted; negate set to true, false, or omitted; string, boolean, time, and integer types used)
  • less (negate set to true, false, or omitted; string, time, and integer types used)
  • lessOrEqual (negate set to true, false, or omitted; string, time, and integer types used)
  • greater (negate set to true, false, or omitted; string, time, and integer types used)
  • greaterOrEqual (negate set to true, false, or omitted; string, time, and integer types used)
  • contains (caseSensitive attribute set to true, set to false, and omitted; negate set to true, false, or omitted)
  • like (caseSensitive attribute set to true, set to false, and omitted; negate set to true, false, or omitted)
  • isNull (negate set to true, false, or omitted)
• Tests using a record type selector:
  • type is exact type match
  • type requested is supertype of existing record
• Tests using an XPath selector:
  • using correct syntax for expression, with and without results
  • using incorrect syntax for expression
• Tests combining 2 item templates and a relationship template, that use a drop selector to exclude the relationship template and the 2nd item template
• Tests using a property subset directive
  • with two properties listed
  • with no properties listed
• Tests using relationship cardinality, using minimum, maximum, and both attributes

Task Breakdown

The following section includes the tasks required to complete this enhancement along with a PERT chart that displays the dependencies of each task.

create sql methods to work with existing CBE COSMOS schema File:QueryService-PertChart.png

1. Create graph representation of query input and response
2. Create the input transformer
3. Create the output transformer
4. Add in the query service as an SML repository operation
5. Create a generic framework for implementing selectors
6. Add in the instance id selector
7. Create JUnit tests for the instance id selector
8. Add in the record type selector
9. Create JUnit tests for the record type selector
10. Add in the property value selector
11. Create JUnit tests for the property value selector
12. Add in the xpath selector
13. Create JUnit tests for the xpath selector
14. Implement the property subset directive
15. Ensure that all selectors work with itemTemplate elements
16. Implement the relationship template constraint
17. Add Junit tests for querying items and relationships

Related Links

• 204959 Design for refactoring of this work to be extensible

Open Issues/Questions

All reviewer feedback should go in the Talk page for 200222.

Issues (and potential errors) regarding the CMDBf spec, v0.95. Email responses from a member of the CMDBf expert group are marked in red.

• All of the examples and psuedo-schema in section 4.3.1 for <prefixMapping> have the attributes "prefix" and "value". However, the data model XSD in appendix B (and on the CMDBf site) lists the following as the two prefixMapping attributes as "prefix" and "namespace". When I wrote some <query> testcases based on the samples in the CMDBf spec, the schema validation failed.
• In the query result example in section 4.4, both <relationship> instances have the same recordId as each other. Agree
• In the relationship records in that example, the "foo" namespace maps to http://example.com/computerModel in both cases? Earlier, "comp" was used with that same URL, so it would seem the URL would need to be changed in the "foo" case, or the "foo" example should use "comp" again. Agree
• On line 1205, it says "http://example.com/administers/PeteTheLabLabTechToLabMachineA". Note the "LabLab" typo. Agree
• All of the included examples (such as on line 1086) begin with <query>. I believe they should instead look like this: <query xmlns="http://schemas.cmdbf.org/0-9-5/datamodel"> Making this change will make the query service code a lot easier to write, as the SAX validating parser only works if it has the correct xmlns value. It looks like some revisions of the document did include an xmlns attribute, but for some reason it was not included in the current one available at cmdbf.org. That's definitely the namespace for the <query> so including it is certainly correct & you should feel free to do so.
• Line 757 should be </contains> instead of </stringContains> Agree
• According to the pseudo schema of xpath1Selector, there are 0 or more prefix mappings that can be included under <xpath1Selector>. The CMDBf data model schema does not specify a minOccurs/maxOccurs for the prefixMapping element. <xs:element name="prefixMapping" type="cmdbf:PrefixMappingType" /> should be: <xs:element name="prefixMapping" type="cmdbf:PrefixMappingType" minOccurs="0" maxOccurs="unbounded"/>

---