Difference between revisions of "Talk:COSMOS Design 193420"

Latest revision as of 13:06, 7 August 2007

COSMOS_Design_193420

We might be missing some context here. We jump right into the first 10 entries talking about key sets etc... If this is "the most simple example", and I have to know all this stuff about key sets, do we need to spend some time simplifying what we are doing?

What will the demo do? How will we show a collection assembly and a query assembly with no additional prerequisites? The last time we discussed this, I was thinking we'd have something like the following:

• Everything runs in the same eclipse environment. This was because we might want to tell people where to put break points in the code.
• We would have a singleton that held a collection of random values or even just a random value.
• We would have a data collection assembly that went off and added a new value to the collection or updated the one that was already there.

At this point, what you could do is show a simple client that did the control of the data collection assembly. This would be based on Joel's simple client that's there already. Start/stop is all we need here. Once that was done, you would have a query assembly that went off and got the values from the singleton.

If we wanted to, we could then take the simple example and have the information written to a database, e.g. derby. This would be a single table with a string value. It would show how all we need to do is drop in a new piece of the collection and query assemblies, not re-invent everything else. We could even do an XML document data store as well if we wanted.

In my head, this is where the simple demo stopped. It gives us plenty to get into and should be able to cover everything you've outlined in the previous page.

Query Assembly

Does anyone have a good description for this ?

--Marty 02:11, 7 August 2007 (EDT)

Collection Assembly

Does anyone have a good description for this ?

--Marty 02:11, 7 August 2007 (EDT)

Key Sets

If keysets are not included in the simple example, then it does not show a true data collection, and how the processing works. The fact that you make your data source simple, should make the concept of how keysets are used at its simplest form. If we just take the simplest example, perhaps we can just pass straight XML around with no fancy stuff?

--Marty 01:37, 7 August 2007 (EDT)

I agree we need keysets and should explain them. I didn't mean to imply that we didn't. I would like to see how we can start with the simple concept, then move to the more complex.

Not sure I agree. One of our original design goals was to remain agnostic towards how data was catalogued. That's the reason for the use of opaque identifiers in our dialect introspection and generic query apis. While I personally like the keyset concept, I don't think it applies universally.

Simple Demo Data

From a Query Assembly perspective the data will consist of two Data Sources, a flat file and a database table.

1. Flat File - Contains two fields, FirstName, LastName, delimited by space
2. Derby table - Table name is Names, it contains two columns, FirstName, LastName

From a Collection Assembly perspective.....

--Marty 02:30, 7 August 2007 (EDT)

Target Environment

I am not so sure having a demo run just in eclipse is what we want. We want to show a client communicating with a couple of data managers and doing something with the data. I see that as being something that could just be deployed to a Tomcat server. I think demoing in an OSGI framework is not a simple thing. You would have to explain what OSGI is and how you set that up, which defeats the simple bit.

I agree that it will be useful to see it in Eclipse for debugging purposes, and to see where things go, but a demo should not just involve a debugging session, I think it should also show how it looks to the outside world. I think it should show the concepts of Data Collection. I think that the next stage after a demo is another Tutorial on how to debug this simple Demo.

--Marty 01:38, 7 August 2007 (EDT)

I guess this is who the "demo" is targeted towards. I thought we'd be targeting a developer that wants to integrate an existing data store, hence the eclipse focus. This is the scenario we will be in with wrapping to CMDBf APIs. So we could get the simple example up and running in eclipse, then it's a question of how to incorporate "my real APIs". I'm OK either way... let's just decide.

I agree that the demo should be targeting a developer.

--Marty 01:38, 7 August 2007 (EDT)

In my opinion, TOMCAT is a more likely and more popular environment to run this demo in. It is also easier to set up. Cosmos will deploy to both Tomact and OSGI.

--Marty 03:00, 7 August 2007 (EDT)

Mechanisms for generating a Keyset

1. Dynamically generate from database schema
2. Create a GUI application that generates keysets
3. Manually type the keyset

--Marty 18:21, 6 August 2007 (EDT)

I suggest that for the demo we manually create the Keyset File.

--Marty 02:52, 7 August 2007 (EDT)

If you look at the Tomcat JMX example, you'll see that the keyset definitions are registered by the collector:

		<sink cosmos:factory="org.eclipse.cosmos.dc.sample.components.sink.StatisticalPersistor" cosmos:optimizable="true">
			<sample:binding>
				<cosmos:keyset name="TomcatSet" register="true">
					<cosmos:key name="CurrentBusyThreads" type="integral"/>
					<cosmos:key name="CurrentThreadCount" type="integral"/>
					<cosmos:key name="HeapMemoryUsed" type="integral"/>
					<cosmos:key name="NonHeapMemoryUsed" type="integral"/>
				</cosmos:keyset>
				<sample:mappings>
					<sample:map attribute="currentThreadsBusy" key="CurrentBusyThreads"/>
					<sample:map attribute="currentThreadCount" key="CurrentThreadCount"/>
					<sample:map key="HeapMemoryUsed" attribute="HeapMemoryUsage:used"/>
					<sample:map key="NonHeapMemoryUsed" attribute="NonHeapMemoryUsage:used"/>
				</sample:mappings>
			</sample:binding>
		</sink>

There's a remotable API in the current registry implementation that will allow you to query for keyset defnintions, etc.

The console command dc list datasets will dump datasets by keyset definition, etc. You can look at the implementation to see how to use the registry API.

Sample output:

dc list datasets
DataSet ApacheInstance1181791800074
  Source ApacheInstance
  Start Thu Jan 16 21:22:59 EST 2003
  Stop Thu Jan 16 21:22:59 EST 2003
  Flow Apache Server:CBE
  Source ApacheInstance
  Dimensions 
      Dim severity
      Dim msg
DataSet ApacheInstance1181792383473
  Source ApacheInstance
  Start Thu Jan 16 21:22:59 EST 2003
  Stop Thu Jan 16 21:22:59 EST 2003
  Flow Apache Server:CBE
  Source ApacheInstance
  Dimensions 
      Dim severity
      Dim msg
DataSet server 33 tomcat1186500179357
  Source server 33 tomcat
  Start Tue Aug 07 11:22:59 EDT 2007
  Stop Tue Aug 07 11:22:59 EDT 2007
  Flow Tomcat Server:TomcatSet
  Source server 33 tomcat
  Dimensions 
      Dim CurrentBusyThreads
      Dim CurrentThreadCount
      Dim HeapMemoryUsed
      Dim NonHeapMemoryUsed

Key Family File - Examples

--Marty 02:50, 7 August 2007 (EDT)

Key Family File - Location

1. In Memory
2. In a database
3. In a file on a share
4. In an application server
5. In a ftp server
6. In a Web Server

Do we go for one? Do we allow for a choice? If there is a choice, how do you select it?

--Marty 02:50, 7 August 2007 (EDT)

I suggest for the Simple Demo, the file should be in Tomcat.

--Marty 02:56, 7 August 2007 (EDT)

Key Family File - How the Client gets it

1. The Data Manager pushes it to the client?
2. The Client gets it from the data manager?
3. Does it get the whole file?
4. Does it just get passed the reference to where the Key Family File is ?

--Marty 04:09, 7 August 2007 (EDT)

Key Family File - Formats

1. Straight XML?
2. SDMX compliant XML?
3. POJO?

--Marty 04:38, 7 August 2007 (EDT)

Client - How to parse the Key Family File

1. Straight XML - Sax parser? Dom parser?
2. SDMX compliant XML - Sdmx parser? Is there such a thing?
3. POJO - Java

--Marty 05:15, 7 August 2007 (EDT)

Client - What Format is the Key Family File?

--Marty 05:31, 7 August 2007 (EDT)

Client - What to do with the Key Family File?

1. How does the client decide on what it wants?
2. Assuming the client knows how to parse the file, do we store that information internally in an array? as an object?

--Marty 07:13, 7 August 2007 (EDT)

Client - How to formulate the query to the Data Manager

--Marty 07:16, 7 August 2007 (EDT)