Difference between revisions of "SDMX Example"

Revision as of 11:41, 20 August 2007

Back to 197867 Back to Talk for 197867

An example of using SDMX

 <dataSet start="00:00:00" end="01:30:00">

  <dataFlow id="statistical data for intranet web server">

    <keyFamily id="statisticalData">
      <dimension id="CurrentBusyThreads"/>
      <dimension id="CurrentThreadCount"/>
      <dimension id="HeapMemoryUsed"/>
      <dimension id="NonHeapMemoryUsed"/>
    </keyFamily>

    <dataSource id="intranet web server">
      <dataSourceType id="Tomcat Server">
    </dataSource>

  </dataFlow>
  
  <observations keyFamily="statisticalData">
    <observation captureTime="00:00:10">
      <key dimension="CurrentBusyThreads" value="1"/>
      <key dimension="CurrentThreadCount" value="25"/>
      <key dimension="HeapMemoryUsed" value="19711768"/>
      <key dimension="NonHeapMemoryUsed" value="30862992"/>
    </observation>
    <observation captureTime="00:00:15">
      <key dimension="CurrentBusyThreads" value="1"/>
      <key dimension="CurrentThreadCount" value="25"/>
      <key dimension="HeapMemoryUsed" value="19711768"/>
      <key dimension="NonHeapMemoryUsed" value="30862992"/>
    </observation>
  </observations>
  
 </dataSet>

Metadata that can be part of the broker registry:

  <dataFlow id="statistical data for intranet web server">

    <keyFamily id="statisticalData">
      <dimension id="CurrentBusyThreads"/>
      <dimension id="CurrentThreadCount"/>
      <dimension id="HeapMemoryUsed"/>
      <dimension id="NonHeapMemoryUsed"/>
    </keyFamily>

    <dataSource id="intranet web server">
      <dataSourceType id="Tomcat Server">
    </dataSource>

  </dataFlow>

We can associate an EPR with a dataFlow.

  <dataFlow id="statistical data for intranet web server">
    <epr>...</epr>
    <keyFamily>...</keyFamily>
    <dataSource>...</dataSource>
  </dataFlow>

Data sets and observations of each data set can be retrieve from data managers via query API.

Jimmy's comments on the SDMX example

In your SDMX example, you refer to the <keyFamily id="statisticalData">..

Comment 1: This keyFamily should live in the Data Manager. This means that I am questioning your follow on section that states "Metadata that can be part of the broker registry". Why do you want to keep the Metadata in the Broker? Will this not DRASTICALLY increase the traffic between the Broker and the plethora of Data Manager which may come and go often? What value can we derive by keeping the metadata in the Broker?

Comment 2: Hubert, above you state that "The SDMX concepts can be hidden behind some easy-to-use APIs". Given that we may have such API's, this reduces the need even FURTHER for us to have the metadata live in the Broker.

Comment 3: The observation, i.e. the data obviously is ALSO NOT stored in the Broker...

SML & SDMX

Notes from conversation on 20-Aug-07

• The amount of metadata the broker maintains should be minimal

• Two concerns w/SDMX format
  • Changes over time
  • Amount of data--(lots of it)
• Concerned about storing the "dimension" stuff inside the broker

• Can we start with simply the classification of the data and the type of the resource?
  • This would translate to the data source id and the data family id
  • Can we align this up with SML concepts

• Need to walk through query examples
  • Martin's subnet query
  • Statistical data & config data