Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "COSMOS DC Design Discussions"
| (5 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | * Binding service: The binding service is used for instantiating and initializing components. The current design requires exploiters of the framework to provide a binding service for user defined components. However, the information required for the binding service can be extracted from the assembly XML and can be abstracted into the framework level. If you look at the SampleBindingService and GLABindingService, they are almost in the same pattern. | + | [http://wiki.eclipse.org/Image:DesignDiscussions.zip Presentation] |
| + | |||
| + | * Binding service: The binding service is used for instantiating and initializing components. The current design requires exploiters of the framework to provide a binding service for user defined components. However, the information required for the binding service can be extracted from the assembly XML and can be abstracted into the framework level. If you look at the SampleBindingService and GLABindingService, they are almost in the same pattern. | ||
* The coupling between components | * The coupling between components | ||
** The components should be loosely coupled. | ** The components should be loosely coupled. | ||
| Line 13: | Line 15: | ||
</source> | </source> | ||
</context> | </context> | ||
| − | This is a more generic design but brings with it complexity. We can adapt a serial design which is more straight forward: | + | This is a more generic design but brings with it complexity. This document structure will also make it difficult to deine an XML scheme to describe the structure. We can adapt a serial design which is more straight forward: |
<context> | <context> | ||
<source>...</source> | <source>...</source> | ||
| Line 24: | Line 26: | ||
*# metadata definition: Mapping of legacy data models to the generic metadata specification (SDMX-like model) | *# metadata definition: Mapping of legacy data models to the generic metadata specification (SDMX-like model) | ||
*# WSDM management features | *# WSDM management features | ||
| − | * Currently, the above three concerns are too intertwined, making the framework very difficult to use. | + | * Currently, the above three concerns are too intertwined, making the framework very difficult to use. The definition of datasource, keyset, etc, should separated from the assembly XML, and not cached in component implementations. |
* Query framework: I find the query framework still quite ad hoc. There is no standard query mechanism. The query assembly binds a query name with a class, which implements an application specific query interface. The implementation of the query can use any method. I somehow find the concept of "inbound" vs. "outbound" in a data assembly unnecessary. We can provide standard datasource that makes SQL queries to a relation database, or XPATH queries on XML documents. These kinds of generic "data source" or "queries" can be used in inbound or outbound data collection. | * Query framework: I find the query framework still quite ad hoc. There is no standard query mechanism. The query assembly binds a query name with a class, which implements an application specific query interface. The implementation of the query can use any method. I somehow find the concept of "inbound" vs. "outbound" in a data assembly unnecessary. We can provide standard datasource that makes SQL queries to a relation database, or XPATH queries on XML documents. These kinds of generic "data source" or "queries" can be used in inbound or outbound data collection. | ||
* Other things that can be done for data collection: | * Other things that can be done for data collection: | ||
| Line 33: | Line 35: | ||
** a more user friendly interface (command line or UI) for interacting with the dc framework. | ** a more user friendly interface (command line or UI) for interacting with the dc framework. | ||
** documentation | ** documentation | ||
| + | |||
| + | |||
| + | [[Talk:COSMOS_DC_Design_Discussions | Joel's comments]] | ||
Latest revision as of 12:08, 8 August 2007
- Binding service: The binding service is used for instantiating and initializing components. The current design requires exploiters of the framework to provide a binding service for user defined components. However, the information required for the binding service can be extracted from the assembly XML and can be abstracted into the framework level. If you look at the SampleBindingService and GLABindingService, they are almost in the same pattern.
- The coupling between components
- The components should be loosely coupled.
- The current framework design uses an event model to pass events/data from component to component in the assembly.
- after each event is processed, the object is dispatched by calling the wire method of the next "component" in the assembly. This is potentially inefficient and make optimization difficult. For example, when I wrote the CBE data sink, I had to do an insert for each CBE. If I have access to the whole list of CBE objects, I can do a batch data insert operation.
- Question: Do we need to support the use case where the same data source need to be dispatched to multiple data transformers or data sinks concurrently? The current assembly XML schema design uses a containment structure for components, and allows a data source to pipe data into multiple filters, and each filter in turn pipe data into multiple data sinks. (The code example has not exploited this scenario yet.)
<context>
Invalid language.
You need to specify a language like this: <source lang="html4strict">...</source>
Supported languages for syntax highlighting:
4cs, 6502acme, 6502kickass, 6502tasm, 68000devpac, abap, actionscript, actionscript3, ada, algol68, apache, applescript, apt_sources, arm, asm, asp, asymptote, autoconf, autohotkey, autoit, avisynth, awk, bascomavr, bash, basic4gl, bf, bibtex, blitzbasic, bnf, boo, c, c_loadrunner, c_mac, caddcl, cadlisp, cfdg, cfm, chaiscript, cil, clojure, cmake, cobol, coffeescript, cpp, cpp-qt, csharp, css, cuesheet, d, dcl, dcpu16, dcs, delphi, diff, div, dos, dot, e, ecmascript, eiffel, email, epc, erlang, euphoria, f1, falcon, fo, fortran, freebasic, freeswitch, fsharp, gambas, gdb, genero, genie, gettext, glsl, gml, gnuplot, go, groovy, gwbasic, haskell, haxe, hicest, hq9plus, html4strict, html5, icon, idl, ini, inno, intercal, io, j, java, java5, javascript, jquery, kixtart, klonec, klonecpp, latex, lb, ldif, lisp, llvm, locobasic, logtalk, lolcode, lotusformulas, lotusscript, lscript, lsl2, lua, m68k, magiksf, make, mapbasic, matlab, mirc, mmix, modula2, modula3, mpasm, mxml, mysql, nagios, netrexx, newlisp, nsis, oberon2, objc, objeck, ocaml, ocaml-brief, octave, oobas, oorexx, oracle11, oracle8, otj, oxygene, oz, parasail, parigp, pascal, pcre, per, perl, perl6, pf, php, php-brief, pic16, pike, pixelbender, pli, plsql, postgresql, povray, powerbuilder, powershell, proftpd, progress, prolog, properties, providex, purebasic, pycon, pys60, python, q, qbasic, rails, rebol, reg, rexx, robots, rpmspec, rsplus, ruby, sas, scala, scheme, scilab, sdlbasic, smalltalk, smarty, spark, sparql, sql, stonescript, systemverilog, tcl, teraterm, text, thinbasic, tsql, typoscript, unicon, upc, urbi, uscript, vala, vb, vbnet, vedit, verilog, vhdl, vim, visualfoxpro, visualprolog, whitespace, whois, winbatch, xbasic, xml, xorg_conf, xpp, yaml, z80, zxbasic
<filter>
<sink>
</sink>
</fliter>
</context>
This is a more generic design but brings with it complexity. This document structure will also make it difficult to deine an XML scheme to describe the structure. We can adapt a serial design which is more straight forward:
<context>
Invalid language.
You need to specify a language like this: <source lang="html4strict">...</source>
Supported languages for syntax highlighting:
4cs, 6502acme, 6502kickass, 6502tasm, 68000devpac, abap, actionscript, actionscript3, ada, algol68, apache, applescript, apt_sources, arm, asm, asp, asymptote, autoconf, autohotkey, autoit, avisynth, awk, bascomavr, bash, basic4gl, bf, bibtex, blitzbasic, bnf, boo, c, c_loadrunner, c_mac, caddcl, cadlisp, cfdg, cfm, chaiscript, cil, clojure, cmake, cobol, coffeescript, cpp, cpp-qt, csharp, css, cuesheet, d, dcl, dcpu16, dcs, delphi, diff, div, dos, dot, e, ecmascript, eiffel, email, epc, erlang, euphoria, f1, falcon, fo, fortran, freebasic, freeswitch, fsharp, gambas, gdb, genero, genie, gettext, glsl, gml, gnuplot, go, groovy, gwbasic, haskell, haxe, hicest, hq9plus, html4strict, html5, icon, idl, ini, inno, intercal, io, j, java, java5, javascript, jquery, kixtart, klonec, klonecpp, latex, lb, ldif, lisp, llvm, locobasic, logtalk, lolcode, lotusformulas, lotusscript, lscript, lsl2, lua, m68k, magiksf, make, mapbasic, matlab, mirc, mmix, modula2, modula3, mpasm, mxml, mysql, nagios, netrexx, newlisp, nsis, oberon2, objc, objeck, ocaml, ocaml-brief, octave, oobas, oorexx, oracle11, oracle8, otj, oxygene, oz, parasail, parigp, pascal, pcre, per, perl, perl6, pf, php, php-brief, pic16, pike, pixelbender, pli, plsql, postgresql, povray, powerbuilder, powershell, proftpd, progress, prolog, properties, providex, purebasic, pycon, pys60, python, q, qbasic, rails, rebol, reg, rexx, robots, rpmspec, rsplus, ruby, sas, scala, scheme, scilab, sdlbasic, smalltalk, smarty, spark, sparql, sql, stonescript, systemverilog, tcl, teraterm, text, thinbasic, tsql, typoscript, unicon, upc, urbi, uscript, vala, vb, vbnet, vedit, verilog, vhdl, vim, visualfoxpro, visualprolog, whitespace, whois, winbatch, xbasic, xml, xorg_conf, xpp, yaml, z80, zxbasic
...
<filter>...</fliter> <sink>...</sink> </context>
- wire method: The method name is not in the interface. Reason is probably because we need a different name for each component, and the parameter object type is also different. Assumption of the method name and introspection on the available methods and comparison with a list of acceptable object types is done. I find this process quite complicated, and potentially difficult to debug. An alternative is to use a generic method name such as process, that pass in an array of Objects. It's up to the implementation of the component to decide how to handle the object.
- Separation of concerns: There are three crosscutting concerns in the data collection code:
- assembly definition: the definition of the assembly XML files, and writing code for a assembly component (e.g. a transformer).
- metadata definition: Mapping of legacy data models to the generic metadata specification (SDMX-like model)
- WSDM management features
- Currently, the above three concerns are too intertwined, making the framework very difficult to use. The definition of datasource, keyset, etc, should separated from the assembly XML, and not cached in component implementations.
- Query framework: I find the query framework still quite ad hoc. There is no standard query mechanism. The query assembly binds a query name with a class, which implements an application specific query interface. The implementation of the query can use any method. I somehow find the concept of "inbound" vs. "outbound" in a data assembly unnecessary. We can provide standard datasource that makes SQL queries to a relation database, or XPATH queries on XML documents. These kinds of generic "data source" or "queries" can be used in inbound or outbound data collection.
- Other things that can be done for data collection:
- Hello world example for using the dc framework
- More useful data source and sinks:
- GLA datasource: The current example uses an "embedded GLA source". The GLA source can be an external GLA instance.
- Revisit CBE schema and decision regarding interoperability with TPTP.
- a more user friendly interface (command line or UI) for interacting with the dc framework.
- documentation