Jump to: navigation, search

Difference between revisions of "ATL VM Testing"

(Non-regression tests)
(Non-regression tests)
Line 30: Line 30:
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/SpreadSheetML2PerformanceUMLStateChart/Trace2PerformanceMetrics/?root=Modeling_Project Trace2PerformanceMetrics]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/SpreadSheetML2PerformanceUMLStateChart/Trace2PerformanceMetrics/?root=Modeling_Project Trace2PerformanceMetrics]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
 +
! colspan=1 | 0.015s.
 
! colspan=1 | 0.0s.
 
! colspan=1 | 0.0s.
! colspan=1 | 0.016s.
 
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/BibTeX2DocBook/?root=Modeling_Project BibTeX2DocBook]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/BibTeX2DocBook/?root=Modeling_Project BibTeX2DocBook]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.015s.
+
! colspan=1 | 0.031s.
 
! colspan=1 | 0.047s.
 
! colspan=1 | 0.047s.
 
! colspan=1 |  
 
! colspan=1 |  
Line 48: Line 48:
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Book2Publication/XML2Book/?root=Modeling_Project XML2Book]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Book2Publication/XML2Book/?root=Modeling_Project XML2Book]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.016s.
+
! colspan=1 | 0.0s.
 
! colspan=1 | 0.016s.
 
! colspan=1 | 0.016s.
 
! colspan=1 |  
 
! colspan=1 |  
Line 55: Line 55:
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | 0.0s.
 
! colspan=1 | 0.0s.
! colspan=1 | 0.016s.
+
! colspan=1 | 0.0s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Class2Relational/Ecore2Class/?root=Modeling_Project Ecore2Class]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Class2Relational/Ecore2Class/?root=Modeling_Project Ecore2Class]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
 +
! colspan=1 | 0.0s.
 
! colspan=1 | 0.016s.
 
! colspan=1 | 0.016s.
! colspan=1 | 0.015s.
 
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/DSL2EMF/DSL2KM3/?root=Modeling_Project DSL2KM3]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/DSL2EMF/DSL2KM3/?root=Modeling_Project DSL2KM3]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.031s.
+
! colspan=1 | 0.032s.
 
! colspan=1 | 0.031s.
 
! colspan=1 | 0.031s.
 
! colspan=1 |  
 
! colspan=1 |  
Line 72: Line 72:
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Book2Publication/Book2Publication/?root=Modeling_Project Book2Publication]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Book2Publication/Book2Publication/?root=Modeling_Project Book2Publication]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
 +
! colspan=1 | 0.016s.
 
! colspan=1 | 0.015s.
 
! colspan=1 | 0.015s.
! colspan=1 | 0.0s.
 
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Class2Relational/Class2Relational/?root=Modeling_Project Class2Relational]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Class2Relational/Class2Relational/?root=Modeling_Project Class2Relational]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.015s.
+
! colspan=1 | 0.0s.
! colspan=1 | 0.016s.
+
! colspan=1 | 0.0s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
Line 90: Line 90:
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/EMF2DSL/DSL2XML/?root=Modeling_Project DSL2XML]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/EMF2DSL/DSL2XML/?root=Modeling_Project DSL2XML]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.11s.
+
! colspan=1 | 0.094s.
! colspan=1 | 0.171s.
+
! colspan=1 | 0.156s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
Line 97: Line 97:
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | 0.078s.
 
! colspan=1 | 0.078s.
! colspan=1 | 0.156s.
+
! colspan=1 | 0.063s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/ModelBridge/XML2DSLModel/?root=Modeling_Project XML2DSLModel]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/ModelBridge/XML2DSLModel/?root=Modeling_Project XML2DSLModel]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 1.0s.
+
! colspan=1 | 0.938s.
! colspan=1 | 10.703s.
+
! colspan=1 | 10.203s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Ant2Maven/XML2Ant/?root=Modeling_Project XML2Ant]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Ant2Maven/XML2Ant/?root=Modeling_Project XML2Ant]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.266s.
+
! colspan=1 | 0.25s.
! colspan=1 | 2.094s.
+
! colspan=1 | 2.063s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/ModelBridge/DSLModel2KM2/?root=Modeling_Project DSLModel2KM2]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/ModelBridge/DSLModel2KM2/?root=Modeling_Project DSLModel2KM2]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.015s.
+
! colspan=1 | 0.016s.
! colspan=1 | 0.046s.
+
! colspan=1 | 0.031s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Ant2Maven/Ant2Maven/?root=Modeling_Project Ant2Maven]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Ant2Maven/Ant2Maven/?root=Modeling_Project Ant2Maven]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.094s.
+
! colspan=1 | 0.015s.
! colspan=1 | 0.109s.
+
! colspan=1 | 0.031s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/EMF2DSL/KM32DSL/?root=Modeling_Project KM32DSL]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/EMF2DSL/KM32DSL/?root=Modeling_Project KM32DSL]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.016s.
+
! colspan=1 | 0.032s.
! colspan=1 | 0.047s.
+
! colspan=1 | 0.031s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Ant2Maven/Maven2XML/?root=Modeling_Project Maven2XML]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/Ant2Maven/Maven2XML/?root=Modeling_Project Maven2XML]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.062s.
+
! colspan=1 | 0.031s.
! colspan=1 | 0.062s.
+
! colspan=1 | 0.047s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/ATL2Problem/?root=Modeling_Project ATL2Problem]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/ATL2Problem/?root=Modeling_Project ATL2Problem]
! colspan=1 | <b style="color:red">FAIL</b>
+
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 |  
+
! colspan=1 | 0.046s.
! colspan=1 | 0.094s.
+
! colspan=1 | 0.078s.
! colspan=1 | Missing EnumLiteral implementation.
+
! colspan=1 | EnumLiteral support added on 25/01/2008.
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/DSL2EMF/XML2DSL/?root=Modeling_Project XML2DSL]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/MetamodelBridge/DSL2EMF/XML2DSL/?root=Modeling_Project XML2DSL]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | 0.078s.
 
! colspan=1 | 0.078s.
! colspan=1 | 0.485s.
+
! colspan=1 | 0.453s.
 
! colspan=1 |  
 
! colspan=1 |  
 
|-
 
|-
Line 151: Line 151:
 
! colspan=1 | <b style="color:red">FAIL</b>
 
! colspan=1 | <b style="color:red">FAIL</b>
 
! colspan=1 |  
 
! colspan=1 |  
! colspan=1 | 0.031s.
+
! colspan=1 | 0.015s.
 
! colspan=1 | Transformation fails without errors.
 
! colspan=1 | Transformation fails without errors.
 
|-
 
|-
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/ModelBridge/KM32ATL_KM22MM/?root=Modeling_Project KM32ATL_KM22MM]
 
! colspan=1 | [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2m/org.eclipse.m2m.atl/tests/org.eclipse.m2m.atl.tests/data/inputs/DSLBridge/ModelBridge/KM32ATL_KM22MM/?root=Modeling_Project KM32ATL_KM22MM]
 
! colspan=1 | <b style="color:green">PASS</b>
 
! colspan=1 | <b style="color:green">PASS</b>
! colspan=1 | 0.015s.
+
! colspan=1 | 0.031s.
! colspan=1 | 0.032s.
+
! colspan=1 | 0.031s.
 
! colspan=1 |  
 
! colspan=1 |  
 +
|-
 +
! colspan=2 | Total time :
 +
! colspan=1 | 13.34s.
 +
! colspan=1 | 1.75s.
 
<!-- END TESTS -->
 
<!-- END TESTS -->
 
|}
 
|}
  
Tests realized on 04/12/2007 with :
+
Tests realized on 29/01/2008 with :
  
 
Hardware
 
Hardware

Revision as of 13:48, 29 January 2008

Intent

This page consists in a report of emfvm non-regression tests. Those tests are available on CVS and reports all performances and regression issues between the two versions of the ATL VM. Here are also centralized all development informations about the emfvm, like missing features.

Processus

Some relevant EMF-based transformations are integrated to the test set and are launched with the Regular VM and the emfvm. The results of the transformations are comparated using EMF Compare. When a test pass successfully, execution times are reported for both VMs in order to detect performance issues. If a test fails, a short description of the problem (or eventually a bug) is reported in the "Comments" field. The test is excluded from the emfvm test set until it is corrected.

Non-regression tests

Non-regression tests on emfvm
Test case Status Time Comments
emfvm Regular VM
Trace2PerformanceMetrics PASS 0.015s. 0.0s.
BibTeX2DocBook PASS 0.031s. 0.047s.
FlattenTest PASS 0.047s. 0.032s. Collections implementation completed on 12/12/2007.
XML2Book PASS 0.0s. 0.016s.
Families2Persons PASS 0.0s. 0.0s.
Ecore2Class PASS 0.0s. 0.016s.
DSL2KM3 PASS 0.032s. 0.031s.
Book2Publication PASS 0.016s. 0.015s.
Class2Relational PASS 0.0s. 0.0s.
UML2Relational FAIL 0.046s. Unsupported model elements containing "::".
DSL2XML PASS 0.094s. 0.156s.
AssertionModification PASS 0.078s. 0.063s.
XML2DSLModel PASS 0.938s. 10.203s.
XML2Ant PASS 0.25s. 2.063s.
DSLModel2KM2 PASS 0.016s. 0.031s.
Ant2Maven PASS 0.015s. 0.031s.
KM32DSL PASS 0.032s. 0.031s.
Maven2XML PASS 0.031s. 0.047s.
ATL2Problem PASS 0.046s. 0.078s. EnumLiteral support added on 25/01/2008.
XML2DSL PASS 0.078s. 0.453s.
SpreadsheetMLSimplified2Trace FAIL 0.015s. Transformation fails without errors.
KM32ATL_KM22MM PASS 0.031s. 0.031s.
Total time : 13.34s. 1.75s.

Tests realized on 29/01/2008 with :

Hardware

  • Processor : Pentium 2 GHz
  • RAM : 2 Go

Software

  • Operating system : Windows XP
  • JVM : JRE 1.6
  • Eclipse 3.3 Europa
  • ATL 2.0

emfvm missing features

Here is the list of the missing features of the emfvm. The purpose of this list is to attempt to prevent duplicate works, by signalizing current developments. So, if you are committer, or contributor, do not hesitate to update this list by shortly describing your current work (possibly with your contact):

  • Support for "metamodels"-specific EMF extensions:
    • UML2 support (e.g., reordering of calls to applyProfile(), and applyStereotype()).
  • Compilation support using one of the following alternatives:
    • Update ACG to directly generate an XML model instead of relying on specific "native types" like ASMEmitter.
    • Add the ASMEmitter type to emfvm, so that compilers generated with the current version of ACG can work.
  • Support for Java method calls (e.g., metamodel-specific methods). Note that it may be more efficient to implement them directly as Operations, instead of using reflection. One possible implementation is to provide a default version using reflection, and provide direct support for common operations (e.g., UML2 operations) later.
  • ADWP debugging protocol.
  • Exceptions logger.
  • Superimposition.
  • New bytecodes:
    • "delete": to remove an element, and all references that point to it.
    • "unset": to unset a feature (e.g., to support changing the value of a multi-valued feature).
    • "newin": to create a given element in a specific model.

The "delete" and "unset" new bytecodes will for instance be usefull to implement the refining mode of ATL 2006 (with in-place model transformation support).

How to contribute

To contribute, you can add your own test to the non-regression test. Please make sure that:

  • You are allowed to release the work under EPL.
  • The transformation works with EMF (i.e., the only model handler currently supported by emfvm).
  • The transformation correctly runs on the Regular VM.

NOTE : At this time, profiled UML2 target models are not supported. However, you can already submit any related test, in prevision of a future implementation.

To contribute a test, you can either put it into CVS (if you are committer), or contribute it on bugzilla. You may ask questions about how to do this on the M2M newsgroup, using the [ATL] prefix in your subject. You may also directly contact William Piers if you cannot follow the procedures described above.

A non-regression test is never complete, so feel free to contribute your tests!