Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "EclipseLink/Development/Testing/Frameworks"

(Requirements)
 
(21 intermediate revisions by 2 users not shown)
Line 1: Line 1:
__NOTOC__
+
EclipseLink has a significant number of tests, and supports many different test configurations, platforms, databases and parsers that need to be tested.
  
== Requirements ==
+
This article attempts to detail each component's testing requirements, framework and strategy.
  
* simple unit-testing inside Eclipse IDE environment
+
== Components ==
* support nightly test runs
+
Each EclipseLink component defines its own tests under <component>/eclipselink.<component>.test.  Each component has different testing requirements so uses slightly different methodologies.
* support running tests against eclipselink.jar
+
 
** report pass/fail/ignore
+
The goals for test consistency across the components include the following:
** post results to EclipseLink dashboard
+
* Each component defines an Eclipse test project in the root of the test directory.
** compare results to 'known' good results
+
* The test project should include pre-configured launch targets for running/debugging the tests, ideally through the Eclipse JUnit runner.
* must be able to run tests within multiple servers
+
* All tests should be runnable through ant, and scripted in the <test>/build.xml.
* object-to-relational tests
+
* The ant build file should define any test configuration in a <test>/test.properties file.
** must be able to run tests against different databases
+
* The test.properties should be able to be overwritten in {user.home}/test.properties.
* object-to-XML tests
+
* Ideally all tests should be written in JUnit 3.
** must be able to run tests against different XML parsers
+
* The ant test run should output results in html format in a <test>/report directory.
* <s>must be able to run CTS nightly and publish CTS results</s>
+
* The ant build file should define targets for test, test.srg and test.lrg.
** CTS cannot be run outside of Oracle; we can publish high-level 'pass/fail' results, but cannot say how many tests passed or failed, nor how many tests there are in the CTS
+
* A root test.srg and test.lrg will be defined to run each srg/lrg in each component.
* each component has unique testing requirements so will need its own framework extensions (JPA, POJO, OX, JAXB, web-services)
+
* need to be able to run regression performance and concurrency tests
+
* need to be able to runs tests with spring
+
  
 
== Current testing frameworks ==
 
== Current testing frameworks ==
  
''Yikes! <s>Nine</s> '''Ten''' different testing frameworks and their extensions''
+
=== Core ===
: Yikes what.  Each component of EclipseLink has very different usages, supports different APIs and absolutely requires their own testing extensions, it would be incredible illogical not to reuse common testing functionality across the test cases.  Most are based on JUnit3 so it is not as if they have nothing in common.
+
* framework: org.eclipse.persistence.testing.framework.**
=== core/POJO: TopLink Testing Framework (1) ===
+
* ported to EclipseLink: org.eclipse.testing.framework.**
+
 
* legacy framework that has been migrated to be an extension of JUnit3
 
* legacy framework that has been migrated to be an extension of JUnit3
 
* provides UI runner with useful debugging functionality (SQL, Java, session properties, cache, logging, individual test running, storage of results in result database)
 
* provides UI runner with useful debugging functionality (SQL, Java, session properties, cache, logging, individual test running, storage of results in result database)
Line 31: Line 26:
 
* provides advanced performance and concurrency tests which rely on result database storage
 
* provides advanced performance and concurrency tests which rely on result database storage
 
* tests are fully JUnit3 compatible and runnable in any JUnit3 runner including Eclipse
 
* tests are fully JUnit3 compatible and runnable in any JUnit3 runner including Eclipse
* provides command line runner and ant scripts (ant scripts not currently ported to EclipseLink)
+
* provides command line runner and ant scripts
* ant scripts for cross database testing (ant scripts not currently ported to EclipseLink)
+
* ant scripts for cross database testing
 
* provides extended functionality for accessing POJO Session, table creation/population, object comparison, transactional tests, etc.
 
* provides extended functionality for accessing POJO Session, table creation/population, object comparison, transactional tests, etc.
 
* 1000's of existing tests, POJO, cache-coordination, remote, POJO performance, JPA performance, POJO concurrent, MW integration, sessions XML, EIS, XDB, etc.
 
* 1000's of existing tests, POJO, cache-coordination, remote, POJO performance, JPA performance, POJO concurrent, MW integration, sessions XML, EIS, XDB, etc.
Line 42: Line 37:
 
:* provide UI many developers are dependent on for debugging
 
:* provide UI many developers are dependent on for debugging
 
:* should not be adding any new tests for POJO API
 
:* should not be adding any new tests for POJO API
 +
:* ant test targets required for non LRG tests
 +
:* need to be able to run tests in the server
  
=== core/server: Server Testing Framework (2) ===
+
=== Core : server ===
 
* (not currently ported to EclipseLink)
 
* (not currently ported to EclipseLink)
 
* framework: oracle.toplink.testing.ejb.testframework
 
* framework: oracle.toplink.testing.ejb.testframework
Line 60: Line 57:
 
:* should not be adding any new tests for POJO API or CMP
 
:* should not be adding any new tests for POJO API or CMP
  
=== legacy CMP3/JPA (3) ===
+
----
* framework: org.eclipse.testing.jpa.EntityContainerTestBase/CMP3TestModel
+
 
 +
=== JPA : legacy ===
 +
* framework: org.eclipse.persistence.testing.jpa.EntityContainerTestBase/CMP3TestModel
 
* extension of POJO framework for JPA tests
 
* extension of POJO framework for JPA tests
 
* compatible with JUnit3
 
* compatible with JUnit3
Line 69: Line 68:
 
:* not too many tests, should be port to new JPA JUnit3 framework
 
:* not too many tests, should be port to new JPA JUnit3 framework
  
=== JUnit3 ===
+
=== JPA ===
'''Used by many components, each with their own extensions to locate resources, setup sessions, etc.'''
+
* Framework: org.eclipse.persistence.testing.framework.*/junit.JUnitTestCase
 +
* Extension of JUnit3 for running JPA tests.
 +
* Ant scripts for running tests with multiple weaving options.
 +
* 1000's of existing tests, JPA, weaving, static weaving, orm XML, annotations, EclipseLink annotations.
  
==== JPA (4) ====  
+
: '''Issues'''
* framework: org.eclipse.testing.framework.junit.JUnitTestCase/framework.*/framework.server.*
+
:* Tests need to verify results, currently most test write objects, but do not read them back and ensure they were persisted correctly.
* extension of JUnit3 for running JPA tests
+
:* Require an SRG suite.
* ant scripts for running tests with multiple weaving options (ant scripts not currently ported to eclipselink)
+
 
* recently extended to allow tests to run in JEE servers (not checked in yet)
+
=== JPA : server ===
* 1000's of existing tests, JPA, weaving, orm XML, SessionBean (not checked in yet)
+
* Framework: org.eclipse.persistence.testing.framework.junit.JUnitTestCase/framework.*/framework.server.*
 +
* Extension of JUnit3 for running JPA tests on a JEE server using a EJB 3 SessionBean.
 +
* Generic Ant scripts for running tests on any server, including scripts for OC4J.
 +
* Can run any JPA test, also includes tests for SessionBeans.
  
 
: '''Issues'''
 
: '''Issues'''
:* tests need to be migrated to allow running in JEE server
+
:* JPA tests need to be migrated to allow running in JEE server (redirect transactions, entity manager access)
:* tests need to verify result
+
:* Scripts required for other supported JEE servers (WLS, WAS, JBoss, Glassfish).
 +
:* Also need to support running tests with Spring, (Tomcat?).
 +
:* Ant test targets for all JPA tests.
 +
:* Require clustering and stress tests.
  
==== Orajtst (5) ====
+
=== JPA : Orajtst ===
  
 
* not currently ported to eclipselink
 
* not currently ported to eclipselink
Line 98: Line 106:
 
:* once JPA JUnit tests are migrate to support running in server these can be removed
 
:* once JPA JUnit tests are migrate to support running in server these can be removed
  
==== OXM (6) ====
+
----
* framework: org.eclipse.testing.ox.XMLTestCase/*
+
 
 +
=== MOXy : OXM ===
 +
* framework: org.eclipse.persistence.testing.ox.XMLTestCase/*
 
* extension of JUnit3 for running OXM tests
 
* extension of JUnit3 for running OXM tests
 
:* appears to be stateless; could be migrated to almost any other framework
 
:* appears to be stateless; could be migrated to almost any other framework
* ant scripts for running tests (not yet ported to eclipselink)
+
* ant scripts for running tests
 
* 1000's of existing tests, SAX, DOM
 
* 1000's of existing tests, SAX, DOM
 
:* EISMappingTestCases(6b) extends OXTestCase
 
:* EISMappingTestCases(6b) extends OXTestCase
* sub framework:  org.eclipse.testing.oxm.OXTestCase (same tests are run multiple times)
+
* sub framework:  org.eclipse.persistence.testing.oxm.OXTestCase (same tests are run multiple times)
 
** Tests are run once using a SAX parser
 
** Tests are run once using a SAX parser
 
** Tests are run once using a DOM parser
 
** Tests are run once using a DOM parser
Line 111: Line 121:
 
** Tests are run once using deployment XML as the source of the metadata
 
** Tests are run once using deployment XML as the source of the metadata
  
==== JAXB (7) ====
+
=== MOXy : JAXB ===
* framework: org.eclipse.testing.ox.jaxb
+
* framework: org.eclipse.persistence.testing.ox.jaxb
 
* extension of OXM tests for running JAXB tests
 
* extension of OXM tests for running JAXB tests
* ant scripts for running tests (not yet ported to eclipselink)
+
* ant scripts for running tests
 
* 1000's of existing tests
 
* 1000's of existing tests
  
==== SDO (8) ====
+
=== SDO ===
* framework: org.eclipse.sdo.testing
+
* framework: org.eclipse.persistence.sdo.testing
 
* extension of JUnit3 for running SDO tests
 
* extension of JUnit3 for running SDO tests
 
* ant scripts for running tests (not yet ported to eclipselink)
 
* ant scripts for running tests (not yet ported to eclipselink)
Line 126: Line 136:
 
** Tests are run generating static classes from the XML Schema
 
** Tests are run generating static classes from the XML Schema
  
=== JUnit4 PB4 (9) ===
+
----
 +
 
 +
=== DBWS ===
 
Used by DBWS components, some misc. core tests
 
Used by DBWS components, some misc. core tests
 
* extension of JUnit4: use @RunWith annotation
 
* extension of JUnit4: use @RunWith annotation

Latest revision as of 10:30, 20 December 2007

EclipseLink has a significant number of tests, and supports many different test configurations, platforms, databases and parsers that need to be tested.

This article attempts to detail each component's testing requirements, framework and strategy.

Components

Each EclipseLink component defines its own tests under <component>/eclipselink.<component>.test. Each component has different testing requirements so uses slightly different methodologies.

The goals for test consistency across the components include the following:

  • Each component defines an Eclipse test project in the root of the test directory.
  • The test project should include pre-configured launch targets for running/debugging the tests, ideally through the Eclipse JUnit runner.
  • All tests should be runnable through ant, and scripted in the <test>/build.xml.
  • The ant build file should define any test configuration in a <test>/test.properties file.
  • The test.properties should be able to be overwritten in {user.home}/test.properties.
  • Ideally all tests should be written in JUnit 3.
  • The ant test run should output results in html format in a <test>/report directory.
  • The ant build file should define targets for test, test.srg and test.lrg.
  • A root test.srg and test.lrg will be defined to run each srg/lrg in each component.

Current testing frameworks

Core

  • framework: org.eclipse.persistence.testing.framework.**
  • legacy framework that has been migrated to be an extension of JUnit3
  • provides UI runner with useful debugging functionality (SQL, Java, session properties, cache, logging, individual test running, storage of results in result database)
  • UI can also run any JUnit3 test, and allows JPA JUnit3 tests to be run and be debuggable.
  • provides advanced performance and concurrency tests which rely on result database storage
  • tests are fully JUnit3 compatible and runnable in any JUnit3 runner including Eclipse
  • provides command line runner and ant scripts
  • ant scripts for cross database testing
  • provides extended functionality for accessing POJO Session, table creation/population, object comparison, transactional tests, etc.
  • 1000's of existing tests, POJO, cache-coordination, remote, POJO performance, JPA performance, POJO concurrent, MW integration, sessions XML, EIS, XDB, etc.
Issues
  • not 'normal' JUnit3 tests
  • too many tests to port to anything new
  • contain functionality required for performance regression testing
  • provide UI many developers are dependent on for debugging
  • should not be adding any new tests for POJO API
  • ant test targets required for non LRG tests
  • need to be able to run tests in the server

Core : server

  • (not currently ported to EclipseLink)
  • framework: oracle.toplink.testing.ejb.testframework
  • extension of POJO framework for running in JEE servers
  • not currently compatible with JUnit3
  • command line runner, ant scripts for building, deployment, and running
  • provides extended functionality for JTA transactions, JEE platform independence
  • supports running of existing POJO tests inside the JEE server
  • ant scripts for cross server testing, clustering
  • 1000's of existing tests, CMP (not part of EclipseLink), SessionBeans, BMP (not part of EclipseLink), CMP performance (not part of EclipseLink), JTA, resource, EIS/JCA, class-loader, clustering, stress
Issues
  • not compatible with JUnit3
  • too many tests to port to anything new, but most tests are CMP and obsolete
  • contain functionality required for performance regression testing in server
  • should not be adding any new tests for POJO API or CMP

JPA : legacy

  • framework: org.eclipse.persistence.testing.jpa.EntityContainerTestBase/CMP3TestModel
  • extension of POJO framework for JPA tests
  • compatible with JUnit3
  • 100's of tests
Issues
  • not too many tests, should be port to new JPA JUnit3 framework

JPA

  • Framework: org.eclipse.persistence.testing.framework.*/junit.JUnitTestCase
  • Extension of JUnit3 for running JPA tests.
  • Ant scripts for running tests with multiple weaving options.
  • 1000's of existing tests, JPA, weaving, static weaving, orm XML, annotations, EclipseLink annotations.
Issues
  • Tests need to verify results, currently most test write objects, but do not read them back and ensure they were persisted correctly.
  • Require an SRG suite.

JPA : server

  • Framework: org.eclipse.persistence.testing.framework.junit.JUnitTestCase/framework.*/framework.server.*
  • Extension of JUnit3 for running JPA tests on a JEE server using a EJB 3 SessionBean.
  • Generic Ant scripts for running tests on any server, including scripts for OC4J.
  • Can run any JPA test, also includes tests for SessionBeans.
Issues
  • JPA tests need to be migrated to allow running in JEE server (redirect transactions, entity manager access)
  • Scripts required for other supported JEE servers (WLS, WAS, JBoss, Glassfish).
  • Also need to support running tests with Spring, (Tomcat?).
  • Ant test targets for all JPA tests.
  • Require clustering and stress tests.

JPA : Orajtst

  • not currently ported to eclipselink
  • framework: oracle.toplink.testing.tests.internal.ejb, oraj.jar
  • extension of JUnit3 and oraj framework
  • ant scripts for running tests
  • cannot current run outside of ADE environment nor from IDE
  • 1000's of existing tests, clone of JPA JUnit tests ported to JEE, advanced JPA, spring
Issues
  • clone of existing tests, causes major maintenance effort keeping in synch
  • clone prevent news tests from being run in server
  • oraj and ADE dependency
  • once JPA JUnit tests are migrate to support running in server these can be removed

MOXy : OXM

  • framework: org.eclipse.persistence.testing.ox.XMLTestCase/*
  • extension of JUnit3 for running OXM tests
  • appears to be stateless; could be migrated to almost any other framework
  • ant scripts for running tests
  • 1000's of existing tests, SAX, DOM
  • EISMappingTestCases(6b) extends OXTestCase
  • sub framework: org.eclipse.persistence.testing.oxm.OXTestCase (same tests are run multiple times)
    • Tests are run once using a SAX parser
    • Tests are run once using a DOM parser
    • Tests are run once using document preservation
    • Tests are run once using deployment XML as the source of the metadata

MOXy : JAXB

  • framework: org.eclipse.persistence.testing.ox.jaxb
  • extension of OXM tests for running JAXB tests
  • ant scripts for running tests
  • 1000's of existing tests

SDO

  • framework: org.eclipse.persistence.sdo.testing
  • extension of JUnit3 for running SDO tests
  • ant scripts for running tests (not yet ported to eclipselink)
  • 1000's of existing tests
  • Sub Framework: Loading metadata from XML Schema
    • Tests are run using generic DataObjects
    • Tests are run generating static classes from the XML Schema

DBWS

Used by DBWS components, some misc. core tests

  • extension of JUnit4: use @RunWith annotation
Issues
  • newest framework, fewest tests (~200)
  • more 'future-proof' than JUnit3 which is now end-of-life

Back to the top