Difference between revisions of "Papyrus/Papyrus Developer Guide/JUnit Test Framework"

Revision as of 09:51, 6 April 2015

Papyrus provides various utilities to assist with common tasks/problems in constructing JUnit tests.

Core JUnit Test Infrastructure

The org.eclipse.papyrus.junit.utils plug-in provides the core testing framework for Papyrus.

Abstract JUnit Suite Classes

All test-suite classes are encouraged to extend (directly or indirectly) the org.eclipse.papyrus.junit.utils.tests.AbstractPapyrusTest class. Amongst potentially other future benefits, this class employs the ClassificationRunner that supports selectivity of test execution for varying levels of coverage. When it is necessary to extend some other class, the test class can be annotated with this runner.

TBD: Description of the test classifications

AllTestsRunner

In the Mars release, the AllTests master test-suite class in the org.eclipse.papyrus.tests bundle was refactored to extract out its flexible plug-in/fragment suite aggregation capability. The AllTestsRunner class is a JUnit runner that you can use with the @RunWith annotation to aggregate a bunch of bundle suites into a super-suite for some component/layer or other area of concern. Annotate a static field of some type conforming to Iterable<? extends ITestSuiteClass> or ITestSuiteClass[] to assemble a suite (the collection can be populated by a static initializer) and annotate that field with @SuiteSpot.

ScenarioRunner

Sometimes we want to use JUnit to automate slightly more complex test scenarios involving a sequence of steps with verification points along the way. These might be UI tests, integration-style tests, regression tests for specific user scenarios, or tests of closely related function such as execute/undo/redo of editing commands. The common structure of such tests is that they have multiple discrete points at which they implement assertions and these assertions should be able to pass or fail independently, but splitting them out into separate tests would be cumbersome as it would repeat initialization and tear-down in each test, as well as subsets of the scenario (for example, a command undo test would always first have to execute the command and a redo test would have to execute and undo the command).

The ScenarioRunner can be specified by a test class's @RunWith annotation to implement tests of this kind of complex scenario format. This runner supports test methods using the usual @Test annotation, but also methods with a @Scenario annotation that denotes a scenario with distinct named verification points. Enclose assertions in an if block with invocation of the ScenarioRunner::verificationPoint() method as the condition to implement a verification point. Each verification point of the scenario is exposed in the JUnit model as a discrete test, using the corresponding name supplied by the @Scenario annotation, and can pass or fail independently. When some assertion in a verification point fails, that verification point reports a failure and the scenario re-starts from the beginning to evaluate the verification points following the failure, skipping all those that previously passed or failed.

An example from the Papyrus canonical-edit-policy tests. The highlighted line shows the verification point corresponding to the "undo" label in the @Scenario annotation:

The resulting JUnit model presents the verification points as discrete tests:

    AllTests
        ...
        CanonicalViewDeletionInClassDiagramTest
            ...
            deleteClassViewFromPackage:execute
            deleteClassViewFromPackage:undo
            deleteClassViewFromPackage:redo
            ...
        ...

Useful JUnit Rules

The org.eclipse.papyrus.junit.utils.rules package provides some rules that help to automate common tasks.

Workspace project fixtures

• ProjectFixture: this rule creates a project in the workspace for each test method, named according to the test method name
  • at the conclusion of the test, the rule ensures total deletion of the project
  • various convenience methods obtain URIs for paths in the project, obtain IFile for URIs, manipulate read-only state of resources, etc.

Model-set fixtures

These are useful for tests that need to work on model resources in a ModelSet of some kind:

• AbstractModelSetFixture incorporates a ProjectFixture and creates a model in the test project from a template, in the context of a ModelSet
  • before the test starts, the project is created and a model is created from a template specified by an annotation on the test:
    • the @PluginResource annotation specifies a path, relative to the bundle root, of a resource in the plug-in to copy into the test project
    • the @JavaResource annotation specifies a path, relative to the test class's location in the Java classpath, or a resource to copy into the test project
  • the ModelSetFixture is a concrete subclass that creates a plain old ModelSet with a basic TransactionalEditingDomain with a minimal ServicesRegistry
  • at the conclusion of the test, the model-set is disposed of properly, along with its associated service registry
• the org.eclipse.papyrus.infra.emf.readonly.tests plug-in provides two additional subclasses:
  • PapyrusROEditingDomainFixture creates a model-set with Papyrus's editing domain implementation supporting the Read-Only Framework
  • PapyrusModelSetFixture creates the highest-priority model-set contributed to the Service Registry by the extension point, according to the plug-ins currently installed

Memory leak testing

• MemoryLeakRule provides an add(Object) API to track references to an object that must not be leaked outside the scope of the test case. After the test completes successfully (if it makes any other assertions of its own), the rule asserts that all tracked objects added to it during the execution of the test are reclaimed by the garbage collector.
  • tests that need to track objects that may be retained by SoftReferences (such as objects that are shown in views based on the Common Navigator Framework can be annotated with the nested @SoftReferenceSensitive annotation. This ensures that the rule will take extra aggressive measures to force the JVM to clear soft references before asserting that tracked objects have been reclaimed

Memory leak prevention

To ensure that a large test suite can run reliably on any system, it helps that the tests don't leak memory.

• the HouseKeeper JUnit rule provides a simple framework for creating and tracking resources (in the generic sense of the word) that need to be disposed of automatically after execution of the test completes. It offers also special cases for:
  • creating ResourceSets and TransactionalEditingDomains for the test case to use
  • creating projects and other workspace resources
  • reflectively clearing (setting to null) fields of the test instance
    • for any test that has a HouseKeeper rule, non-final fields of type conforming to EObject are cleared automatically
• the HouseKeeper::Static nested class may be instantiated for @ClassRule usage. This disposes of resources after completion of the entire class's test suite and clears non-final static fields of conforming-to-EObject type

Conditional test execution

• Tests annotated with @Conditional reference conditions under which they should be executed
  • the @Condition annotation is attached to a boolean-valued method or field that is evaluated to determine whether the test is included. The annotation optionally specifies the condition name to match the @Conditional reference, if the method/field name is different for some reason
• The ClassificationRunner used by default by AbstractPapyrusTests supports evaluation of these conditions and reports skipped tests clearly in the JDT's JUnit view (and without Hudson reporting them as failures). If this runner cannot be used (perhaps because some other runner such as Parameterized is employed), then the ConditionRule may be used to evaluate test conditions. However, this rule cannot report skipped tests in a way that Hudson will not report as test failures, so skipped tests show up (somewhat misleadingly) as green in the JUnit view when using the rule