Skip to main content
Jump to: navigation, search

Papyrus/Papyrus Developer Guide/JUnit Test Framework

< Papyrus‎ | Papyrus Developer Guide
Revision as of 09:27, 3 March 2016 by Give.a.damus.gmail.com (Talk | contribs) (Add subsection about @Headless tests)

Papyrus provides various utilities to assist with common tasks/problems in constructing JUnit tests.

Core JUnit Test Infrastructure

The org.eclipse.papyrus.junit.utils plug-in provides the core testing framework for Papyrus.

Abstract JUnit Suite Classes

All test-suite classes are encouraged to extend (directly or indirectly) the org.eclipse.papyrus.junit.utils.tests.AbstractPapyrusTest class. Amongst potentially other future benefits, this class employs the ClassificationRunner that supports selectivity of test execution for varying levels of coverage. When it is necessary to extend some other class, the test class can be annotated with this runner.

ClassificationRunner

This is the default test runner for all test classes extending the AbstractPapyrusTest class, as described above. Other runners also support a subset of its capabilities.

This runner classifies tests into configuration sets that can be specified on the command-line to run only a certain subset of the tests at run-time (it is a declarative filtering mechanism). The categories of test available in the classification scheme are defined by the TestCategory enumeration, and each maps to a test annotation as follows:

Test Category Test Annotation Description
NotImplemented @NotImplemented The test is not yet implemented but is planned.
InvalidTest @InvalidTest The test is incorrectly specified and its failure does not imply a problem in the software.
FailingTest @FailingTest The test is failing because of a suspected problem in the software.
InteractiveTest @InteractiveTest The test requires interaction with the user.
ExpensiveTest @ExpensiveTest The test has an unusually long running time.
GeneratedTest @GeneratedTest The test or suite is generated by the diagram test generation framework.
Standard N/A The test has no special disposition.

These categories are selectively excluded by classification configurations recognized by the command-line -testConfig=<name> option, where the <name> is one of the values of the ClassificationConfig enumeration:

Classification Configuration Excluded Categories Description
CI_TESTS_CONFIG NotImplemented, InvalidTest, FailingTest, InteractiveTest, GeneratedTest Tests run by the continuous integration builds.
FAILING_TESTS_CONFIG Standard, InteractiveTest, GeneratedTest, ExpensiveTest Runs tests identified as failing (known regressions).
LIGTHWEIGHT_TESTS_CONFIG [sic] NotImplemented, FailingTest, InvalidTest, ExpensiveTest, InteractiveTest, GeneratedTest Quick subset of the tests appropriate for "smoke tests", as in Gerrit builds.
GENERATED_TESTS_CONFIG InteractiveTest, NotImplemented, FailingTest, InvalidTest, ExpensiveTest, Standard Runs the generated tests for generated diagrams.
FULL_CI_TESTS_CONFIG InteractiveTest All automatable tests (not requiring user interaction).
FULL_TESTS_CONFIG Runs absolutely all tests.

AllTestsRunner

In the Mars release, the AllTests master test-suite class in the org.eclipse.papyrus.tests bundle was refactored to extract out its flexible plug-in/fragment suite aggregation capability. The AllTestsRunner class is a JUnit runner that you can use with the @RunWith annotation to aggregate a bunch of bundle suites into a super-suite for some component/layer or other area of concern. Annotate a static field of some type conforming to Iterable<? extends ITestSuiteClass> or ITestSuiteClass[] to assemble a suite (the collection can be populated by a static initializer) and annotate that field with @SuiteSpot.

Headless Tests

Many Papyrus plug-ins offer APIs that are suitable for execution and testing in "headless" contexts: where the Eclipse Workbench UI is not present. An AllTests suite class of tests that can run headless to test a headless API should be annotated with @Headless in order that it be included automatically in the org.eclipse.papyrus.tests.AllHeadlessTests suite, which serves as a convenient fast-running (because the UI is not involved) "smoke test" suite for the core Papyrus APIs. This suite selects the subset of suites aggregated in the org.eclipse.papyrus.tests.AllTests that are annotated with @Headless.

ScenarioRunner

Sometimes we want to use JUnit to automate slightly more complex test scenarios involving a sequence of steps with verification points along the way. These might be UI tests, integration-style tests, regression tests for specific user scenarios, or tests of closely related function such as execute/undo/redo of editing commands. The common structure of such tests is that they have multiple discrete points at which they implement assertions and these assertions should be able to pass or fail independently, but splitting them out into separate tests would be cumbersome as it would repeat initialization and tear-down in each test, as well as subsets of the scenario (for example, a command undo test would always first have to execute the command and a redo test would have to execute and undo the command).

The ScenarioRunner can be specified by a test class's @RunWith annotation to implement tests of this kind of complex scenario format. This runner supports test methods using the usual @Test annotation, but also methods with a @Scenario annotation that denotes a scenario with distinct named verification points. Enclose assertions in an if block with invocation of the ScenarioRunner::verificationPoint() method as the condition to implement a verification point. Each verification point of the scenario is exposed in the JUnit model as a discrete test, using the corresponding name supplied by the @Scenario annotation, and can pass or fail independently. When some assertion in a verification point fails, that verification point reports a failure and the scenario re-starts from the beginning to evaluate the verification points following the failure, skipping all those that previously passed or failed.

An example from the Papyrus canonical-edit-policy tests. The highlighted line shows the verification point corresponding to the "undo" label in the @Scenario annotation:

  1.         @Scenario({ "execute", "undo", "redo" })
  2.         public void deleteClassViewFromPackage() {
  3.                 // Execute a command to delete some view
  4.                 delete(requireEditPart(types_subfoo));
  5.  
  6.                 if (verificationPoint()) { // The 'execute' verification point
  7.                         assertDetached(types_subfoo);
  8.                 }
  9.  
  10.                 // Undo the command. This may fail even if executing and redoing work
  11.                 undo();
  12.  
  13.                 if (verificationPoint()) { // The 'undo' verification point
  14.                         requireEditPart(types_subfoo);
  15.                         assertAttached(types_subfoo);
  16.                 }
  17.  
  18.                 // Redo the command. This may work even if undo fails
  19.                 redo();
  20.  
  21.                 if (verificationPoint()) { // The 'redo' verification point
  22.                         assertDetached(types_subfoo);
  23.                 }
  24.         }

The resulting JUnit model presents the verification points as discrete tests:

   AllTests
       ...
       CanonicalViewDeletionInClassDiagramTest
           ...
           deleteClassViewFromPackage:execute
           deleteClassViewFromPackage:undo
           deleteClassViewFromPackage:redo
           ...
       ...

The ScenarioRunner also supports the @Ignore annotation for skipping tests and the configuration annotations such as @FailingTest implemented by the ClassificationRunner (see above).

Useful JUnit Rules

The org.eclipse.papyrus.junit.utils.rules package provides some rules that help to automate common tasks.

Workspace project fixtures

  • ProjectFixture: this rule creates a project in the workspace for each test method, named according to the test method name
    • at the conclusion of the test, the rule ensures total deletion of the project
    • various convenience methods obtain URIs for paths in the project, obtain IFile for URIs, manipulate read-only state of resources, etc.

Model-set fixtures

These are useful for tests that need to work on model resources in a ModelSet of some kind:

  • AbstractModelSetFixture incorporates a ProjectFixture and creates a model in the test project from a template, in the context of a ModelSet
    • before the test starts, the project is created and a model is created from a template specified by an annotation on the test:
      • the @PluginResource annotation specifies a path, relative to the bundle root, of a resource in the plug-in to copy into the test project
      • the @JavaResource annotation specifies a path, relative to the test class's location in the Java classpath, or a resource to copy into the test project
    • the ModelSetFixture is a concrete subclass that creates a plain old ModelSet with a basic TransactionalEditingDomain with a minimal ServicesRegistry
    • at the conclusion of the test, the model-set is disposed of properly, along with its associated service registry
  • the org.eclipse.papyrus.infra.emf.readonly.tests plug-in provides two additional subclasses:
    • PapyrusROEditingDomainFixture creates a model-set with Papyrus's editing domain implementation supporting the Read-Only Framework
    • PapyrusModelSetFixture creates the highest-priority model-set contributed to the Service Registry by the extension point, according to the plug-ins currently installed

Memory leak testing

  • MemoryLeakRule provides an add(Object) API to track references to an object that must not be leaked outside the scope of the test case. After the test completes successfully (if it makes any other assertions of its own), the rule asserts that all tracked objects added to it during the execution of the test are reclaimed by the garbage collector.
    • tests that need to track objects that may be retained by SoftReferences (such as objects that are shown in views based on the Common Navigator Framework can be annotated with the nested @SoftReferenceSensitive annotation. This ensures that the rule will take extra aggressive measures to force the JVM to clear soft references before asserting that tracked objects have been reclaimed

Memory leak prevention

To ensure that a large test suite can run reliably on any system, it helps that the tests don't leak memory.

  • the HouseKeeper JUnit rule provides a simple framework for creating and tracking resources (in the generic sense of the word) that need to be disposed of automatically after execution of the test completes. It offers also special cases for:
    • creating ResourceSets and TransactionalEditingDomains for the test case to use
    • creating projects and other workspace resources
    • reflectively clearing (setting to null) fields of the test instance
      • for any test that has a HouseKeeper rule, non-final fields of type conforming to EObject are cleared automatically
  • the HouseKeeper::Static nested class may be instantiated for @ClassRule usage. This disposes of resources after completion of the entire class's test suite and clears non-final static fields of conforming-to-EObject type

Conditional test execution

  • Tests annotated with @Conditional reference conditions under which they should be executed
    • the @Condition annotation is attached to a boolean-valued method or field that is evaluated to determine whether the test is included. The annotation optionally specifies the condition name to match the @Conditional reference, if the method/field name is different for some reason
  • The ClassificationRunner used by default by AbstractPapyrusTests supports evaluation of these conditions and reports skipped tests clearly in the JDT's JUnit view (and without Hudson reporting them as failures). If this runner cannot be used (perhaps because some other runner such as Parameterized is employed), then the ConditionRule may be used to evaluate test conditions. However, this rule cannot report skipped tests in a way that Hudson will not report as test failures, so skipped tests show up (somewhat misleadingly) as green in the JUnit view when using the rule

Scheduling clean-up tasks

The ExecutorRule is a JUnit rule that implements the Java Executor API, executing tasks submitted to it after the test finishes (whether successfully or not). So, it is useful for scheduling test clean-up tasks that are not handled by other rules or fixtures.

Moreover, because it is an Executor, an instance of this rule can be passed to other APIs that take executors to schedule tasks. For example, the org.eclipse.papyrus.infra.emf.readonly.Activator provides API to assign an executor for scheduling the expiry of the resource read-only state cache (be sure to post a Runnable that restores the default executor!) and various Guava APIs for concurrent programming, such as the ListenableFuture, work with executors.

Managing Papyrus editors

Many tests need the context of a Papyrus Editor, to make assertions on the state of nested diagram/table/etc. editor panes. These tests can use the PapyrusEditorFixture rule to simplify set-up, tear-down, and manipulation of editor instances. This rule supports both the @Rule and @ClassRule usage. It provides a number of APIs for accessing and activating sub-editor panes, GMF diagram infrastructure, and more. Also, several annotations on test methods (or classes, providing defaults for contained tests) are recognized by the rule:

  • @PluginResource<tt> and <tt>@JavaResource: the editor fixture is a model-set fixture as described above, so one of these annotations is required to specify the Papyrus model that the editor is to open
  • @ActiveDiagram: names the diagram in the *.notation file that is to be made active at the start of the test. If necessary, this diagram will be opened if not already referenced in the *.di file (legacy DI resource)
  • @ShowView: names one or more views, by their registered IDs, that should be open and visible when the test starts. The views may optionally be positioned relative to some other view (or the editor area) according to the Location enumeration. For tests that do not use the PapyrusEditorFixture, the ShowViewRule may be employed instead, but it does not use the annotation and does not support the relative positioning of the view.

A corollary to the ShowViewRule is the HideViewRule that identifies a view that should be hidden/closed for the duration of a test. If the view was open when the test started, it is restored to that state at the conclusion of the test.

Back to the top