EASE/Unit Testing

Unit Testing with EASE

EASE comes with a full fledged unit test framework that allows to write and execute tests written in script languages. The benefit of such tests is that they are very flexible, can be adapted without the need for recompilation and deployment while still provide all the flexibility of EASE scripts like full access to the underlying application.

The following code snippets are part of our unit test tutorial available on our script repository.

The very first test

Typically (yet not mandatory) tests are hosted within a workspce project and are written as classical script files. Test functionality is provided by the Unittest module, which needs to be loaded for each test file. In its most simplest form a test may be written by simply calling some module functions:

loadModule("/Unittest", false);

startTest("Valid test", "this test will always pass");
assertTrue(true);
endTest();

The startTest()/endTest() markers help to provide some structure within a test file, but are not mandatory. Assertion methods for the typical use cases known from other Unit frameworks are provided by the Unittest module.

Test execution

To execute tests we first need to create a test.suite which essentially contains some information which tests should get executed. Therefore EASE provides a simple wizard in the File / New menu to create a file of type Scripting / Script Testsuite. Once the editor opens, the most important tab at the beginning would be the Test Selection. Here you may define which test files you are planning to execute along with the testsuite. The upper section of this page allows to provide inclusion/exclusion filters that will denote the set of files which should be part of the testsuite. When one of the textboxes looses focus, the tree selection below will refresh its content. By default all JavaScript and Python files from the current project will be included.

After having set the filters you may select the test files you want to execute. Unselected files will be marked as disabled and will still be part of test reports. To totally remove files from a testsuite, add a matching pattern to the Exclude patterns field.

Once you are satisfied with the test file selection use the run button in the upper right corner of the editor to launch your testsuite. *.suite files do have their own scripting engine, so you may run then like any other script, eg by using launch shortcuts, launch configurations or even in headless mode.

When running from the UI you will be presented with a Script Unittest view that displays execution results and allows to create reports. Note that you do get error markers on your workspace files for any assertion mismatch or thrown exception.

More details on *.suite files

Suite files may also contain variables, which will be magically injected into each executed test file. The variables injected will always be of type String, so in case you provide numbers or booleans make sure to convert them accordingly. While the UI allows to create folders and put variables into them this is a pure visual aid to get more structure in the editor. The folder part will be stripped from the variable name once it gets injected into the script engines later.

When using the functional way of providing your tests (see below for a more modern approach) any exception will terminate the execution of the current test file. This happens as an uncaught exception simply terminates the executing script engine. The test will be marked with an error then. Assertion mismatches on the other hand will result in a failure and execution continues on the next line. Therefore a single test may result in multiple failures reported from the test framework.

When executing a testsuite each test file will be launched in its own script engine, which is totally unaware of the other test files. Even the order in which test files are executed is not guaranteed. This requires your test files to be self contained, which is an important criteria from regression testing. The suite file allows to inject specific user code at certain points during test execution. On the Custom Code tab you may provide TestSuite Setup and TestSuite Teardown code which gets executed at the very beginning and at the very end of the whole testsuite. This code will run in its own, dedicated engine and will typically do sanity checks before tests get launched or does some necessary cleanup afterwards. When the setup phase fails, the whole testsuite will be stopped and no tests get executed!

TestFile Setup and TestFile Teardown do get executed when a new engine is created for each test file.

Additional test control and your own assertions

The Unittest module allows to set timing constraints on a testcase by calling setTestTimeout() somewhere within a testcase. Further you may want to provide your own assertion methods to build your personal test language. To do so simply provide a new EASE module and let your assertion functions return an IAssertion. Such return values will be automatically picked up by the unittest framework and result in pass/fail checks when executed within a test.

Modern JS Unit tests

While the pure functional way of writing tests is very simple and easy to read even for everybody you sometimes might want to have more control on your tests. Therefore we also support an OO way of writing your tests:

loadModule("/Unittest");

var modernTest = {

	// special marker to indicate this instance as unit test
	__unittest : "",

	// unit test description
	__description : 'Test class demonstrating modern testing',

	// globally ignore this instance and all its test methods
	// __ignore : 'tests ignored -> list ignored testcases',

	/**
	 * Method to be called before each test function. The name is not important,
	 * instead we are looking for our custom annotations
	 */
	setup : function() {
		// annotation indicating that this is the test setup method
		'@before';

		print("this is the test setup")
	},

	/**
	 * Method to be called after each test, even in case of errors.
	 */
	teardown : function() {
		'@after';

		print("this is the test teardown")
	},

	/**
	 * Method to be called once before all tests if this instance get executed.
	 */
	setupClass : function() {
		'@beforeclass';

		print("this is the testclass setup")
	},

	/**
	 * Method to be called once after all tests of this instance got executed.
	 */
	teardownClass : function() {
		'@afterclass';

		print("this is the testclass teardown")
	},

	/**
	 * Simple test case. The
	 * 
	 * @test marker indicates it as a test.
	 */
	valid : function() {
		'@test';
		'@description(Working testcase)';

		assertTrue(true);
	},

	/**
	 * Even after throwing further tests of this instance will get executed.
	 */
	testError : function() {
		'@description(testcase throwing an exception)';

		throw "this does not work";
	},

	/**
	 * Expect a java exception to be thrown. Test is pass when exception is
	 * detected.
	 */
	testExpectError : function() {
		'@description(testcase expecting an exception)';
		'@expect(java.lang.ClassNotFoundException)';

		java.lang.Class.forName("NonExistingClass");
	},

	/**
	 * This test will fail as the expected exception is not thrown.
	 */
	testExpectErrorButIsOK : function() {
		'@description(testcase expecting an exception)';
		'@expect(java.lang.ClassNotFoundException)';

	},

	/**
	 * Disable a test.
	 */
	testIgnored : function() {
		'@description(ignored testcase)';
		'@ignore(manually disabled testcase)';

		assertTrue(false);
	},

	/**
	 * Test failing due to exceeded execution time.
	 */
	testTimeout : function() {
		'@description(test running in a timeout)';
		'@timeout(100)';

		sleep(150);
	},
}

Unlike the functional tests test execution will continue after an exception with the next test method. On the other hand each assertion failure will result in an exception. Therefore just the first assertion failure will be visible within a test. This behavior is more in line with JUnit.

While you may execute classic and modern tests in the same testsuite, do not mix them within a single source file as then the modern tests would not get executed at all.

Tests written in Python

Python tests may be written either using the functional way as seen in the very first test or they may be written as PyUnit tests:

import unittest

class Test(unittest.TestCase):

    def setUp(self):
        print("\t\ttest setup")
        pass

    def tearDown(self):
        print("\t\ttest teardown")
        pass

    def testValid(self):
        self.assertTrue(True)
        pass

    def testInvalid(self):
        self.assertTrue(False)
        pass
    
    def testError(self):
        raise(BaseException("does not work"))
    
    @unittest.expectedFailure
    def testExpectError(self):
        raise(BaseException("expected exception"))

    @unittest.expectedFailure
    def testExpectErrorButIsOK(self):
        pass

    @unittest.skip("demonstrating skipping")
    def testIgnored(self):
        self.assertTrue(False)
        pass
    
if __name__ == "__main__":
    print("running from python directly")
    unittest.main()

The nice thing about PyUnit tests is that you may execute them right away as EASE script. When run from within a suite file they will simply use a different main method provided by EASE and redirect their output to the Script Unittest view.

Reporting and headless execution

The Unittest module provides a function to create test reports. Currently we provide reports based on JUnit xml files, which can be interpreted by CI systems like Jenkins. EASE provides an extension point where you may add your own output formatters in case you prefer more fancy reports.

To automate test execution you may either push the code for report creation to yoir TestSuite Teardown section of your suite file or use a launcher script that executes a testsuite and then writes the report to some location.

loadModule("/System/Scripting", false);
result = fork("workspace://Unit Testing Tutorial/Testsuite.suite")
result.waitForResult()

loadModule("/Unittest", false);
createReport("JUnit", result.getResult(), "C:\\Userdata\\report.xml", "My Test Report", "This is some test description");