Virgo/Community/Testing Extensions

Mutilthreaded Tests

In testing concurrent, or multi-threaded, code the JUnit framework doesn't really help us. The way JUnit detects assertion violations is by catching AssertionExceptions at the top level. There are essentially three problems:

1. Child threads that make assertions have the results of those assertions undetected.
2. The (child) threads can terminate before or after the main thread (which is running the JUnit test), so even if the child assertions are correctly reported, there may be no-one to report them to. (Note that the overall result of a test needs to be gathered, too, so some sort of synchronisation is vital.)
3. Mock objects (which make implicit assertions during use and on verify) shared between threads can have their assertions caught (as in the answer to the first problem) but verify calls need to be made after all the child threads have finished, or at least not too soon.

Of course there may be more problems, too.

I actually asked Brian Goetz about this (principal author of Java Concurrency in Practice) and he pointed me to Bill Pugh (MultithreadedTestCase), TestNG (though without much hope) and made the following intriguing suggestion:

• ...test frameworks should be able to catch this (for example, by creating an uncaught exception handler and watching for AssertionErrors in other threads.)

I suggest we start a test framework experiment to see what we could do to extend our JUnit test framework to allow us to make multi-threaded JUnit tests less painful.

Current Usage

There are already many multi-threaded tests in Virgo -- mainly integration tests -- which make special arrangements to gather the results (and make the assertions) on the JUnit test thread. These mechanisms are, by and large, ad hoc. It would be a good idea to try to bring them together in some way.