Virgo/Community/Testing Extensions

Mutilthreaded Tests

In testing concurrent, or multi-threaded, code the JUnit framework doesn't really help us. The way JUnit detects assertion violations is by catching AssertionExceptions at the top level. There are essentially three problems:

1. Child threads that make assertions have the results of those assertions undetected.
2. The (child) threads can terminate before or after the main thread (which is running the JUnit test), so even if the child assertions are correctly reported, there may be no-one to report them to. (Note that the overall result of a test needs to be gathered, too, so some sort of synchronisation is vital.)
3. Mock objects (which make implicit assertions during use and on verify) shared between threads can have their assertions caught (as in the answer to the first problem) but verify calls need to be made after all the child threads have finished, or at least not too soon.

Of course there may be more problems, too.

I actually asked Brian Goetz about this (principal author of Java Concurrency in Practice) and he pointed me to Bill Pugh (MultithreadedTestCase), TestNG (though without much hope) and made the following intriguing suggestion:

• ...test frameworks should be able to catch this (for example, by creating an uncaught exception handler and watching for AssertionErrors in other threads.)

I suggest we start a test framework experiment to see what we could do to extend our JUnit test framework to allow us to make multi-threaded JUnit tests less painful.

Current Usage

There are already many multi-threaded tests in Virgo -- mainly integration tests -- which make special arrangements to gather the results (and make the assertions) on the JUnit test thread. These mechanisms are, by and large, ad hoc. It would be a good idea to try to bring them together in some way.

Hints and ways ahead

That's an ambitious goal and many headaches have already happened. I think we should explore several solutions (old ones and new ones) :

• (old one) Brian Goetz mentionned in Java Concurrency in Practice that for the JSR 166 (creating the java.util.concurrent package) a test framework was created to handle assertions error in generated threads. I found back the class : JSR166TestCase and we should take a look;

• We can handle these problems using ThreadGroups exceptions handling, but exceptions will be ducked back into the "root" ThreadGroup. Killing potentially generated Threads;

• Study how the Junit extension [MultithreadedTestCase] handles it and how we can learn from it;

• (new one) Using aspects to watch for AssertionsError, no matter what thread will launch it. This is the less intrusive way but its scope must be carefully studied. (Binary weaving can be painfull (especially on the 16 000 jdk classes), i think we should use Load Time Weaving as much as possible but it has to be tested).

We can improve greatly the existing test framework and the way Virgo tests handle thread-safety and performance issues.