Jump to: navigation, search

Difference between revisions of "Virgo/Community/Testing Extensions"

(Test discussion initiation.)
 
(Mutilthreaded Tests)
Line 3: Line 3:
 
= Mutilthreaded Tests =
 
= Mutilthreaded Tests =
  
In testing concurrent, or multi-threaded, code the JUnit framework doesn't really help us.  The way JUnit detects assertion violations is by catching AssertionExceptions at the top level.  There are essentially three problems:
+
In testing concurrent, or multi-threaded, code the JUnit framework doesn't really help us.  The way JUnit detects assertion violations is by catching <code lang="java">AssertionException</code>s at the top level.  There are essentially three problems:
  
 
# Child threads that make assertions have the results of those assertions undetected.
 
# Child threads that make assertions have the results of those assertions undetected.
 
+
# The (child) threads can terminate before ''or after'' the main thread (which is running the JUnit test), so even if the child assertions are correctly reported, there may be no-one to report them to. (Note that the overall result of a test needs to be gathered, too, so some sort of synchronisation is vital.)
# The (child) threads can terminate before >or after< the main thread (which is running the JUnit test), so even if the child assertions are correctly reported, there may be no-one to report them to. (Note that the overall result of a test needs to be gathered, too, so some sort of synchronisation is vital.)
+
# Mock objects (which make implicit assertions during use and on verify) shared between threads can have their assertions caught (as in the answer to the first problem) but verify calls need to be made ''after'' all the child threads have finished, or at least not too soon.
 
+
# Mock objects (which make implicit assertions during use and on verify) shared between threads can have their assertions caught (as in the answer to 1.) but verify calls need to be made >after< all the child threads have finished.
+
  
 
Of course there may be more problems, too.
 
Of course there may be more problems, too.
  
I actually asked Brian Goetz about this (principal author of Java Concurrency in Practice) and he pointed me to Bill Pugh (MultithreadedTestCase), TestNG (though without much hope) and made the following intriguing suggestion:
+
I actually asked Brian Goetz about this (principal author of ''Java Concurrency in Practice'') and he pointed me to Bill Pugh (MultithreadedTestCase), TestNG (though without much hope) and made the following intriguing suggestion:
  
 
*  ...test frameworks should be able to catch this (for example, by  
 
*  ...test frameworks should be able to catch this (for example, by  

Revision as of 11:32, 4 August 2010


Mutilthreaded Tests

In testing concurrent, or multi-threaded, code the JUnit framework doesn't really help us. The way JUnit detects assertion violations is by catching AssertionExceptions at the top level. There are essentially three problems:

  1. Child threads that make assertions have the results of those assertions undetected.
  2. The (child) threads can terminate before or after the main thread (which is running the JUnit test), so even if the child assertions are correctly reported, there may be no-one to report them to. (Note that the overall result of a test needs to be gathered, too, so some sort of synchronisation is vital.)
  3. Mock objects (which make implicit assertions during use and on verify) shared between threads can have their assertions caught (as in the answer to the first problem) but verify calls need to be made after all the child threads have finished, or at least not too soon.

Of course there may be more problems, too.

I actually asked Brian Goetz about this (principal author of Java Concurrency in Practice) and he pointed me to Bill Pugh (MultithreadedTestCase), TestNG (though without much hope) and made the following intriguing suggestion:

  • ...test frameworks should be able to catch this (for example, by

creating an uncaught exception handler and watching for AssertionErrors in other threads.)

I suggest we start a test framework experiment to see what we could do to extend our JUnit test framework to allow us to make multi-threaded JUnit tests less painful.

Current Usage

There are already many multi-threaded tests in Virgo -- mainly integration tests -- which make special arrangements to gather the results (and make the assertions) on the JUnit test thread. These mechanisms are, by and large, ad hoc. It would be a good idea to try to bring them together in some way.