Fuzzy Testing

PRELIMINARY DOCUMENTATION

Test Configuration

Each fuzzy test is configured with annotations. The basic structure of a test always looks like the following:

 @RunWith(ESFuzzyRunner.class)
 @DataProvider(ESEMFDataProvider.class)
 public class ESXMIProviderDataMutatorTest {
   @Data
   private EObject root;
   @Util
   private ESMutateUtil util;
   // Actual Tests follow
   [...]
 }

Let's look at each annotation:

• The Fuzzy Testing component comes with its own runner that allow to execute tests multiple times and more. Thus, if one wants to write fuzzy tests, the @RunWith(ESFuzzyRunner.class) is mandatory.
• The @DataProvider annotation specifies how to provide the data that is used within the tests. Currently, there are two types of data providers available.
  • ESEMFDataProvider randomly generates data according to the specified mutator parameters. The root object will be the type that was specified via rootEClass
  • ESXMIResourceDataProvider does not randomly generate data, but rather reads all model instances from a relative folder that must be named 'models'. Thus, the usage of rootEClass specified for a test using the ESXMIResourceDataProvider will have no effect. The data provider will read all files placed in the models folder and use them as root objects. Current restrictions are:
    • only XMI resources are allowed to be in the models folder
    • only resource.getContents().get(0) is considered when reading the resource
• The @Data annotation specifies the field that gets the data injected. Note that the types must of course confirm with the ones either specified in the XML configuration or of the object read from a XMI resource.
• The @Util annotation specifies the object that should get the utility read from the XML configuration injected.

An complete example that uses EMFStore's Bowling example model as well as the ESXMIResourceDataProvider is given below:

 @RunWith(ESFuzzyRunner.class)
 @DataProvider(ESXMIResourceDataProvider.class)
 public class ESXMIResourceDataProviderTest {
   @Data
   private EObject obj;
   @Util
   private ESMutateUtil util;
 
   /**
    * Tests if two generated models are equal.
    *
    * @throws SerializationException
    */
   @Test
   public void readProvidedModel() throws SerializationException {
     assertTrue(League.class.isInstance(obj));
   }
 
   @Test
   public void changeProvidedModel() {
     final EObject nonMutatedCopiedObject = EcoreUtil.copy(obj);
     final EObject copiedEObject = EcoreUtil.copy(obj);
 
     ESDefaultModelMutator.changeModel(getConfig(obj));
     ESDefaultModelMutator.changeModel(getConfig(copiedEObject));
 
     assertTrue(EcoreUtil.equals(obj, copiedEObject));
     assertFalse(EcoreUtil.equals(obj, nonMutatedCopiedObject));
   }
 
   private ESModelMutatorConfiguration getConfig(EObject eObject) {
     final ESModelMutatorConfiguration mmc = new ESModelMutatorConfiguration(util.getEPackages(), eObject, util.getSeed());
     mmc.setMinObjectsCount(util.getMinObjectsCount());
     return mmc;
   }
}

The configuration parameters that are used by the object annotated with @Util can either be configured progrmatically or read from a XML file, where the latter is more common. The XML file needs to be named fuzzyConfig.fuzzy and stored in a relatived folder called fuzzy. A configuration usually like this:

<org.eclipse.emf.emfstore.fuzzy.emf.config:TestConfig seed="1" count="10" testClass="org.eclipse.emf.emfstore.fuzzy.emf.test.OperationApplyTest" id="_sLQasYvPEeG1B5HDFm-I-w">
   <mutatorConfig minObjectsCount="10">
     <rootEClass href="http://eclipse.org/emf/emfstore/common/model#//Project"/>
     <eStructuralFeaturesToIgnore xsi:type="ecore:EReference" href="http://eclipse.org/emf/emfstore/common/model#//Project/cutElements"/>
     <ePackages href="http://org/eclipse/example/bowling#/"/>
   </mutatorConfig>
 </org.eclipse.emf.emfstore.fuzzy.emf.config:TestConfig>

We'll briefly look at teach element and attribute:

• seed specifies the starting seed used by the test run
• count specifies how often the test should be run
• testClass specifies the class that actually contains the tests to be run
• id is usually generated but you can also set it manually; just make sure that it is unique within the file
• the mutatorConfig element holds configuration parameters about model mutation & generation
  • the minObjectsCount field is used to specify the number of model elements to be generated within the root object. This value will be ignored if the test execution happens with the ESXMIResourceDataProvider.
  • rootEClass specifies the type of the root object
  • eStructuralFeatureToIgnore can be used to ignore certain elements during mutation/generation
  • ePackages specifies the EPackages that are considered when generation model instances. Only EClass contained in one of the packages are considered to be created.

Note that you actually don't need to write this XML file by hand, although you can of course. There's also a basic EMF editor that allows more convient editing of the configuration.

Hudson integration

To integrate the fuzzy testing into hudson three jobs are needed:

• the explorer job
• the differ job
• and the normal job

First, a fuzzy.properties file has to be added in order to specify the parameters like the hudson URL. This file has to be in the test plugin in a fuzzy/ folder:

fuzzy.hudson.url=https://hudson.eclipse.org/hudson
fuzzy.hudson.job=emf-emfstore-fuzzytests-explorer
fuzzy.hudson.diffjob=emf.emfstore-fuzzytests-differ
fuzzy.hudson.artifact.folder=/artifact/git/org.eclipse.emf.emfstore.other/

• fuzzy.hudson.url specifies the hudson url.
• fuzzy.hudson.job specifies the hudson explorer job.
• fuzzy.hudson.diffjob specifies the hudson diff job.
• fuzzy.hudson.artifact.folder specifies the folder where hudson places the artifacted results

The execution of the tests on the server depends on the building system. For buckminster a launch config for the tests has to be added and executed as JUnit tests. The explorer executes all tests in a normal way. Then the TestReports contained in the fuzzy folder of the execution root can be artifacted. In the differ job you have to execute some code to iterate over all available configuration and build the DiffReports, which is done by the following code:

DiffGenerator diffGenerator = new DiffGenerator() ;
HudsonTestRunProvider runProvider = new HudsonTestRunProvider() ;

for (TestConfig config : runProvider.getAllConfigs()) {
 runProvider.setConfig(config) ;
 TestRun[] runs = runProvider.getTestRuns();
 diffGenerator.createDiff(runs[0], runs[1]) ;
}

Again the result of this differencing has to artifacted for usage in the normal job. The same launch config used before in the explorer job has to be added, but with a special system property. To only execute the tests selected by the differ job the flag

-DfilterTests=true

is required.