Running an Automated Experiment
If you're trying to fit a disease model to a reference, for instance to actual incidence data collected by public health surveillance, manually determining the parameter values for a good fit can be very time consuming and often impossible. That's where the Automated Experiment feature in STEM comes in handy.
First, to familiarize yourself with automated experiments in STEM, download the sample project available here:
and also read the documentation for this example here:
An an automated experiment will run a sequence of STEM simulations, each time varying parameters of the model and compare to a reference. It tries to minimize a error function, calculated by comparing the output (simulation log) to the data in a reference folder (where data is stored in the same format as for STEM log files). It walks the parameters of the model until it is unable to improve upon the calculated error beyond a specified tolerance. At that point the automated experiment either halts, and the optimal set of parameters found can be retrieved from the "Current Values" view on in the Automatic Experiment STEM perspective, or it restarts itself using an initial set of parameter values matching the best parameter values found so far. If the automatic experiment converges to the same set of parameter values (and error) a second time, the automatic experiment is done.
When an automatic experiment is started, STEM automatically switches to the Automatic Experiment Perspective. This perspective has a view with five tabs:
To create a new automated experiment in STEM, click the "New Automatic experiment" toolbar icon. A new wizard shows up similar to the one in the figure below.
First, specify a name for your new automated experiment. You can give it any name you want. Next, pick the algorithm to use. Currently there is only one option available, "AutomaticExperiment" which is an implementation of a Nelder Mead downhill simplex numerical optimization algorithm. See this [page] for a description of this algorithm.
Currently, there is no UI component in STEM that allows you to create an automated experiment. If you want to create one, you need to copy the AutomatedExperiment.automaticexperiment XML file from the sample project above into the experiments folder of your project. This is what the file looks like:
<?xml version="1.0" encoding="ASCII"?> <org.eclipse.stem.analysis.automaticexperiment:AutomaticExperiment xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.stem.analysis.automaticexperiment="http:///org/eclipse/stem/core/automaticexperiment.ecore" uRI="platform:/resource/AutomatedExperimentExample/experiments/AutomatedExperiment.automaticexperiment" typeURI="stemtype://org.eclipse.stem/identifiable209" errorAnalysisAlgorithm="Nedler-Mead Algorithm" errorFunction="Simple error function" tolerance="1E-6" referenceDataDir="platform:/resource/AutomatedExperimentExample/Recorded Simulations/Reference" reInit="true" maximumNumberOfIterations="1000"> <dublinCore title="Automatic Experiment Example" identifier="platform:/resource/AutomatedExperimentExample/experiments/AutomatedExperiment.automaticexperiment" description="An example automated experiment" creator="mesika" format="http:///org/eclipse/stem/core/experiment.ecore" type="stemtype://org.eclipse.stem/identifiable209" created="2010-09-01"/> <baseScenario href="../scenarios/AutomatedExperiment.scenario#/"/> <parameters initialValue="0.8" step="0.2" featureName="transmissionRate" lowerBound="0.001" upperBound="10.0" targetURI="platform:/resource/AutomatedExperimentExample/decorators/SIR.standard" /> <parameters initialValue="0.2" step="0.3" featureName="recoveryRate" lowerBound="0.001" upperBound="2.0" targetURI="platform:/resource/AutomatedExperimentExample/decorators/SIR.standard" /> <parameters initialValue="20" step="30" featureName="inoculatedPercentage" lowerBound="0" upperBound="100" targetURI="platform:/resource/AutomatedExperimentExample/decorators/SIRInoculator.standard"/> </org.eclipse.stem.analysis.automaticexperiment:AutomaticExperiment>
Next, you need to edit the file to fit your project environment. The following XML attributes and elements need to be updated:
- uRI Change this attribute so that it matches the name of your project. If you renamed the automatic experiment file, you also need to change the name here.
- identifier (inside dublinCore). Same here, you need to change the value to match your project name and automatic experiment name.
- baseScenario href Change the value to match the name of the base scenario used for running the experiment
- referenceDataDir This is the directory containing the incidence reference data. The directory must contain the same files that STEM generates in its log output folders, e.g. an I_2.csv, Incidence_2.csv, S_2.csv etc. The only file used is the Incidence file, but the other files must be created (you can leave the values as 0). If the reference files are located in the project, we recommend using the environment neutral "platform:..." format to specify the location. This way the project can be shared with others.
- parameters Create as many parameters elements you need for your particular scenario. Make sure the featureName, initialValue, step. lowerBound and upperBound contain valid values.
OBSERVE: It is currently not possible to create an automated experiment that changes a multi-valued parameter such as recovery rate or transmission rate in a multi-population disease model
You can configure various parameters for the Nelder Mead algorithm in this file. The parameters are:
- tolerance The error tolerance used to determine when the algorithm stops. If the calculated error has converged to a (local or global) minimum, the variation in the error is small. If the variation is smaller than the tolerance, the algorithm is done.
- reInit If reinit is true, the algorithm will re-start itself once its converged using the current best parameter values found. The parameter step sizes are reset to their original values. If the algorithm converges to the same minimum again (same error value), the algorithm, stops. If reinit is false, the algorithm stops after the first convergence. We recommend setting re-init to true to reduce the risk of getting stuck in a local minimum.
- maximumNumberOfIterations If this number of simulations has run without convergence to an error variation smaller than the tolerance, the algorithm stops or restarts itself (depending upon the setting of reInit)
Often convergence to a global minima can be reached quicker by manually restarting the automated experiment on the Controls tab. If it seems that the error variation is small, restart the algorithm again using the best set of parameters. This can often speed things up compared to waiting for the error variation to fall below the tolerance.