Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Logging Data to Files"

 
(39 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
[[Image:STEM TOP BAR.gif]]
 +
 +
'''[http://wiki.eclipse.org/Tutorials_for_Developers Back]'''
 +
 
'''CSV Logger View'''
 
'''CSV Logger View'''
  
Line 4: Line 8:
 
   
 
   
  
'''1) Loading the CSV Logger View'''
+
==1) Loading the CSV Logger View==
  
To load or activate the CSV Logger View, first launch (and run) stem. On the top menu (See Figure 1) bar click on  
+
To load or activate the CSV Logger View, first launch (and run) STEM. On the top menu bar (See Figure 1) click on  
  
      >window>other...>CSV Logger
+
>window>other...>CSV Logger
  
 
This will open a CSV Logger Tab at the bottom of your Simulation Perspective (See Figure 2).  
 
This will open a CSV Logger Tab at the bottom of your Simulation Perspective (See Figure 2).  
Initially the view will be blank because there are no locations being monitored.
+
Initially the view will be blank because there are no scenarios being monitored.
  
FIGURE 1 and 2 go here
 
  
'''2) Testing the Logger'''
+
[[Image:Csv1.jpg|400px]][[Image:Csv2.jpg|500px]]
  
To use the logger you first have to run a scenario. You can pick either your own scenario or a built in one.  
+
'''Figure 1. Opening the CSV Logger View'''...................and..........................'''Figure 2. Starting a Scenario'''
 +
 
 +
==2) Testing the Logger==
 +
 
 +
To use the logger, you first have to activate logging from the CSV Logger tab. Click the 'On' button and the next time a scenario is started data will be logged, see figure 3.
 +
 
 +
[[Image:Csv2_5.jpg]]
 +
 
 +
'''Figure 3. Enabling Logging from the CSV Logger tab'''
 +
 
 +
 
 +
Next you pick the scenario you want to log. You can pick either your own scenario or a built in one.  
 
In the following example we use the built in scenario for the Spanish Flu in Tokyo, Japan.
 
In the following example we use the built in scenario for the Spanish Flu in Tokyo, Japan.
  
 
Go to the Scenarios Tab in either the Simulation or Designer Perspective and expand the Tabs
 
Go to the Scenarios Tab in either the Simulation or Designer Perspective and expand the Tabs
  
... ->STEM->Geography->Political->Country->Japan
+
->STEM->Geography->Political->Country->Japan
  
Double click on the built in scenario
+
Double click on the built in scenario ''Spanish Flu, Tokyo, Japan, 2000''
... Spanish Flu, Tokyo, Japan, 2000
+
 
to start it (See Figure 2).
 
to start it (See Figure 2).
  
+
Once you have started the scenario, click on the CSV Logger Tab. You will notice that the view now displays the scenario and disease being logged. You can stop logging at any time by selecting the 'Off' button in the CSV Logger tab or simply stopping the simulation from the Simulation Control tab.
  
Once you have started the scenario, click on the CSV Logger Tab. If you have never run the logger before, the logger view for the selected scenario will be blank.
 
To activate the logger double click on locations or regions on the map of the running simulation. You choose what locations to monitor (generate logs for) by clicking on specific location.
 
Right-clicking on a location will show you the context menu where you can enable or disable CSV Logging from the sub-menu Select Reports. This context menu determines which STEM Views will
 
listen to map click events. This is the same selection method used in the Report Views. 
 
  
As an alternative to adding new locations to your preferences for logging, you can also use the "Add All" button in the CSV Logger View (See Figure 3).. Please note that this button is dangerous as is will add a monitor for every
+
==3) Remembering User Preferences==
node in a running simulation. It should only be used for small simulations. For the U.S., for example, pressing this button will create log information for over 3000 locations (over 3000 separate files).
+
 
+
The "Remove All" button clears the list of locations to log and resets the view (See Figure 3).
+
 
+
'''3) Remembering User Preferences'''
+
  
 
Your selections, by scenario, will be remembered in the system preferences for the CSV Logger View. The STEM CSV Logger View uses Eclipse DialogSettings to automatically remember user preferences.  
 
Your selections, by scenario, will be remembered in the system preferences for the CSV Logger View. The STEM CSV Logger View uses Eclipse DialogSettings to automatically remember user preferences.  
This feature is always enabled and begins as soon as you enable any report view. The preferences that are "remembered" are the map locations the user was most recently monitoring with one or more report views.
+
This feature is always enabled and begins as soon as you enable any report view. The preferences that are "remembered" are the logger setting for each scenario. The user's preferences are stored on disk in a directory:
Preferences are stored independently for each type of report so the system will allow (and remember) that the user wishes to monitor, for example, one a specific set of locations for logging. The user's preferences are stored on disk in a directory:
+
  
... runtime-stem.product\.metadata\.plugins\org.eclipse.ohf.stem.util.loggers
+
'''... runtime-stem.product\.metadata\.plugins\org.eclipse.org.diseasemodels\csv'''
  
 
where the directory \runtime-stem.product is located in the same parent directory as the user's STEM Eclipse workspace (for example c:\runtime-stem.product).
 
where the directory \runtime-stem.product is located in the same parent directory as the user's STEM Eclipse workspace (for example c:\runtime-stem.product).
Line 54: Line 57:
 
...CSVLogger_Preferences.txt
 
...CSVLogger_Preferences.txt
  
You never need to edit this file directly. Once you have designated a set of locations to monitor for the logger, these will always be used for the corresponding scenario.
+
You never need to edit this file directly.  
In Figure 3 we show three locations in Japan that have been designated for logging. To add a new location simple double click on the map. To remove a location simply
+
In Figure 4 we show the disease that has been designated for logging for the Japan scenario.
click "remove".
+
  
 
   
 
   
 +
[[Image:Csv3.jpg|850px]]
 +
 +
'''Figure 4. Logging Data for a Scenario'''
 +
 +
==4) Directory Location of Logged Data==
 +
 +
The Disease State data for every location monitored is stored in a separate Comma Separated Variable file. The data directory (Figure 5) may be found under your STEM workspace folder as subfolder of "Recorded Simulations" in your project folder. For instance, if you are logging a scenario in a project called "Vermont", the log files will be found in a subfolder under this directory:
 +
 +
'''...runtime-stem.product\Vermont\Recorded Simulations\'''
 +
 +
For built-in scenarios the log files are located under this folder:
  
NOTE: If you click on a location to monitor while a simulation is actively running, you will only generate a log beginning at the time you selected the new location.
+
'''...runtime-stem.product\BuiltInScenarios\Recorded Simulations\'''
For this reason it is very important that you configure the monitored locations before the start of any important run by starting a simple "test" simulation. Once you have
+
the CSV Logger preferences configured, stop the test simulation by clicking on the red stop button and then restart your scenario to be sure you logged data for each monitored
+
location from the start time of the scenario.
+
  
FIGURE 3 goes here
+
It is possible to run multiple scenarios in STEM, or to simultaneously run more than one instance of a given scenario.
 +
Logged data is stored in a folder with the same name as the scenario just run, but this folder is place in turn in a parent folder named by the start time of the particular scenario instance.
 +
This way, if you run the scenario more than once (for example as a batch process with some parameter changed from run to run) no data will be accidentally overwritten as each scenario will have a unique time stamp. Each scenario folder is further subdivided into subfolders, one folder for each disease (the folder name is the disease name) that is part of the scenario (See Figure 5).
  
'''4) Directory Location of Logged Data'''
+
[[Image:Csv4.jpg|500px]]
  
The Disease State data for every location monitored is stored in a separate Comma Separated Variable file. The data directory (Figure 4) may be found
+
'''Figure 5. The Location (Directory) Where Files Are Stored'''
under your Eclipse runtime folder as subfolder of:
+
  
... runtime-stem.product\.metadata\.plugins\org.eclipse.ohf.stem.util.loggers\csv
+
==5) CSV Logged Data Files==
  
It is possible to run multiple scenarios in STEM, or to simultaneously run more than one instance of a given scenario. Logged data is stored in a folder with
+
In Figure 6 we show the particular data files written by the logger for the Spanish Flu disease. Separate CSV files (with extension .csv) are created for each disease state modeled by the disease. These are named using the ID of the state (e.g. S, E, I or R) followed by an underscore (_) followed by a number. The number indicates the admin level of the locations stored in the file. For instance, 3 means geographical administration level 3, i.e. US cities. A second file (runparameters.csv) is also generated. This will contain data about the run parameters for each disease.  
the same name as the scenario just run, but this folder is place in turn in a parent folder named by the start time of the particular scenario instance. This way, if you run the
+
scenario more than once (for example as a batch process with some parameter changed from run to run) no data will be accidentally overwritten as each scenario will
+
have a unique time stamp (See Figure 4).  
+
  
 +
OBSERVE: The numbers in the log file are not aggregated, so if you are running a scenario down to admin level 2 (counties in US), the data in the log files for admin level 1 and 0 should not be used.
  
FIGURE 4 goes here
+
[[Image:Csv5.jpg|750px]]
  
'''5) CSV Logged Data Files'''
+
'''Figure 6. Individual Log Files. These are created for each location or node monitored. Another file contains the parameters run for the scenario in question.
  
In Figure 5 we show the particular data files written by the logger for the three locations monitored in the example above (Figure 3). Separate CSV files (with extension .txt)
+
Each of the data files for monitored nodes contains comma separated data. The specific columns in the file represents the locations being logged. For example, for the Japan scenario the file S_1.csv has the columns:
are created for each location monitored. These are named using the unique node ID. A second file (runparameters.txt) is generated when running in batch mode. This will contain data
+
about the run parameters for each scenario. Note that if you are not running in batch mode the runparameters.txt file will be empty.
+
  
FIGURE 5 goes here
+
iteration,time,JP-KYU,JP-KNT,JP-0,JP-CHB,JP-TAI,JP-CHG,JP-SKO,JP-HRI,JP-TOH,JP-KIN,JP-RYU
  
Each of the data files for monitored nodes contains comma separated data on the running disease state. The specific columns in the file depend upon the type
+
A data sample is shown in Figure 7.
of disease model run. For example, for any standard SEIR model, the column headings will be:
+
  
            iteration, time, S, I, IRIFRE
+
  1,Fri 10 Oct 08,560042.0,245240.45466207762,5627424.0,79564.86664536313,79332.0,110140.0
 +
  2,Sun 12 Oct 08,560042.0,232427.57523514942,5627424.0,79519.11998544677,79331.93495068772,110140.0
 +
  3,Tue 14 Oct 08,560042.0,198318.63446606975,5627424.0,77166.49459938332,79326.44576007508,110140.0,115284.0
 +
  4,Thu 16 Oct 08,560042.0,104046.9798877738,5627424.0,60049.957044064824,78761.24316196528,110140.0,115284.0
  
If you are running and SIR model, the exposed (E) state does not exist and would not be logged. A data sample is shown in Figure 5.
 
  
FIGURE 6 goes here (need to re-number--two figure 5s in source
+
'''Figure 7. Sample Data from a CSV Log File'''

Latest revision as of 14:19, 21 February 2011

STEM TOP BAR.gif

Back

CSV Logger View

The current distribution of STEM provides users with the ability to output Disease State Data to a Comma Separated Variable (CSV) file. This function is enabled through a view called the CSV Logger View.


1) Loading the CSV Logger View

To load or activate the CSV Logger View, first launch (and run) STEM. On the top menu bar (See Figure 1) click on

>window>other...>CSV Logger

This will open a CSV Logger Tab at the bottom of your Simulation Perspective (See Figure 2). Initially the view will be blank because there are no scenarios being monitored.


Csv1.jpgCsv2.jpg

Figure 1. Opening the CSV Logger View...................and..........................Figure 2. Starting a Scenario

2) Testing the Logger

To use the logger, you first have to activate logging from the CSV Logger tab. Click the 'On' button and the next time a scenario is started data will be logged, see figure 3.

Csv2 5.jpg

Figure 3. Enabling Logging from the CSV Logger tab


Next you pick the scenario you want to log. You can pick either your own scenario or a built in one. In the following example we use the built in scenario for the Spanish Flu in Tokyo, Japan.

Go to the Scenarios Tab in either the Simulation or Designer Perspective and expand the Tabs

->STEM->Geography->Political->Country->Japan

Double click on the built in scenario Spanish Flu, Tokyo, Japan, 2000 to start it (See Figure 2).

Once you have started the scenario, click on the CSV Logger Tab. You will notice that the view now displays the scenario and disease being logged. You can stop logging at any time by selecting the 'Off' button in the CSV Logger tab or simply stopping the simulation from the Simulation Control tab.


3) Remembering User Preferences

Your selections, by scenario, will be remembered in the system preferences for the CSV Logger View. The STEM CSV Logger View uses Eclipse DialogSettings to automatically remember user preferences. This feature is always enabled and begins as soon as you enable any report view. The preferences that are "remembered" are the logger setting for each scenario. The user's preferences are stored on disk in a directory:

... runtime-stem.product\.metadata\.plugins\org.eclipse.org.diseasemodels\csv

where the directory \runtime-stem.product is located in the same parent directory as the user's STEM Eclipse workspace (for example c:\runtime-stem.product).

Preferences for the Time Series views are stored in a file:

...CSVLogger_Preferences.txt

You never need to edit this file directly. In Figure 4 we show the disease that has been designated for logging for the Japan scenario.


Csv3.jpg

Figure 4. Logging Data for a Scenario

4) Directory Location of Logged Data

The Disease State data for every location monitored is stored in a separate Comma Separated Variable file. The data directory (Figure 5) may be found under your STEM workspace folder as subfolder of "Recorded Simulations" in your project folder. For instance, if you are logging a scenario in a project called "Vermont", the log files will be found in a subfolder under this directory:

...runtime-stem.product\Vermont\Recorded Simulations\

For built-in scenarios the log files are located under this folder:

...runtime-stem.product\BuiltInScenarios\Recorded Simulations\

It is possible to run multiple scenarios in STEM, or to simultaneously run more than one instance of a given scenario. Logged data is stored in a folder with the same name as the scenario just run, but this folder is place in turn in a parent folder named by the start time of the particular scenario instance. This way, if you run the scenario more than once (for example as a batch process with some parameter changed from run to run) no data will be accidentally overwritten as each scenario will have a unique time stamp. Each scenario folder is further subdivided into subfolders, one folder for each disease (the folder name is the disease name) that is part of the scenario (See Figure 5).

Csv4.jpg

Figure 5. The Location (Directory) Where Files Are Stored

5) CSV Logged Data Files

In Figure 6 we show the particular data files written by the logger for the Spanish Flu disease. Separate CSV files (with extension .csv) are created for each disease state modeled by the disease. These are named using the ID of the state (e.g. S, E, I or R) followed by an underscore (_) followed by a number. The number indicates the admin level of the locations stored in the file. For instance, 3 means geographical administration level 3, i.e. US cities. A second file (runparameters.csv) is also generated. This will contain data about the run parameters for each disease.

OBSERVE: The numbers in the log file are not aggregated, so if you are running a scenario down to admin level 2 (counties in US), the data in the log files for admin level 1 and 0 should not be used.

Csv5.jpg

Figure 6. Individual Log Files. These are created for each location or node monitored. Another file contains the parameters run for the scenario in question.

Each of the data files for monitored nodes contains comma separated data. The specific columns in the file represents the locations being logged. For example, for the Japan scenario the file S_1.csv has the columns:

iteration,time,JP-KYU,JP-KNT,JP-0,JP-CHB,JP-TAI,JP-CHG,JP-SKO,JP-HRI,JP-TOH,JP-KIN,JP-RYU

A data sample is shown in Figure 7.

1,Fri 10 Oct 08,560042.0,245240.45466207762,5627424.0,79564.86664536313,79332.0,110140.0
2,Sun 12 Oct 08,560042.0,232427.57523514942,5627424.0,79519.11998544677,79331.93495068772,110140.0
3,Tue 14 Oct 08,560042.0,198318.63446606975,5627424.0,77166.49459938332,79326.44576007508,110140.0,115284.0
4,Thu 16 Oct 08,560042.0,104046.9798877738,5627424.0,60049.957044064824,78761.24316196528,110140.0,115284.0


Figure 7. Sample Data from a CSV Log File

Back to the top