Revision as of 14:00, 23 October 2008

Querying & Collating Eclipse.org Project Download Stats

With a few simple steps, you can get download statistics for your project like this:

This engine consists of 4 pieces:

• stats.php
  • script to query sql tables and return html/xml
• genXML.sh(.txt)
  • shell script to generate data from above php script
• downloads.php
  • php script to collate/sort/filter stored data
• xml/nightly/*.xml,

xml/weekly/*.xml, xml/monthly/*.xml

• • sample (real) data for EMF, Jan 01 - Mar 02, 2006.

The above code is in CVS here:

anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/modeling/emf/downloads/

Below are the three steps required to set up some or all of these tools. Note that for graphics (TLD icons and bars) you will also need to copy image files from here:

anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/modeling/emf/images/

Step 1: Query On Demand

1. Install stats.php on your website, eg. commit it to /cvsroot/org.eclipse/www/modeling/emf/downloads/stats.php so that it's accessible via http as http://www.eclipse.org/modeling/emf/downloads/stats.php

2. Register it with the webmaster so that you will have access to the SQL class, dbconnection_downloads_ro.class.php. Without this, the queries will fail.

3. Tweak the script using your own username/password restrictions, and set your own filenames for which to query.

4. When you are satisfied with the queries, displayed as either HTML or XML, you can now automate the nightly/weekly/monthly collection of data snapshots.

Step 2: Query On Schedule / Archived Snapshots

1. Install genXML.sh.txt on some linux-capable machine (native or cygwin). Rename it to genXML.sh and set it executable (chmod 700 genXML.sh).

2. Check that the script works - you will need bash and wget installed on your linux system. Run the script w/o options (./genXML.sh) to see its usage instructions. You can also read the script to see additional crontab examples.

3. To install the script to your crontab, edit your crontab and add entries for what you'd like to do - set a nightly, weekly, monthly schedule for when you want to collect new data. (crontab -e)

4. For example, you can copy the following 6 lines into your crontab:

crontab entries

   # nightly stats (previous day) @ 6am, do yesterday's data
   00 6 * * * ~/crontab/genXML.N.sh > /dev/null
   
   # weekly (previous week, starting on Sunday) @ 6:20am on Sunday, do previous week's data
   20 6 * * 0 ~/crontab/genXML.W.sh > /dev/null
   
   # monthly (previous full month) @ 6:40am on 1rst of the month, do prev month's data
   40 6 1 * * ~/crontab/genXML.M.sh > /dev/null

5. The above-referenced scripts are wrappers for genXML.sh. Create them thus:

~/crontab/genXML.N.sh

   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -dates \
     `date --date="$(date +%Y-%m-%d) -1 day" +%Y%m%d` -l \
     /var/www/modeling/emf/downloads/xml/nightly 2>&1 | tee ~/crontab/logs/genXML.N.log.txt

   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -dates \
     `date --date="$(date +%Y-%m-%d) -1 day" +%Y%m%d` -l \
     /var/www/uml2/downloads/xml/nightly 2>&1 | tee ~/crontab/logs/genXML.N.log.txt

~/crontab/genXML.W.sh

   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -weeks \
     `date --date="$(date +%Y-%m-%d) -1 week" +%U` -l \
     /var/www/modeling/emf/downloads/xml/weekly 2>&1 | tee ~/crontab/logs/genXML.W.log.txt

   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -weeks \
     `date --date="$(date +%Y-%m-%d) -1 week" +%U` -l \
     /var/www/uml2/downloads/xml/weekly 2>&1 | tee ~/crontab/logs/genXML.W.log.txt

~/crontab/genXML.M.sh

   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -months \
     `date --date="$(date +%Y-%m-15) -1 month" +%m` -l \
     /var/www/modeling/emf/downloads/xml/monthly 2>&1 | tee ~/crontab/logs/genXML.M.log.txt

   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -months \
     `date --date="$(date +%Y-%m-15) -1 month" +%m` -l \
     /var/www/uml2/downloads/xml/monthly 2>&1 | tee ~/crontab/logs/genXML.M.log.txt

Step 3: Displaying, Comparing, Plotting & Collating Archived Snapshot Data

1. To view your stored data in different ways, you can use downloads.php. This should be installed next to wherever your data is collected, eg., if you have a webserver with a /var/www/ root, and you place your data into /var/www/modeling/emf/downloads/xml/, this file should be /var/www/modeling/emf/downloads/downloads.php.

For a real-world example, go here: http://www.eclipse.org/modeling/emf/downloads/downloads.php

2. You can customize the way the Files By Type grouping works to suit your specific file names by editing the function getFileType($url). For the EMF case, this is:

downloads.php#getFileType($url)

   function getFileType($url) {
     $matches = array(
       "Standalone Zip"  => "emf-sdo-xsd-Standalone-",
       "Full SDK Zip" 	 => "emf-sdo-xsd-SDK-",
       "EMF SDK Zip" 	 => "emf-sdo-SDK-",
       "EMF RT Zip" 	 => "emf-sdo-runtime-",
       "EMF Update Manager Jar" 	 => "org.eclipse.emf.ecore", 
       "XSD SDK Zip" 	 => "xsd-SDK-",
       "XSD RT Zip" 	 => "xsd-runtime-",
       "XSD Update Manager Jar" 	 => "org.eclipse.xsd");
	 foreach ($matches as $label => $match) {
       if (false!==strpos($url,$match)) return $label;	
     } 
     return "Other Files";
   }

--Nickb 17:45, 3 March 2006 (EST)