Difference between revisions of "Project Download Stats"

From Eclipsepedia

Jump to: navigation, search
(HOWTO get download stats (file, domain, country) and trend plots - the easy way!)
(update urls)
Line 1: Line 1:
 
= Querying & Collating Eclipse.org Project Download Stats =
 
= Querying & Collating Eclipse.org Project Download Stats =
  
With a few simple steps, you can get [http://www.eclipse.org/emf/downloads/downloads.php download statistics] for your project like this:
+
With a few simple steps, you can get [http://www.eclipse.org/modeling/emf/downloads/downloads.php download statistics] for your project like this:
  
 
<table>
 
<table>
Line 26: Line 26:
 
The above code is in CVS here:  
 
The above code is in CVS here:  
  
''<tt style="color:DarkGreen">anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/emf/downloads/</tt>''
+
''<tt style="color:DarkGreen">anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/modeling/emf/downloads/</tt>''
  
 
Below are the three steps required to set up some or all of these tools. Note that for graphics (TLD icons and bars) you will also need to copy image files from here:
 
Below are the three steps required to set up some or all of these tools. Note that for graphics (TLD icons and bars) you will also need to copy image files from here:
  
''<tt style="color:DarkGreen">anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/emf/images/</tt>''
+
''<tt style="color:DarkGreen">anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/modeling/emf/images/</tt>''
  
 
== Step 1: Query On Demand ==
 
== Step 1: Query On Demand ==
 
 
 
1. Install ''<tt style="color:DarkGreen">stats.php</tt>'' on your website, eg. commit it to
 
1. Install ''<tt style="color:DarkGreen">stats.php</tt>'' on your website, eg. commit it to
''<tt style="color:DarkGreen">/cvsroot/org.eclipse/www/emf/downloads/stats.php</tt>'' so that  
+
''<tt style="color:DarkGreen">/cvsroot/org.eclipse/www/modeling/emf/downloads/stats.php</tt>'' so that  
it's accessible via http as http://www.eclipse.org/emf/downloads/stats.php
+
it's accessible via http as http://www.eclipse.org/modeling/emf/downloads/stats.php
  
 
2. Register it with the webmaster so that you will have access to the  
 
2. Register it with the webmaster so that you will have access to the  
Line 84: Line 84:
 
   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -dates \
 
   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -dates \
 
     `date --date="$(date +%Y-%m-%d) -1 day" +%Y%m%d` -l \
 
     `date --date="$(date +%Y-%m-%d) -1 day" +%Y%m%d` -l \
     /var/www/emf/downloads/xml/nightly 2>&1 | tee ~/crontab/logs/genXML.N.log.txt
+
     /var/www/modeling/emf/downloads/xml/nightly 2>&1 | tee ~/crontab/logs/genXML.N.log.txt
  
 
   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -dates \
 
   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -dates \
Line 96: Line 96:
 
   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -weeks \
 
   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -weeks \
 
     `date --date="$(date +%Y-%m-%d) -1 week" +%U` -l \
 
     `date --date="$(date +%Y-%m-%d) -1 week" +%U` -l \
     /var/www/emf/downloads/xml/weekly 2>&1 | tee ~/crontab/logs/genXML.W.log.txt
+
     /var/www/modeling/emf/downloads/xml/weekly 2>&1 | tee ~/crontab/logs/genXML.W.log.txt
  
 
   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -weeks \
 
   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -weeks \
Line 108: Line 108:
 
   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -months \
 
   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -months \
 
     `date --date="$(date +%Y-%m-15) -1 month" +%m` -l \
 
     `date --date="$(date +%Y-%m-15) -1 month" +%m` -l \
     /var/www/emf/downloads/xml/monthly 2>&1 | tee ~/crontab/logs/genXML.M.log.txt
+
     /var/www/modeling/emf/downloads/xml/monthly 2>&1 | tee ~/crontab/logs/genXML.M.log.txt
  
 
   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -months \
 
   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -months \
Line 120: Line 120:
 
''<tt style="color:DarkGreen">downloads.php</tt>''. This should be installed next to wherever your data is collected,  
 
''<tt style="color:DarkGreen">downloads.php</tt>''. This should be installed next to wherever your data is collected,  
 
eg., if you have a webserver with a ''<tt style="color:DarkGreen">/var/www/</tt>'' root, and you place your data into  
 
eg., if you have a webserver with a ''<tt style="color:DarkGreen">/var/www/</tt>'' root, and you place your data into  
''<tt style="color:DarkGreen">/var/www/emf/downloads/xml/</tt>'', this file should be  
+
''<tt style="color:DarkGreen">/var/www/modeling/emf/downloads/xml/</tt>'', this file should be  
''<tt style="color:DarkGreen">/var/www/emf/downloads/downloads.php</tt>''.
+
''<tt style="color:DarkGreen">/var/www/modeling/emf/downloads/downloads.php</tt>''.
  
For a real-world example, go here: http://www.eclipse.org/emf/downloads/downloads.php
+
For a real-world example, go here: http://www.eclipse.org/modeling/emf/downloads/downloads.php
  
 
2. You can customize the way the Files By Type grouping works to suit your specific file names by editing the function getFileType($url).  
 
2. You can customize the way the Files By Type grouping works to suit your specific file names by editing the function getFileType($url).  

Revision as of 14:00, 23 October 2008

Contents

Querying & Collating Eclipse.org Project Download Stats

With a few simple steps, you can get download statistics for your project like this:

Download stats some countries.gif Download stats 2week trend.gif Download stats some files.gif

This engine consists of 4 pieces:

  • stats.php
    • script to query sql tables and return html/xml
  • genXML.sh(.txt)
    • shell script to generate data from above php script
  • downloads.php
    • php script to collate/sort/filter stored data
  • xml/nightly/*.xml,

xml/weekly/*.xml, xml/monthly/*.xml

    • sample (real) data for EMF, Jan 01 - Mar 02, 2006.

The above code is in CVS here:

anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/modeling/emf/downloads/

Below are the three steps required to set up some or all of these tools. Note that for graphics (TLD icons and bars) you will also need to copy image files from here:

anonymous@dev.eclipse.org:/cvsroot/org.eclipse/www/modeling/emf/images/

Step 1: Query On Demand

1. Install stats.php on your website, eg. commit it to /cvsroot/org.eclipse/www/modeling/emf/downloads/stats.php so that it's accessible via http as http://www.eclipse.org/modeling/emf/downloads/stats.php

2. Register it with the webmaster so that you will have access to the SQL class, dbconnection_downloads_ro.class.php. Without this, the queries will fail.

3. Tweak the script using your own username/password restrictions, and set your own filenames for which to query.

4. When you are satisfied with the queries, displayed as either HTML or XML, you can now automate the nightly/weekly/monthly collection of data snapshots.

Step 2: Query On Schedule / Archived Snapshots

1. Install genXML.sh.txt on some linux-capable machine (native or cygwin). Rename it to genXML.sh and set it executable (chmod 700 genXML.sh).

2. Check that the script works - you will need bash and wget installed on your linux system. Run the script w/o options (./genXML.sh) to see its usage instructions. You can also read the script to see additional crontab examples.

3. To install the script to your crontab, edit your crontab and add entries for what you'd like to do - set a nightly, weekly, monthly schedule for when you want to collect new data. (crontab -e)

4. For example, you can copy the following 6 lines into your crontab:

crontab entries

   # nightly stats (previous day) @ 6am, do yesterday's data
   00 6 * * * ~/crontab/genXML.N.sh > /dev/null
   
   # weekly (previous week, starting on Sunday) @ 6:20am on Sunday, do previous week's data
   20 6 * * 0 ~/crontab/genXML.W.sh > /dev/null
   
   # monthly (previous full month) @ 6:40am on 1rst of the month, do prev month's data
   40 6 1 * * ~/crontab/genXML.M.sh > /dev/null

5. The above-referenced scripts are wrappers for genXML.sh. Create them thus:

~/crontab/genXML.N.sh

   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -dates \
     `date --date="$(date +%Y-%m-%d) -1 day" +%Y%m%d` -l \
     /var/www/modeling/emf/downloads/xml/nightly 2>&1 | tee ~/crontab/logs/genXML.N.log.txt

   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -dates \
     `date --date="$(date +%Y-%m-%d) -1 day" +%Y%m%d` -l \
     /var/www/uml2/downloads/xml/nightly 2>&1 | tee ~/crontab/logs/genXML.N.log.txt

~/crontab/genXML.W.sh

   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -weeks \
     `date --date="$(date +%Y-%m-%d) -1 week" +%U` -l \
     /var/www/modeling/emf/downloads/xml/weekly 2>&1 | tee ~/crontab/logs/genXML.W.log.txt

   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -weeks \
     `date --date="$(date +%Y-%m-%d) -1 week" +%U` -l \
     /var/www/uml2/downloads/xml/weekly 2>&1 | tee ~/crontab/logs/genXML.W.log.txt

~/crontab/genXML.M.sh

   ~/crontab/genXML.sh -user emf-dev -pass trilobyt3 -F -D -C -months \
     `date --date="$(date +%Y-%m-15) -1 month" +%m` -l \
     /var/www/modeling/emf/downloads/xml/monthly 2>&1 | tee ~/crontab/logs/genXML.M.log.txt

   ~/crontab/genXML.sh -uml2 -user emf-dev -pass trilobyt3 -F -D -C -months \
     `date --date="$(date +%Y-%m-15) -1 month" +%m` -l \
     /var/www/uml2/downloads/xml/monthly 2>&1 | tee ~/crontab/logs/genXML.M.log.txt

Step 3: Displaying, Comparing, Plotting & Collating Archived Snapshot Data

1. To view your stored data in different ways, you can use downloads.php. This should be installed next to wherever your data is collected, eg., if you have a webserver with a /var/www/ root, and you place your data into /var/www/modeling/emf/downloads/xml/, this file should be /var/www/modeling/emf/downloads/downloads.php.

For a real-world example, go here: http://www.eclipse.org/modeling/emf/downloads/downloads.php

2. You can customize the way the Files By Type grouping works to suit your specific file names by editing the function getFileType($url). For the EMF case, this is:

downloads.php#getFileType($url)

   function getFileType($url) {
     $matches = array(
       "Standalone Zip"  => "emf-sdo-xsd-Standalone-",
       "Full SDK Zip" 	 => "emf-sdo-xsd-SDK-",
       "EMF SDK Zip" 	 => "emf-sdo-SDK-",
       "EMF RT Zip" 	 => "emf-sdo-runtime-",
       "EMF Update Manager Jar" 	 => "org.eclipse.emf.ecore", 
       "XSD SDK Zip" 	 => "xsd-SDK-",
       "XSD RT Zip" 	 => "xsd-runtime-",
       "XSD Update Manager Jar" 	 => "org.eclipse.xsd");
	 foreach ($matches as $label => $match) {
       if (false!==strpos($url,$match)) return $label;	
     } 
     return "Other Files";
   }

--Nickb 17:45, 3 March 2006 (EST)