Revision as of 12:50, 3 December 2013

Background

We have three separate forges, Eclipse, PolarSys, and LocationTech. Each of these forges has its own repositories, bugzilla instance, mailing list domain, forums, etc. We may add additional forges in the future.

Our current dashboard is here:

http://dash.eclipse.org/dash/commits/web-app/

This is all based on some in-house code that was written a very long time ago. It was initially concerned exclusively with CVS activity and has been extended to support SVN and Git. We have since removed CVS. We currently do nothing with every other source of data.

You'll notice that it allows for some very dynamic querying. You can dig down into relatively detailed information, like number of commits by individuals affiliated with a specific company in a specific time-frame. I don't believe that we need 1:1 matching of functionality, but it would be good to be able to permit relatively easy drill-down functionality.

Eclipse projects are structured hierarchically. At the top of the hierarchy is a collection of "top level" projects, each having zero or more subprojects. Some subprojects have subprojects of their own. The project nesting level never goes more than three levels deep.

A committer is considered "active" if they have made at least one commit in the last three months. A committer is considered "participating" if they have made at least one commit in the last six months.

Where possible, we need to associate activity with member company affiliation.

Sources of Data

The central authority regarding the sources of data is the Project Management Infrastructure (PMI). Most of the data managed by the PMI is provided by project members themselves. A project may, for example, opt to not include a Git repository in their official list. We generally discourage this, but if a project so-opts, we honour that decision.

We have a currently-undocumented API that provides forge-specific project metadata:

• http://projects.eclipse.org/json/projects/all
• http://polarsys.org/json/projects/all
• http://locationtech.org/json/projects/all

Below is a example of the data provided by this API. This example only shows a single project and only relevant data (much more data is provided, including, for example, the project description).

{
    "projects": {
        "technology.egit": {
            "bugzilla": [
                {
                    "component": "", 
                    "create_url": "https://bugs.eclipse.org/bugs/enter_bug.cgi?product=EGit", 
                    "product": "EGit", 
                    "query_url": "https://bugs.eclipse.org/bugs/buglist.cgi?product=EGit"
                }
            ],
            "dev_list": {
                "email": "egit-dev@eclipse.org", 
                "name": "egit-dev", 
                "url": "https://dev.eclipse.org/mailman/listinfo/egit-dev"
            }, 
            "forums": [
                {
                    "description": "Questions and technical discussions about how to use the Git team provider (EGit)", 
                    "name": "eclipse.egit", 
                    "url": "http://www.eclipse.org/forums/eclipse.egit"
                }
            ], 
            "id": [
                {
                    "value": "technology.egit"
                }
            ],
            "mailing_lists": [
                {
                    "email": "egit-build@eclipse.org", 
                    "name": "egit-build", 
                    "url": "https://dev.eclipse.org/mailman/listinfo/egit-build"
                }
            ], 
            "marketplace": [
                {
                    "attributes": [], 
                    "title": "EGit in Eclipse Marketplace", 
                    "url": "http://marketplace.eclipse.org/node/1336"
                }
            ], 
            "parent_project": [
                {
                    "id": "technology"
                }
            ], 
            "source_repo": [
                {
                    "name": "egit", 
                    "path": "/gitroot/egit/egit.git", 
                    "type": "git", 
                    "url": "http://git.eclipse.org/c/egit/egit.git"
                }, 
                {
                    "name": "egit-github", 
                    "path": "/gitroot/egit/egit-github.git", 
                    "type": "git", 
                    "url": "http://git.eclipse.org/c/egit/egit-github.git"
                }, 
                {
                    "name": "egit-pde", 
                    "path": "/gitroot/egit/egit-pde.git", 
                    "type": "git", 
                    "url": "http://git.eclipse.org/c/egit/egit-pde.git"
                }
            ], 
            "state": [
                {
                    "value": "Regular"
                }
            ]
        }
    }
}

The content is generated directly from live data. The nature of the data is, I believe, evident from the structure itself. As we move forward with the dashboard implementation, we will stabilize and fully document this API.

We do provide additional data, including information about the releases produced by the project, a textual description and scope, and numerous links to related projects.

We can provide the project data in other forms as necessary.

We use standard Mailman to for mailing lists, FudForum for project forums, Bugzilla for issue/bug tracking, and Git and SVN (deprecated) for source code management.

General

Must-have Features

• Automatic detection and inclusion of new sources of data

Forge Dashboard

Each of the separate forges needs its own independent dashboard. We'd also like a consolidated dashboard view.

• Liveliness assessment

Project Dashboard

Must-have Features

• Top-level Project Overview
  • A top-level project can quickly review the liveliness of all nested projects
  • Consolidated rendering of all metrics for each project
• Embeddable charts for project pages (e.g. https://projects.eclipse.org/projects/technology.egit)
  • Lifetime commit activity
  • Individual commit activity (active contributors)
  • Company commit activity (member companies with active contributors)
  • Lifetime forum/mailing list activity
  • Bugzilla activity

Nice-to-have Features

• Identify contributor links between projects
  • Projects are considered contributor-related if they share contributors (i.e. if one or more contributors contribute to both projects)
  • For any given project, identify all contributor-related projects.

IP Logs

We also use data collected by Dash to populate our IP Logs. e.g.

https://eclipse.org/projects/ip_log.php?id=rt.ecf We use dash data to determine the active committers (past and present). Basically any project committer who has ever made a single commit appears in the "active" table; all others appear in the "never active" table. The "Contributors and their Contributions" section is populated from the Git repositories and Bugzilla. For Git, we include contributions made by non-committers. From Bugzilla, we identify bug attachments marked with the iplog+ flag.

I'd also like to add some aggregation charts.

Does it make sense to include downloads and builds into the metrics gathering? e.g. gather Hudson build statistics, or include the currency of downloads in the livliness considerations?