Revision as of 15:41, 20 November 2013

Background

We have three separate forges, Eclipse, PolarSys, and LocationTech. Each of these forges has its own repositories, bugzilla instance, mailing list domain, forums, etc. We may add additional forges in the future.

Our current dashboard is here:

http://dash.eclipse.org/dash/commits/web-app/

This is all based on some in-house code that was written a very long time ago. It was initially concerned exclusively with CVS activity and has been extended to support SVN and Git. We have since removed CVS. We currently do nothing with every other source of data.

You'll notice that it allows for some very dynamic querying. You can dig down into relatively detailed information, like number of commits by individuals affiliated with a specific company in a specific time-frame. I don't believe that we need 1:1 matching of functionality, but it would be good to be able to permit relatively easy drill-down functionality.

Eclipse projects are structured hierarchically. At the top of the hierarchy is a collection of "top level" projects, each having zero or more subprojects. Some subprojects have subprojects of their own. The project nesting level never goes more than three levels deep.

Where possible, we need to associate activity with member company affiliation.

Sources of Data

The central authority regarding the sources of data is the Project Management Infrastructure (PMI). Most of the data managed by the PMI is provided by project members themselves. A project may, for example, opt to not include a Git repository in their official list. We generally discourage this, but if a project so-opts, we honour that decision.

We have a currently-undocumented API that provides forge-specific project metadata:

• http://projects.eclipse.org/json/projects/all
• http://polarsys.org/json/projects/all
• http://locationtech.org/json/projects/all

Below is a example of the data provided by this API. This example only shows a single project and excludes a lot of the data provided.

{
    "projects": {
        "technology.egit": {
            "bugzilla": [
                {
                    "component": "", 
                    "create_url": "https://bugs.eclipse.org/bugs/enter_bug.cgi?product=EGit", 
                    "product": "EGit", 
                    "query_url": "https://bugs.eclipse.org/bugs/buglist.cgi?product=EGit"
                }
            ],
            "dev_list": {
                "email": "egit-dev@eclipse.org", 
                "name": "egit-dev", 
                "url": "https://dev.eclipse.org/mailman/listinfo/egit-dev"
            }, 
            "forums": [
                {
                    "description": "Questions and technical discussions about how to use the Git team provider (EGit)", 
                    "name": "eclipse.egit", 
                    "url": "http://www.eclipse.org/forums/eclipse.egit"
                }
            ], 
            "id": [
                {
                    "value": "technology.egit"
                }
            ],
            "mailing_lists": [
                {
                    "email": "egit-build@eclipse.org", 
                    "name": "egit-build", 
                    "url": "https://dev.eclipse.org/mailman/listinfo/egit-build"
                }
            ], 
            "marketplace": [
                {
                    "attributes": [], 
                    "title": "EGit in Eclipse Marketplace", 
                    "url": "http://marketplace.eclipse.org/node/1336"
                }
            ], 
            "parent_project": [
                {
                    "id": "technology"
                }
            ], 
            "source_repo": [
                {
                    "name": "egit", 
                    "path": "/gitroot/egit/egit.git", 
                    "type": "git", 
                    "url": "http://git.eclipse.org/c/egit/egit.git"
                }, 
                {
                    "name": "egit-github", 
                    "path": "/gitroot/egit/egit-github.git", 
                    "type": "git", 
                    "url": "http://git.eclipse.org/c/egit/egit-github.git"
                }, 
                {
                    "name": "egit-pde", 
                    "path": "/gitroot/egit/egit-pde.git", 
                    "type": "git", 
                    "url": "http://git.eclipse.org/c/egit/egit-pde.git"
                }
            ], 
            "state": [
                {
                    "value": "Regular"
                }
            ]
        }
    }
}

The content is generated directly from live data. The nature of the data is, I believe, evident from the structure itself. As we move forward with the dashboard implementation, we will stabilize and fully document this API.

We do provide additional data, including information about the releases produced by the project, a textual description and scope, and numerous links to related projects.

We can provide the project data in other forms as necessary.

Forge Dashboard

Each of the separate forges needs its own independent dashboard. We'd also like a consolidated dashboard view.

• Livliness assessment

Project Dashboard

I'd like to have a page with charts that a project leader can use to quickly review the liveliness all of the nested projects. I can provide more concrete examples.

This data is used to generate charts that we render on project pages. For example:

https://projects.eclipse.org/projects/technology.egit

Click on the "Participate" tab (there's also a chart on the "Engage" tab).

The "individual commit activity" chart shows the "active" developers on the project (active meaning that they've committed at least one thing in the last three months). The developer identity should be their name (fixing that is on my list).

The "organization" chart uses the same definition of active, but for organization affiliation.

The "commit activity" chart on the "Engage" tab shows absolute commit activity over the entire life of the project.

I'd be interested in including other charts regarding livlieness of the project. I'd also like to include Sonar-generated charts on the project page.

We also use data collected by Dash to populate our IP Logs. e.g.

https://eclipse.org/projects/ip_log.php?id=rt.ecf

IP Logs

We use dash data to determine the active committers (past and present). Basically any project committer who has ever made a single commit appears in the "active" table; all others appear in the "never active" table. The "Contributors and their Contributions" section is populated from the Git repositories and Bugzilla. For Git, we include contributions made by non-committers. From Bugzilla, we identify bug attachments marked with the iplog+ flag.

I'd also like to add some aggregation charts.

Does it make sense to include downloads and builds into the metrics gathering? e.g. gather Hudson build statistics, or include the currency of downloads in the livliness considerations?