Summer Vacation 2007 IPzilla Improvements

Goals

The goal of our Year 2007 Summer Vacation IPzilla Improvements project is to improve the user experiences of both the Eclipse committers and the Eclipse Legal staff without changing the Eclipse IP Policy. Although there are many issues around the IP Policy and process, we have carefully selected high-value items within reach of our resources. Other issues will be addressed by other groups (e.g., the IP Working Group and the Board) and/or other future tools and documentation.

The three major pain points we are addressing this summer are:

1. Unclear Process. Careful definition of the states and transitions, fields and responsibilities that an IP bug goes through.
2. Unclear Delays. Notifications to responsible parties when an IP bug is waiting for their input.
3. Difficult To Submit. An Eclipse client plug-in that pre-processes and correctly formats one or more CQ submissions.

Project Plan

(1) Upgrade IPZilla to Bugzilla 3.0

The first step is to upgrade IPZilla to 3.0 so that we can take advantage of new features. bug 191528

(2) New Fields

(2.1) Custom Fields

• License: bug 166688 (Custom Field)
• PMC Approver: bug 169918 (Custom Field)
• Shipping binary or source: bug 171460 (Keyword?)
• Modified or unmodified: bug 171460 (Keyword?)
• Entire binary or subset: bug 171460 (Keyword?)
• Request Parallel IP: (Keyword?) default is yes for incubation conforming projects; forced to no (with an explanation) for non-conforming projects
• Orbit Bundle: (Custom Field) the location of the Orbit bundle where this third-party package is stored for reuse
• Request URL where mechanism by which developers contributing to the project have agreed to have their contribution licensed to others under the projects license.
• Ask if the submission is an update to a previously approved package (yes/no). If yes, link to previously approved version.

(2.2) Real Requestor

In addition to custom fields, we need to change the requestor to be the actual requestor, but at the same time not give that person extra privileges on bug that they don't already have. This will allow experience bugzilla users (i.e., our Committers) to search through IPzilla using "My Bugs" and other similar queries. bug 194656

(2.3) PMC Mailing List

Another addition is that the PMC approval person should be changed from a person to the appropriate PMC mailing list.

(3) Protect Fields

Change the IPzilla pages so that people cannot edit the parts of the bug that they are not allowed to. For example, today people can try to add "depends on" or change the "state" only to be rejected on a submit. Change this so that these fields are read-only unless the person is on the Legal team. See Field Ownership below.

(4) Add Quick Action Buttons

Change the IPZilla web pages so that the common transitions have dedicated buttons rather than requiring the user to click here, drop-down there, etc, and then press the single submit button. In the section below, means a quick action button to perform this task within the bug, without submitting the changes to the server; means a revised submit button that submits the comment (or a default comment if none is entered) and changes the state; means a revised submit button that submits the comment and changes the state back to the previous state. bug 195174

Ideas for implementing :

• Perhaps query the bug's history to find the previous state value, or
• Maybe a new table to remember the previous state and code to populate that table, or
• Possibly the easiest solution would be to split the [awaiting_committer] and similar states into multiple states based on where they came from. Thus [triage] -> [awaiting_committer_from_triage] and then goes back to [triage]. And [under_analysis] -> [awaiting_committer_from_analysis] -> [under_analysis]. Etc.

(5) Automatically Run IP Scan Tool

In the [new] state described below, once the IP bug has sufficient approvals and source code attachments, automatically run the IBM scan tool against the attachments and save the logs as an attachment and comment on the bug. bug 166876

(5.1) Automatically Transition Out Of [new]

See state diagram below for logical expression to trigger these transitions.

(6) Change Emails

There are two aspects of our current emails that we want to change:

1. Change the wording of the emails to be more developer-centric.
2. Use the portal system to send reminder emails of pending status.

(6.1) Unique IPzilla Style

First, we should change the generic email message sent by IPzilla so that it has a different look than the generic bugzilla email. This could be as simple as including an ASCII-art header of IPzilla at the top. The goal here is for the developers who get a lot of bugzilla email to be able to identify that IPzilla email is different.

(6.2) Simpler Initial IPzilla Email

Second, we should change the text of the emails sent by IPzilla, particularly that first email that is sent. It's a very long and complex email right now, attempting to tell the users the whole process they need to follow all in one email. Practice has shown that the long email is basically being ignored by the readers. We should examine all the emails and plan a strategy for what we want to say to people when.

One task here is to examine all current IPzilla emails and write a spec/plan for better communications (e.g., the notification when "[new] AND has source code" should be directed at the PMC (please approve or veto) rather than just be the generic "bug updated" message.

(6.3) Reminder Emails

Third, we should use the portal to send reminder emails - bugzilla has a nag function, but it doesn't have business logic abilities and we don't want to invent yet another way to do workflow, so let's just use the portal. This is bug 195173. For example (all of these happen at midnight so "every 1 day" means "once a day at midnight" and "every 3 days" means "every third day at midnight" and "once 1 day" means "the first midnight this is true"):

• every 1 day: [new] AND no source code = reminder submitter to attach source code; if there are attachments, explain why those attachments are not considered source code
• every 2 days: [awaiting_pmc] = remind PMC to approve/veto the IP bug
• every 3 days: [awaiting_committer] = remind committer that the IP bug is waiting for their input
• every 1,3,3,3,5* days: [awaiting_project] = remind dev-list that the IP bug is waiting for their input. The initial 1 day is because IPzilla sends an email to its list of people which will include the committer but not the dev list, so the initial 1 day sends to the dev list.
• once 1 day: [approved_all_projects] = notify all projects that use older versions of this package that a newer version is approved (see 10)

(7) Change CQ Submission System

One of the larger changes is to eliminate the web form CQ entry and replace it with an Eclipse plug-in client. The plug-in will gather all the data about the submission into a potentially set of CQs and then create fully and correctly formed CQs, cross linked and all. Committers will be required to install and use this plug-in to submit CQs. See Eclipse Plug-in CQ Submission below.

(8) Change the IPzilla User Interface

(8.1) Portal-Similar User Interface

We should change the IPzilla user interface to match the portal UI design as much as possible. For example:

• When the state is [awaiting_pmc_vote] and we expect the PMC member to vote +1 or -1, the IPzilla user interface should show +1 and -1 radio buttons and a "vote" button for PMC members, just like a portal component would. For everyone else, it would show a "submit comment" button.

Note that we could put this behavior in the portal, but that's not really a clean user interface solution: then users would never be sure which system to go to, etc. I think it's a better user experience to put this behavior in an improved IPzilla skin, perhaps calling web-apis of the portal to determine roles, perhaps making that determination on its own. I admit that this is a little more coding than the perfect ideal, but I think a better user experience outweighs a little extra code here.

(8.2) Show Incubation Status

Show the incubation status of a project next to the project name bug 194768

(9) Update the IPzilla Portal View

Verify that the poral component that shows pending and completed IPzilla requests works with the new IPzilla fields and states and schema.

(10) Notify Projects of New Versions

When a new version of an existing jar is [approved_all_projects], send email to the dev lists of the other projects using the older versions informing everyone that the newer version is approved if they want to use it. Include instructions on how they can use the newer version (i.e., submit a CQ for the newer version by doing A, B, and C).

IPZilla State Diagram

• initial incoming CQs -> [new]
• [new] - see above - effectively this is "awaiting_source_code"
  1. if no source code -> [awaiting_committer] with comment "attach source code"
     • Source code is defined as:
       • a zip file
       • that does not contain nested jars
       • and does contain .java files
       • if it contains .class files then there is a .java file for each .class file
  2. if there is an attachment, but no PMC approval, automatically -> [awaiting_pmc_vote]
  3. if both, automatically run IP scan tool and attach results as a hidden attachment
  4. if approval and source code, automatically -> [awaiting_triage]
• [awaiting_triage]
  1. when legal starts working on this one -> [triage]
• [triage]
  1. add keywords. most common: apache, reuse, checkin_to_cvs, using_parallel_ip
  2. open and examine attachments
  3. if issues -> [awaiting_committer]
  4. if issues -> [awaiting_project]
  5. if reuse -> [approved_all_projects]
  6. otherwise -> [awaiting_analysis]
• [awaiting_analysis]
  1. when legal starts working on this one -> [under_review]
• [under_review]
  1. scanning code, etc.
  2. if issues -> [awaiting_committer]
  3. if issues -> [awaiting_project]
  4. if uncommon issues -> [awaiting_emo]
  5. if uncommmon issues -> [awaiting_board_approval]
  6. and then either -> [rejected]
  7. or -> [approved_all_projects]
• [approved_all_projects] - a terminal state
• add keyword: checkin_to_cvs
  • notify users of older versions that a new version is available
• [rejected] - a terminal state
• [withdrawn] - a terminal state
• [awaiting_board_approval]
  1. after answer from board -> [approved_all_projects]
  2. after answer from board -> [rejected]
  3. only allow emo to answer; everyone else just adds comments
• [awaiting_committer]
  1. after answer from committer -> returns to prev state
  2. only allow committers on the project to answer; all other committers just add comments
• [awaiting_project]
  1. after answer from committer -> returns to prev state
  2. only allow committers on the project to answer; all other committers just add comments
• [awaiting_pmc] - when need to approve
  1. after answer from pmc -> returns to prev state
• [awaiting_pmc_vote] - when need to approve
  1. after answer from pmc -> returns to prev state
  2. only allow pmc members on the top-level project to answer; all other committers just add comments
• [awaiting_emo] - policy related, external council, etc
  1. after answer from emo -> returns to prev state
  2. only allow emo to answer; everyone else just adds comments
• in all non-terminal states
  • if the submitter decides this is no longer needed -> [withdrawn]
  • only allow committers on the project, pmc members on the top-level project, and emo to withdraw an ip bug
  • any logged in user (committer) can always add an additional comment but doing so will not advance through the states
• in all terminal states
  • submitter can choose to reopen the issue -> [awaiting_triage]
• [approved_one_project] - not used anymore
• [closed] - not used
• [reuse] - not used

Field Ownership

All fields are owned by Legal (with an initial population by the CQ form) except:

• Add CC
• Remove CC
• Add Attachment
• Obsolete an Attachment
• Additional Comments
• Add a Depends On (but not "Remove a Depends On")

Once an IP bug has reached a terminal state, all fields are owned by Legal.

Eclipse Plug-in CQ Submission

One of the larger changes is to eliminate the web form CQ entry and replace it with an Eclipse plug-in client. The plug-in will gather all the data about the submission into a potentially set of CQs and then create fully and correctly formed CQs, cross linked and all. Committers will be required to install and use this plug-in to submit CQs. bug 195175

This tool will:

• find nested jars and create a separate CQ for each one
• prompt the user for the source code location, the project url, the license, etc.
• scan for declined licenses from the start
• query IPzilla for similar jars (name and version) so as to submit CQs with:
  • "reuse" of previous approved; ref the previous approval
  • "same jar; newer version; already under consideration"; ref the previously opened
  • "same jar; newer version"; ref the older versions
  • "same jar; older version"; ref the newer version and ask the user if they'd rather use the newer one
  • and ask the user if they can use a previously approved version
  • it would be best to be able to query for these cases by name and version before the user takes the time to download the binary and the source code and gather all those details

Related bugs: bug 171470, bug 166377, bug 166864, bug 171460, bug 174321, bug 166269

• An Eclipse plug-in. Source code in Project Dash. Distributed through the Project Dash update site.
• Will start with a File > New > Other wizard categories "Eclipse Foundation" and named "IP Contribution Questionnaire"
• The wizard will step through a number of questions and will create a properties text file in the current project with an ".efipcq" extension.
  • Or multiple .efipcq files if multiple CQs are needed.
• The plug-in will have a two-page editor for .efipcq files similar in style to the PDE's site.xml editor.
• The editor will have a button for "Submit This CQ" which will make the connection to IPzilla, enter all the data, upload the source code, etc.
  • This button will submit multiple CQs if they are all linked due to being created by the same wizard.
• The editor will remember (through a extra property) that the CQ has been submitted and not allow duplicates and will have a link in the editor to go to the CQ for further edits.
• Or perhaps the wizard will just create the basic file and then the editor will do all the work - that's probably a better idea.
• The editor will check the incubation conformance state of the project and if the project conforms, it will provide a "request Parallel IP" option button. Pressing that button will annotate the CQ will the correct keyword. If the project does not conform, the button will be replaced by a "explain why my project does not have this option" link.

Editor Fields

• Committer:
• Project: (from which we determine the appropriate PMC by querying the Foundation database via a web-api)
• Component:
• Contribution:
  • third-party jar or
  • source code zip
  • if nested jars or dependencies are found, create additional .efipcq files for those jars. Note that a nested jar file containing the code from the source code is not a nested jar file for our purposes - this can happen if someone compiles the source code and stores the jar in the source code directory
  • if source code is not full source code, notify the user (i.e., if there are class files that do not have source files)
• Contribution Name:
  • with assistance for the common cases: initial contribution; large source code contribution; reuse of third-party jar
  • if Contribution is a jar, then try to auto-extract the name
• Version:
  • with assistance for common cases: 0.0 and jar version
  • if Contribution is a jar or zip, then try to auto-extract the version
  • look up the name and version if IPzilla and tell the user if this
    • is already approved
    • is pending
    • other versions are pending
    • other versions are approved
• Description:
  • with assistance for what sort of text we are expecting to see here
• Source Code:
  • if it's a third-party jar, then also ask them for the source code zip
• Licenses:
  • first we can try to auto-extract the license(s) from the source code
  • then we can show the list of common licenses
  • multi-choice list
  • should we include GPL and LGPL on this list so that we can show an error "not allowed"?
• URL to third-party project website
• How required is this contribution? (stop-ship vs. ease-of-use)
  • If stop-ship, what is the drop-dead date?
• other questions that we need to ask (Janet, what are these?)
• Supporting Information: (not sure what goes here)
• Cryptography: yes/no, if yes, description with details (with assistance explaining what kind of details are needed)

Attachment/Jar Scanner

Here are the things we will try to determine from scanning the zip/jar attachment:

• The name of the contribution - use regular expressions on the attachment file name
• The version of the contribution - use regular expressions on the attachment file name
• The licenses that apply - search all files and nested files for certain string patterns
• If the zip/jar has both EPL and non-EPL code either:
  • Tell the user they need to separate the code into two separate zips/jars, or
  • (ideally and if possible) automatically separate the code and create two new zips/jars and two CQs
• If the zip/jar has binaries without source code, that's an error. Binaries with source code (.class files with corresponding .java files) are ok.
• If the zip/jar has nested zips/jars then:
  • If the nested zips/jars are just repackagings of the enclosed source, that's fine. In other words, if the zip contains X.java, Y.java, and Z.java plus two jars, one contains X.class and Y.class, the other containing Y.class and Z.class, that's just a repackaging. However, if the second jar contained Y.class and B.class, and there is no B.java, then that is not a repackaging.
  • If the nested zips/jars contain additional source code, they must be filed as separate CQs. The plug-in will extract the nested zips/jars, creating new temporary zips/jars and associate those with new (linked) CQs. The original zip/jar will be recreated without the nested zips/jars for the original CQ.
  • If the nested zips/jars contain additional binaries, the same applies, but of course the user will need to provide the source code for them.
• If the code is EPLed, scan for the required header on all source files.

We will determine regular expressions and patterns to search for by examining all the current IPzilla bugs. For example, we will determine regular expressions for name and version extraction by considering the names and versions of all existing IPzilla bugs and the name of their attached files. Etc.

Other Considerations For Attachments

• Some attachments are C/C++ code, not Java. Could be other file types as well (XSD, Ruby, ...?).
• We need to find a way to encourage the submitter to submit only the code they will be using and not extra files. For example, if the submitter downloads distro X from Apache, but only wants to use a subset of X, they need to submit only the subset of X. Not sure how to help with this.

Automating Parallel IP

(Not currently being addressed, but a good idea that could probably be added to the project...)

If we were able to get a box that said yes/no to binary code distribution, together clear information on the license(s) that apply (often there are more than one, though these are sometimes buried – perhaps multiple drop down menus so that we get clarity and have an ability to match against a list Janet prepares), and some certainty that they have actually identified all applicable licenses, then Janet thinks we could automate a lot of parallel IP approvals. The big challenge will be proving that they have identified all the relevant licenses - not quite sure how to do that.

Proving that they have identified all the relevant licenses would be preferable but not strictly speaking required to implement this. The parallel IP process currently contemplates us relying the Committer's assertion of the licenses that apply.

Bugs That We Are Not Addressing

• bug 178664 because (7) will replace the form.
• bug 194654 is a good idea and needed, but not part of this work
• bug 166282 also a good idea, but not part of this work
• bugs 196246, 196247, and 196248 need a policy decision and are also out of scope for this work