Difference between revisions of "WTP CVS Restructuring"

From Eclipsepedia

Jump to: navigation, search
(adding details)
 
(32 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
==Purpose==
 
==Purpose==
  
In an effort to [https://bugs.eclipse.org/bugs/show_bug.cgi?id=198541 standardized groups on dev.eclipse.org] especially in light of our recent WTP Refactoring, the page is to propose and document how to restructure our CVS directories, so their access can be managed by associating directories with Unix Groups.  
+
In an effort to [https://bugs.eclipse.org/bugs/show_bug.cgi?id=198541 standardized groups on dev.eclipse.org] especially in light of our recent [http://www.eclipse.org/webtools/devProcess/wtpProjects/WTP%20Projects.pdf WTP Projects Refactoring], the page is to propose and document how to restructure our CVS directories, so their access can be managed by associating directories with Unix Groups. In principle, we could leave all the directories the way they are now, and just carefully assign the Unix groups on a directory by directory bases ... but, this would be very complicated, error prone, and hard to scale in the future.  
  
The document will start by focusing on the old wst and jst directories, as they need the most work, and other subprojects addressed later, towards the end of this proposal.  
+
The document will start by focusing on the old wst and jst directories, as they need the most work, and other subprojects addressed later, towards the end of this proposal.
  
==WST and JST==
+
I have opened [https://bugs.eclipse.org/bugs/show_bug.cgi?id=207212 bug 207212] to track implementation and resolution of this issue. Please comment there if there are questions or suggestions for this document.
  
These initial WTP projects were created with the following sort of hierarchy -- abbreviated and idealized here to simplify explaining the principles of the restructuring.  
+
==The Groups==
 +
 
 +
First, I'll describe my best interpretation of what our [https://bugs.eclipse.org/bugs/show_bug.cgi?id=198541 standardized groups] will be.
 +
 
 +
===Code===
 +
The following are the Unix groups that will be associated with "code", in the cvs repository.
 +
 
 +
webtools.common-dev
 +
webtools.datatools-dev
 +
webtools.ejbtools-dev
 +
webtools.jeetools-dev
 +
webtools.servertools-dev
 +
webtools.sourceediting-dev
 +
webtools.webservices-dev
 +
webtools.maps-dev
 +
webtools.releng-dev
 +
webtools.incubator-dev
 +
webtools.jsf-dev *
 +
webtools.dali-dev *
 +
webtools.atf-dev *
 +
* these are very similar in name to what they already are.
 +
 
 +
===Website(s)===
 +
Per our WTP policy, all committers to any WTP project get full access to all of the WTP websites, so, we just need one group for that:
 +
 
 +
webtools.all    (currently, our "webtools" group is similar,
 +
                  but think webtools-home is used for this)
 +
 
 +
For reference, the current (cvs) website locations are
 +
 
 +
  org.eclipse/www/webtools (main one)
 +
  org.eclipse/www/wtp      (incubator)
 +
  org.eclipse/www/jsf      (used? do we still need this? need separate?)
 +
  org.eclipse/www/dali    Used for redirect purposes; would like to keep the re-direct in place, but we should remove all of the content here
 +
  org.eclipse/www/atf
 +
 
 +
Do we need new, separate ones for each of the new subprojects? (No, I don't think so).
 +
 
 +
===Download site write access===
 +
 
 +
This is currently limited to the "build team".  The group is currently "webtoolsadmin" and I think the proposal for consistent naming is to use "webtools.downloads".
 +
 
 +
===Map File Directories===
 +
 
 +
Notice I'm proposing all the map file directories be moved to under one higher level "webtools.maps" directory.
 +
The purpose of that is, I'd like the map files to be writable by all, hence associated with webtools-all. Some of the divisions of the map files themselves can come after the main
 +
refactoring.
 +
 
 +
The reason is that occasionally someone from another project may have to releaes, or revert a particular map file
 +
from another project.
 +
 
 +
And, I checked, and there's no particular legal reason to "regulate" the map files ... they are similar to the web pages, there's no particular content to protect with IP rules, and I think there's little danger of someone accidently touching someone elses, unless there'd been prior agreement.
 +
 
 +
==WST and JST to Refactored Projects==
 +
 
 +
These initial WTP projects were created with the following sort of hierarchy -- abbreviated and idealized here to simplify explaining the principles of the restructuring. See the [[WTP_CVS_Restructuring#Detailed_Listings | Detailed Listings]] section below the literal contents of the proposed new structure.  
 
    
 
    
 
   wst
 
   wst
Line 35: Line 90:
 
The new proposed structure would look as follows
 
The new proposed structure would look as follows
  
 +
webtools.maps
 +
    releng.common
 +
    releng.sourceediting
 +
    releng.servertools
 +
    releng.webservices
 +
    releng.datatools
 +
    releng.jeetools
 +
    releng.ejbtools
 +
    releng.dali
 +
    releng.jsf
 
  webtools.releng
 
  webtools.releng
 
     features
 
     features
 
     plugins
 
     plugins
webtools.sourceediting
+
    org.eclipse.wtp.releng
 +
    org.eclipse.wtp.releng.tests
 +
    org.eclipse.wtp.releng.webupdatesite
 +
    releng.builder
 +
    releng.control
 +
    releng.wtpbuilder
 +
    releng.wtptools
 +
    working
 +
sourceediting
 
     features
 
     features
 
     plugins
 
     plugins
Line 49: Line 122:
 
     development
 
     development
 
     thirdparty
 
     thirdparty
  webtools.servertools
+
  servertools
 
     features
 
     features
 
     plugins
 
     plugins
Line 58: Line 131:
 
The above examples show a few points related to this restructuring.  
 
The above examples show a few points related to this restructuring.  
  
# The best top-level name should "match" what the access group name will be: toplevel.subproject.component-subcomponent. We do not currently have components controlled by access restrictions, so I think we in webtools need only define the toplevel.subproject directories for now (this can change in the future, but best to start off with simplest, and add detail later, when needed.  
+
# The best top-level directory name should be similar to what the access group name will be: toplevel.subproject.component-subcomponent. We do not currently have components controlled by access restrictions, so I think we in webtools need only define the toplevel.subproject groups for now, hence the top level directories can be simply the subproject name, since 'webtools' is in its own cvs repository, so 'webtools' does not need to be part of the directory name. Also note that component and subcomponents can be added in the future, but best to start off with simplest, and add detail later, when needed.  
# Thus, everyone in a certain subproject can be associated with the "highest" webtools directory, and automatically get access to everything lower, without the need for more than subprojects groups, for now.  
+
# Thus, everyone in a particular Eclipse subproject can be associated with a top level  webtools directory, and automatically get access to everything in that directory, without the need for more complicated administration.  
# Note the previous "assembly" features and plugins will be become part of releng subproject. These are the "product" plugins, and the features that define our builds. Each project will be responsible for their own features delivered to WTP.  
+
# Note the previous "assembly" features and plugins will be become part of releng subproject. These are the Eclipse product plugins, and the features that define our builds. Each project will be responsible for their own features delivered to WTP (currently called the build-component features).  
# Notes the example in source editing where what used to be directories, xml and xsd, are no longer distinguished by cvs directories, but all the plugins flattened under webtools.sourceediting/plugins.  
+
# Note the example in source editing where what used to be directories, xml and xsd, are no longer distinguished by cvs directories, but all the plugins flattened under sourceediting/plugins.  
 
## Conceptually and architecturally, xml and xsd could still be called 'components' but they do not have separate access control, so it's a bit overkill (and ultimately restrictive) to define them in separate cvs directories.
 
## Conceptually and architecturally, xml and xsd could still be called 'components' but they do not have separate access control, so it's a bit overkill (and ultimately restrictive) to define them in separate cvs directories.
## Note that the "access control" for components such as xml and xsd is (still) social in nature. Legally, in theory, anyone in the source editing project could change the code there, but in practice, there are individuals or small teams that normally do. Others, even in the same project, usually would attach a patch to a bugzilla if they wanted one of those experts to review and commit the code.  
+
## Note that the "access control" for components such as xml and xsd is (still) social in nature. Legally, and in principle, anyone in the source editing project could change the code there, but in practice, there are individuals or small teams that normally do. Others, even in the same project, usually would attach a patch to a bugzilla if they wanted one of those experts to review and commit the code.
       
+
# The map files will not have access restrictions per se (all committers will have access to 'webtools.maps' but the maps will be refactored in directories that contain map files only for that subproject.
 +
 
 
==Detailed Listings==
 
==Detailed Listings==
  
The following are the detailed listing of directories in what would be their new structure.  
+
The following are the detailed listing of directories in what would be their new structure. Hopefully the directory names, and groups names are obvious.  
  
[[webtools.common]]
+
:[http://www.eclipse.org/webtools/development/treeout/common.html  common]
[[webtools.servertools]]
+
:[http://www.eclipse.org/webtools/development/treeout/servertools.html servertools]
[[webtools.datatools]]
+
:[http://www.eclipse.org/webtools/development/treeout/datatools.html datatools]
[[webtools.sourceediting]]
+
:[http://www.eclipse.org/webtools/development/treeout/sourceediting.html sourceediting]
[[webtools.webservices]]
+
:[http://www.eclipse.org/webtools/development/treeout/webservices.html webservices]
[[webtools.jeetools]]
+
:[http://www.eclipse.org/webtools/development/treeout/jeetools.html jeetools]
[[webtools.ejbtools]]
+
:[http://www.eclipse.org/webtools/development/treeout/ejbtools.html ejbtools]
[[webtools.releng]]
+
:[http://www.eclipse.org/webtools/development/treeout/org.eclipse.jsf.html org.eclipse.jsf]
 +
:[http://www.eclipse.org/webtools/development/treeout/org.eclipse.jpa.html org.eclipse.jpa]
 +
:[http://www.eclipse.org/webtools/development/treeout/org.eclipse.atf.html org.eclipse.atf]
 +
:[http://www.eclipse.org/webtools/development/treeout/webtools.releng.html webtools.releng]
 +
:[http://www.eclipse.org/webtools/development/treeout/webtools.maps.html webtools.maps]
  
 +
==To Do Items and Schedule==
  
==To Do Items==
+
* 10/25 Proof on Concept
  
# change maps to use module names only, to divide up the change, so once restructuring is done, only the modules file has to be changed, and builds will keep working. Note: there are probably some, JSF or JPA files that will have to be added to modules file. (Remember, any old builds, if they had to be re-created, would have to be changed, but that's unlikely to be needed).
+
::*try using CVS snapshots, on local, test machines with some move scripts to test to be sure we've accounted for everything.
# try using CVS snapshots, on local, test machines with some move scripts to test to be sure we've accounted for everything.  
+
:::Note: I've included some scripts in one of our cvs projects that work against a snapshot to do the reorganization. I'll see if those can be evolved to be the real things, to use when the real thing is done. See
 +
::::org.eclipse.wtp/org.eclipse.wtp.releng.utilities<br />
  
 +
::*Have Web Master sanity check the scripts and whole approach.
  
 +
 +
* 11/ 1  By the 11/1 Status meeting,
 +
 +
::* Project leads to have reviewed the "detailed listings" for accuracy. Post requested revisions to wtp-releng, so revised listings can be created, preferably before the meeting.
 +
 +
::*Releng (dw) to make sure all modules in all map files (for WTP 2.0 and 3.0) are listed in the modules file of CVS, so all (important) plugins show up in the flat listing.
 +
 +
::*JSF, ATF, Dali should make a firm decision if they want to participate in re-org, to stay consistent, or, leave their directory structure as-is.
 +
 +
* 11/ 8 By the 11/8 Status meeting,
 +
 +
::* spell out committer ID's for each Unix group
 +
 +
::* make SED scripts to fix up map files, module files ... maybe project team sets?
 +
 +
* 11/ 13 By end-of-day
 +
 +
::* Committers create patch files of any "unfinished" work that's not checked in, just for their personal backup.
 +
 +
::* Committers check "unfinished" work into a temporary branch, if it's not ready for HEAD, so it will be moved during the mass move.
 +
 +
::* Committers should create "Team Project Sets" of their favorite "in workspace" projects (assuming a SED script, or other editing can convert them).
 +
 +
* 11/ 16 1:00 PM - 5 PM Move day
 +
 +
::* Begin the mass move
 +
 +
::* Update modules file
 +
 +
::* Update map files
 +
 +
::* Do Test Build
 +
 +
::* Do some quick Workspace tests
 +
 +
::* Open the flood gates
 +
 +
==JSF, JPA, and ATF ==
 +
 +
The directory-to-Unix-group associations will be pretty easy for these three subprojects and they can be left the way they are. Or, if any of those projects want to flatten their directory structure, this might be the time to do it. Each of those projects can provide their preference and will remain open for discussion, for now.
  
 
==Reference and Notes==
 
==Reference and Notes==
Line 93: Line 214:
 
Snapshots of our CVS can be obtained at <br />
 
Snapshots of our CVS can be obtained at <br />
 
http://archive.eclipse.org/arch/
 
http://archive.eclipse.org/arch/
 +
 +
There is a SED Script to "edit" old path/module names to new path/module names that can
 +
be found in one of our utility releng projects in cvs: org.eclipse.wtp.releng.utilities.
 +
It is named '''sedscript.sed'''
 +
 +
 +
[[Category:Old, Obsolete and Interesting Things| ]]

Latest revision as of 01:11, 13 September 2008

Contents

[edit] Purpose

In an effort to standardized groups on dev.eclipse.org especially in light of our recent WTP Projects Refactoring, the page is to propose and document how to restructure our CVS directories, so their access can be managed by associating directories with Unix Groups. In principle, we could leave all the directories the way they are now, and just carefully assign the Unix groups on a directory by directory bases ... but, this would be very complicated, error prone, and hard to scale in the future.

The document will start by focusing on the old wst and jst directories, as they need the most work, and other subprojects addressed later, towards the end of this proposal.

I have opened bug 207212 to track implementation and resolution of this issue. Please comment there if there are questions or suggestions for this document.

[edit] The Groups

First, I'll describe my best interpretation of what our standardized groups will be.

[edit] Code

The following are the Unix groups that will be associated with "code", in the cvs repository.

webtools.common-dev
webtools.datatools-dev
webtools.ejbtools-dev
webtools.jeetools-dev
webtools.servertools-dev
webtools.sourceediting-dev
webtools.webservices-dev
webtools.maps-dev
webtools.releng-dev
webtools.incubator-dev
webtools.jsf-dev *
webtools.dali-dev *
webtools.atf-dev *
* these are very similar in name to what they already are.

[edit] Website(s)

Per our WTP policy, all committers to any WTP project get full access to all of the WTP websites, so, we just need one group for that:

webtools.all    (currently, our "webtools" group is similar, 
                 but think webtools-home is used for this)

For reference, the current (cvs) website locations are

 org.eclipse/www/webtools (main one)
 org.eclipse/www/wtp      (incubator)
 org.eclipse/www/jsf      (used? do we still need this? need separate?)
 org.eclipse/www/dali     Used for redirect purposes; would like to keep the re-direct in place, but we should remove all of the content here
 org.eclipse/www/atf

Do we need new, separate ones for each of the new subprojects? (No, I don't think so).

[edit] Download site write access

This is currently limited to the "build team". The group is currently "webtoolsadmin" and I think the proposal for consistent naming is to use "webtools.downloads".

[edit] Map File Directories

Notice I'm proposing all the map file directories be moved to under one higher level "webtools.maps" directory. The purpose of that is, I'd like the map files to be writable by all, hence associated with webtools-all. Some of the divisions of the map files themselves can come after the main refactoring.

The reason is that occasionally someone from another project may have to releaes, or revert a particular map file from another project.

And, I checked, and there's no particular legal reason to "regulate" the map files ... they are similar to the web pages, there's no particular content to protect with IP rules, and I think there's little danger of someone accidently touching someone elses, unless there'd been prior agreement.

[edit] WST and JST to Refactored Projects

These initial WTP projects were created with the following sort of hierarchy -- abbreviated and idealized here to simplify explaining the principles of the restructuring. See the Detailed Listings section below the literal contents of the proposed new structure.

 wst
   components
      assembly
         features
         plugins
      server
         features
         plugins
         tests
      xml
         features
         plugins
            org.eclipse.wst.xml.core
            org.eclipse.wst.xml.ui
         tests
         examples
         development
         thirdparty
      xsd
         plugins
            org.eclipse.wst.xsd.core
            org.eclipse.wst.xsd.ui
         tests

The new proposed structure would look as follows

webtools.maps
   releng.common
   releng.sourceediting
   releng.servertools
   releng.webservices
   releng.datatools
   releng.jeetools
   releng.ejbtools
   releng.dali
   releng.jsf
webtools.releng
   features
   plugins
   org.eclipse.wtp.releng
   org.eclipse.wtp.releng.tests
   org.eclipse.wtp.releng.webupdatesite
   releng.builder
   releng.control
   releng.wtpbuilder
   releng.wtptools
   working
sourceediting
   features
   plugins
      org.eclipse.wst.xml.core
      org.eclipse.wst.xml.ui
      org.eclipse.wst.xsd.core
      org.eclipse.wst.xsd.ui
   tests
   examples
   development
   thirdparty
servertools
   features
   plugins
   tests

[edit] Principles

The above examples show a few points related to this restructuring.

  1. The best top-level directory name should be similar to what the access group name will be: toplevel.subproject.component-subcomponent. We do not currently have components controlled by access restrictions, so I think we in webtools need only define the toplevel.subproject groups for now, hence the top level directories can be simply the subproject name, since 'webtools' is in its own cvs repository, so 'webtools' does not need to be part of the directory name. Also note that component and subcomponents can be added in the future, but best to start off with simplest, and add detail later, when needed.
  2. Thus, everyone in a particular Eclipse subproject can be associated with a top level webtools directory, and automatically get access to everything in that directory, without the need for more complicated administration.
  3. Note the previous "assembly" features and plugins will be become part of releng subproject. These are the Eclipse product plugins, and the features that define our builds. Each project will be responsible for their own features delivered to WTP (currently called the build-component features).
  4. Note the example in source editing where what used to be directories, xml and xsd, are no longer distinguished by cvs directories, but all the plugins flattened under sourceediting/plugins.
    1. Conceptually and architecturally, xml and xsd could still be called 'components' but they do not have separate access control, so it's a bit overkill (and ultimately restrictive) to define them in separate cvs directories.
    2. Note that the "access control" for components such as xml and xsd is (still) social in nature. Legally, and in principle, anyone in the source editing project could change the code there, but in practice, there are individuals or small teams that normally do. Others, even in the same project, usually would attach a patch to a bugzilla if they wanted one of those experts to review and commit the code.
  5. The map files will not have access restrictions per se (all committers will have access to 'webtools.maps' but the maps will be refactored in directories that contain map files only for that subproject.

[edit] Detailed Listings

The following are the detailed listing of directories in what would be their new structure. Hopefully the directory names, and groups names are obvious.

common
servertools
datatools
sourceediting
webservices
jeetools
ejbtools
org.eclipse.jsf
org.eclipse.jpa
org.eclipse.atf
webtools.releng
webtools.maps

[edit] To Do Items and Schedule

  • 10/25 Proof on Concept
  • try using CVS snapshots, on local, test machines with some move scripts to test to be sure we've accounted for everything.
Note: I've included some scripts in one of our cvs projects that work against a snapshot to do the reorganization. I'll see if those can be evolved to be the real things, to use when the real thing is done. See
org.eclipse.wtp/org.eclipse.wtp.releng.utilities
  • Have Web Master sanity check the scripts and whole approach.


  • 11/ 1 By the 11/1 Status meeting,
  • Project leads to have reviewed the "detailed listings" for accuracy. Post requested revisions to wtp-releng, so revised listings can be created, preferably before the meeting.
  • Releng (dw) to make sure all modules in all map files (for WTP 2.0 and 3.0) are listed in the modules file of CVS, so all (important) plugins show up in the flat listing.
  • JSF, ATF, Dali should make a firm decision if they want to participate in re-org, to stay consistent, or, leave their directory structure as-is.
  • 11/ 8 By the 11/8 Status meeting,
  • spell out committer ID's for each Unix group
  • make SED scripts to fix up map files, module files ... maybe project team sets?
  • 11/ 13 By end-of-day
  • Committers create patch files of any "unfinished" work that's not checked in, just for their personal backup.
  • Committers check "unfinished" work into a temporary branch, if it's not ready for HEAD, so it will be moved during the mass move.
  • Committers should create "Team Project Sets" of their favorite "in workspace" projects (assuming a SED script, or other editing can convert them).
  • 11/ 16 1:00 PM - 5 PM Move day
  • Begin the mass move
  • Update modules file
  • Update map files
  • Do Test Build
  • Do some quick Workspace tests
  • Open the flood gates

[edit] JSF, JPA, and ATF

The directory-to-Unix-group associations will be pretty easy for these three subprojects and they can be left the way they are. Or, if any of those projects want to flatten their directory structure, this might be the time to do it. Each of those projects can provide their preference and will remain open for discussion, for now.

[edit] Reference and Notes

Committers can see our current CVS directories
/home/data/cvs/webtools

Snapshots of our CVS can be obtained at
http://archive.eclipse.org/arch/

There is a SED Script to "edit" old path/module names to new path/module names that can be found in one of our utility releng projects in cvs: org.eclipse.wtp.releng.utilities. It is named sedscript.sed