Difference between revisions of "IT Infrastructure Doc"
(→See download statistics?)
(→See which mirror sites are mirroring my files?)
|Line 160:||Line 160:|
===See which mirror sites are mirroring my files?===
===See which mirror sites are mirroring my files?===
You can use the Find a Mirror script to see which mirror sites have your files. The Find a Mirror script is
You can use the Find a Mirror script to see which mirror sites have your files. The Find a Mirror script is: http://www.eclipse.org/downloads/download.php
|Line 172:||Line 172:|
All mirrors of the Lepido project, in XML format:
Get a file from a random mirror, without prompting
'''PLEASE NOTE:''' We have a list of excluded file patterns -- files that are *not* sent to our mirrors.
'''PLEASE NOTE:''' We have a list of excluded file patterns -- files that are *not* sent to our mirrors.
Revision as of 13:10, 11 June 2010
- 1 Website
- 1.1 How do I setup my project website?
- 1.2 How do I author web pages using the Phoenix method?
- 1.3 Access the Bugzilla database using PHP?
- 1.4 Use a database for my website?
- 1.5 I need to put a large file on my website. How should I do this?
- 1.6 Use PHP on my website?
- 1.7 Optimize my PHP code for large-scale use?
- 2 CVS
- 3 Bugzilla
- 4 Downloads
- 4.1 Put files on the download server?
- 4.2 Move files to archive.eclipse.org?
- 4.3 See which mirror sites are mirroring my files?
- 4.4 Use the Find a Mirror script?
- 4.5 Create an Update Site?
- 4.6 See download statistics?
- 4.7 View my disk space quota?
- 4.8 Increase my disk space quota?
- 4.9 Sign my plugins/ZIP files?
- 5 Builds
- 6 Mailing Lists
- 7 Eclipse Wiki
- 8 Eclipse Servers
How do I setup my project website?
Project websites are hosted in a CVS repository separate from the actual project code. The repository path is dev.eclipse.org:/cvsroot/org.eclipse, in the www component.
Once the webmaster
adds a space for your project, files you commit to the website CVS are automatically checked out to www.eclipse.org/xyz, where
xyz is your project's short name.
You are free to use HTML and PHP on your website.
sting a project website is normally done when the project proposal has been approved. If you suspect your files are not being checked out to the www.eclipse.org website, simply commit a small change to one file. This is usually enough to trigger a website refresh.
Access the Bugzilla database using PHP?
Please see the section labeled "[tools]" in the Portal
Use a database for my website?
We currently do not offer projects with database support.
I need to put a large file on my website. How should I do this?
Large (1 MB+) ZIP and JAR files must be put in the downloads area, using the Find A Mirror script to link to them. However, small files (less than 1 MB) can be put on the www.eclipse.org/yourproject website directly without causing too much harm.
The Find A Mirror script supports transparent mirror use, so large screencasts and PDFs can be put in the downloads area as well without imposing the added step of selecting a mirror site for the file. Simply add &r=1 to the URL. For instance, http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.1-200506271435/eclipse-SDK-3.1-linux-gtk.tar.gz&r=1 will fetch you the Eclipse SDK 3.1 for Linux from a random mirror site without asking you which one.
Remember to allow our mirrors at least 24 hours to sync up before using a transparent mirror redirect.
Use PHP on my website?
PHP support is available on www.eclipse.org only. Simply commit files with the .php file extension to your website's CVS repository. Although some projects host PHP files on download.eclipse.org, we do not encourage or recommend it.
Eclipse.org is a high-traffic website. Please make sure your PHP code is optimized to run in this type of environment. See the next item.
Optimize my PHP code for large-scale use?
Eclipse.org is a high-traffic website. To improve PHP's functionality, we have set very liberal limits on how many resources PHP can consume. However. if if your project is very popular, bad PHP code can slow the entire site down.
Of course, we could harden PHP to protect our website, but that would cut some functionality. Some tips for you:
- Never call the web service to include/open files - include("http://www.eclipse.org/somefile.html") and fopen("http://localhost/somefile.xml") are very costly to run, because they call the web service, and can lead to eclipse.org Denial-Of-Servicing itself under heavy load.
- Never include/open remote files - include("http://www.someothersite.org/somefile.html") is forbidden, as someone could launch a Denial-Of-Service attack against a remote site. We don't allow you to establish remote connections from eclipse.org servers other than the build server.
- Sanitize your incoming parameters - include($parameter) is particularly dangerous if $parameter is not sanitized. Someone could freely surf the web anonymously, hiding behind eclipse.org servers, or they could use your page to access local files, or launch Denial-Of-Service attacks against remote servers.
- Cache aggregated, processor-intensive data - SQL aggregations, file system scans, Bugzilla lists can (and should) be cached to avoid redundant processor- and disk-intensive operations. For instance, scanning through download.eclipse.org directories to display the size of a build could be useful, but doesn't need to happen for each website visitor. Cache the results of this operation to a file, and update the file if the file is older than 12 hours.
There are many, many other security and PHP best-practices. These are just the basics.
Connect to Eclipse CVS?
Please see this page.
Connect to Eclipse CVS when PSERVER and/or EXTSSH are firewalled?
Please see the Proxy configuration on this page.
Delete files from CVS?
Although you can use SSH and a terminal to delete files in your CVS repository, we recommend you open a Bugzilla bug, in Community CVS, requesting the files that need to be deleted.
Manage UNIX groups for CVS access?
The unix groups are essentially webmaster tools used to manage commit rights to CVS repositories and to the downloads area. For each project (Eclipse-Foundation-sanctioned project, such as Eclipse Platform, DSDP-DD, Mylar, CDT, etc) we typically create three groups:
- project-dev: the group of accounts that can commit to the project's code repository
- project-home: the group of accounts that can commit to the project'ss website
- projectadmin: those who can store files in the downloads area.
For some projects, having all committers in one group with commit rights across the entire project is not adequate when some committers must be limited to a specific set of modules. In these cases, we create project-module groups that allow specific committers to only commit to that portion of CVS.
Create a new Component/Version/Milestone/Target?
Put files on the download server?
Downloadable files must be placed on the download.eclipse.org server so they can be mirrored to our mirror sites worldwide. Please refer to the File Paths below to see where your files should go.
|/home||This is where all the public data is. Needless to look anywhere else.|
|These are your CVS repositories. We strongly encourage you to contact the WebMaster to move or delete files, instead of doing it from the shell.|
|This is the website repository. Look in www/yourproject. Anything you commit here is published to www.eclipse.org/yourproject within one minute.|
|This is the main download area, accessible only via RSYNC and SSH/SFTP/SCP.|
|/home/data/httpd/download-staging.priv||This is where you can store downloadable files that aren't ready for downloads yet. Perhaps you need to verify them, or sign them. The Eclipse web servers cannot access this area, so it's private.|
|/home/data/httpd/writable||This is where your PHP scripts can write files. Temp files, status files, XML files, whatever. Ask us for a spot here if your project's PHP scripts need write access.|
|/home/data/users||These are your Home directories. Just type cd to get to yours.|
Please ensure only pertinent, current files are in the downloads area, as we cannot store an eternity of nightly, integration and stable builds. Production releases can be kept forever; however, we ask that you move archived releases to archive.eclipse.org (see below).
To transfer your files, use an SCP(committers,release engineers) or SFTP(build engineers only) client and connect to download1.eclipse.org using your committer account. Transfer files to your project's directory in the downloads area (Typically ~/downloads/toplevel/yourproject). Your project's downloads directory is typically communicated to the Project Lead upon project provisioning. Please ensure that the file permissions include world-readable (664; rw-rw-r--) and directory permissions allow for world-executable (775, rwxrwxr-x).
Large projects with frequent builds may find it more convenient to use RSYNC over SSH.
Once your files are on the download.eclipse.org server, they are immediately available to the general public. However, for release builds, we ask that you wait at least four hours for our mirror sites to fetch the new files before linking to them. It typically takes a day or two for all the mirror sites to synchronize with us and get new files.
To make your downloads available to the general public, please do not link directly to download.eclipse.org. Instead, use the Find a Mirror script (info below). Using this script allows you to view download statistics and allows users to pick a nearby mirror site for their download.
Please note that although we tolerate PHP, HTML and JPG/GIF files on download.eclipse.org, we encourage you to put such files on www.eclipse.org. Those files are not mirrored to public mirror servers.
To save disk space on our mirror servers, and to reclaim some quota space, we recommend you move old release builds to archive.eclipse.org.
Move files to archive.eclipse.org?
Because our mirror sites don't have as much disk space for Eclipse files as we do, we have created an http://archive.eclipse.org site for you to store older release builds.
The archive.eclipse.org structure is similar to that of download.eclipse.org. To move your files, we recommend using the SSH prompt as below. If you arenot comfortable with the SSH prompt, you can ask WebMaster to move the files for you.
ssh firstname.lastname@example.org mv ~/downloads/your/project/oldrelease/ /home/data/httpd/archive.eclipse.org/your/project/oldrelease/
See which mirror sites are mirroring my files?
You can use the Find a Mirror script to see which mirror sites have your files. The Find a Mirror script is: http://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip
- file (Required): specify the filename, relative to the downloads home, starting with a "/". This file must exist in the downloads area. Although you can specify a directory name, your mirror list will be more accurate if you specify a file.
- format (Optional): specify html (default) or xml. Useful for building the mirrors.xml for Update sites.
- protocol (Optional): ftp or http: list only ftp or http mirrors only (both are the default)
- r (Optional): specify 1 to automatically redirect to the best mirror (the one that would normally be at the top) without asking the user to choose.
- nf (Optional): specify 1 to get an actual 404 Not Found error if the file doesn't exist (instead of a lovely page saying so).
The script will examine the Last Modified timestamp of the given file and return only those mirrors that have synchronized with Eclipse.org after that time.
All mirrors of the Lepido project, in XML format: http://www.eclipse.org/downloads/download.php?file=/technology/lepido/M1/content.jar&format=xml
Get a file from a random mirror, without prompting http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.1-200506271435/eclipse-SDK-3.1-win32.zip&r=1
PLEASE NOTE: We have a list of excluded file patterns -- files that are *not* sent to our mirrors. Nightly and Integration builds are typically very large and don't get many downloads, therefore it's typically more costly (in terms of bandwidth) to mirror them than to support the few client downloads they generate. At time of writing, our exclusion list is:
Use the Find a Mirror script?
See the section above.
Create an Update Site?
Project update sites are currently not integrated to the Platform update site, and therefore you must provide Update instructions on your website.
# In your downloads directory, create a directory called update-site # in update-site/, put your features in a features/ subdirectory, your plugins in a plugins/ subdirectory, and your site.xml file in the update-site/ directory. # Use the mirrorsURL tag with the Find A Mirror script to prompt the users for a mirror site. See a sample site.xml file below. # Provide instructions on your download pages that tell the user to add a remote Update site, and indicate the url to your update-site/ directory
Sample site.xml file (from Lepido project):
<?xml version="1.0" encoding="UTF-8"?> <site mirrorsURL="http://www.eclipse.org/downloads/download.php?file=/technology/lepido/update-site/site.xml&format=xml"> <description url="http://download.eclipse.org/technology/lepido/update-site/"> Lepido, Tools for Apache Cocoon </description> <feature url="features/org.eclipse.lepido_0.1.0.jar" id="org.eclipse.lepido" version="0.1.0"> <category name="lepido"/> </feature> <category-def name="lepido" label="Lepido, Tools for Apache Cocoon"/> </site>
See download statistics?
The Find a Mirror script tracks download requests once the user has picked a mirror site (or the main Eclipse download site). You can also view download stats for files downloaded via p2 if you enable your p2 repository for download statistics. To view these statistics, use the Live Download Statistics tool (Portal > Project Committer > Tools for all Committers).
For more information, please see the Project Download Stats page.
View my disk space quota?
Because the downloads content is mirrored worldwide, Eclipse.org imposes disk space quotas to not overburden our mirror sites. There are no quotas on mail, CVS or www.eclipse.org website content. New projects are configured with quotas. If this is insufficient, we can increase the quota to suit your needs. However, before increasing a quota, we will make sure that your downloads area doesn't contain old or stale files. We appreciate you keeping the downloads areas as lean and clean as possible.
You can view your project's download.eclipse.org disk usage and quota by logging into the Portal > [tools] for all Committers > Disk space and quotas.
Increase my disk space quota?
Before requesting your quota be increased, please delete any old files that are no longer required, and move older release builds to archive.eclipse.org (instructions above). If you are confident that your download.eclipse.org footprint is as small as it can be and that you're still running out of space, simply send an e-mail to the WebMaster with your request, stating which project you're on.
Sign my plugins/ZIP files?
The Eclipse Foundation will allow one or two committers on each project to sign JAR and ZIP files on its behalf. Signing is done on the build.eclipse.org server, using your CVS userid and an SSH command line. To sign, simply get your PMC or Project Lead to contact the email@example.com to indicate the committers (max. 2) that should have signing privilege. Typically, the release engineers, build teams or whoever puts the files on download.eclipse.org should be the signers.
Access/use the Eclipse Build Server?
Committers can use the build.eclipse.org server to run builds and tests for their project. Unlike the other eclipse.org servers, committers are permitted to run software on this server, and to maintain running software in the background. If you need to run cron jobs, please contact the webmaster, stating the time and frequency at which these jobs are to run, and for how long they typically run. Server details: host: build.eclipse.org username: use your committer account server: IBM P5 550 quad-cpu, 16G of RAM architecture: PPC64
You can use an SSH client to connect to the server. Here are some directories that are of interest: /cvsroot -> the CVS repositories, connected to eclipse.org via a Gigabit connection. Your Build account cannot write to these files directly
/home/data/httpd/download.eclipse.org -> the download.eclipse.org root, connected to eclipse.org via Gigabit connection.
/shared -> a shared disk to store your build files and applications. Please note, however, that we do not maintain backups of this directory. This path is structured like the downloads area, and is accessible via http://build.eclipse.org/
/shared/common -> a common location to store applications. Ant and JDK 5.0 are located there.
If you have any questions, please contact the webmaster.
Access/request Hudson services
- Access is restricted to active committers, You can login to Hudson with your committer credentials.
- Our Hudson build service is managed in the same way that our blog aggregator site(Planeteclipse.org) is. in order to create a build for your project, you must be a member of the callisto-dev group, if so simply file a bug against Community->Hudson, and our team of volunteer admins will help you out.
Setup a new mailing list?
Because Mailing Lists are subject to SPAM and can adversely affect eclipse.org performance (imaging sending 200 e-mails to a list that contains 3000 members), proper care is taken in configuring each list. New mailing lists are set up by the WebMaster for this reason. Also, the webmaster creates an HTML view (called mailing list archives) of mailing list postings for archive and search purposes.
View list members?
Because mailing lists contain private information, such as a member's e-mail address, name and surname, we cannot publicly display this information. However, the PMC or Project Lead can become the list administrator, which would allow you to view the membership information for your lists. The PMC/Project lead can inquire about list administration to the WebMaster, stating which lists they would like to manage.
Create a new page in the Eclipse Wiki
To create a new page, simply type the page name at the end of "/" in the URL. The name can contain spaces. For instance, http://wiki.eclipse.org/Some_Page will allow you to create and edit this new page.
Eclipse Foundation IT SLA
When you become committer, your default shell allows only CVS and SVN commands. If you need a 'real' shell for dealing with distribution files or working with automated builds, you'll need to have your project lead or the project PMC file a bug requesting the upgrade.
This page is moderated by Anne Jacko and Bjorn Freeman-Benson (Eclipse Foundation)