Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "IT Infrastructure Doc"

(Downloads)
(macOS signing: update link to migrated archive)
 
(107 intermediate revisions by 27 users not shown)
Line 3: Line 3:
 
==Website==
 
==Website==
 
===How do I setup my project website?===
 
===How do I setup my project website?===
Project websites are hosted in a CVS repository separate from the actual project code. The repository path is dev.eclipse.org:/cvsroot/org.eclipse, in the www component.
+
Project websites are hosted in a git repository separate from the actual project code. You can browse project website repositories using [https://git.eclipse.org/c/www.eclipse.org cGit]. Once the webmaster adds a space for your project, files you commit to the website repository are automatically published to www.eclipse.org/xyz, where  
Once the webmaster
+
adds a space for your project, files you commit to the website CVS are automatically checked out to www.eclipse.org/xyz, where  
+
 
xyz is your project's short name.
 
xyz is your project's short name.
 
You are free to use HTML and PHP on your website.
 
You are free to use HTML and PHP on your website.
<br >sting a project website is normally done when the project proposal has been approved.
+
<br >Hosting a project website is normally done when the project proposal has been approved.
 
If you suspect your files are not being checked out to the www.eclipse.org website, simply commit a small change to one file. This is usually
 
If you suspect your files are not being checked out to the www.eclipse.org website, simply commit a small change to one file. This is usually
 
enough to trigger a website refresh.
 
enough to trigger a website refresh.
===How do I author web pages using the Phoenix method?===
 
Please see <a href="phoenix.php">this document</a> for information on using Phoenix.
 
You can also check out: [[Using Phoenix]] and [http://www.eclipse.org/phoenix/docs/sample_pages.php Sample Pages]
 
  
===Access the Bugzilla database using PHP?===
+
===How do I use the Solstice theme?===
Please see the section labeled "[tools]" in the [http://portal.eclipse.org|MyFoundation Portal]
+
Please see [https://eclipse.org/eclipse.org-common/themes/solstice/docs/ this document] for information on using Solstice.
  
 
===Use a database for my website?===
 
===Use a database for my website?===
Line 24: Line 19:
 
Large (1 MB+) ZIP and JAR files must be put in the downloads area, using the Find A Mirror script to link to them.
 
Large (1 MB+) ZIP and JAR files must be put in the downloads area, using the Find A Mirror script to link to them.
 
However, small files (less than 1 MB) can be put on the www.eclipse.org/yourproject website directly without causing too much harm.
 
However, small files (less than 1 MB) can be put on the www.eclipse.org/yourproject website directly without causing too much harm.
 
The Find A Mirror script supports transparent mirror use, so large screencasts and PDFs can be put in the downloads area as well without imposing the added step of selecting
 
a mirror site for the file.  Simply add &amp;r=1 to the URL.  For instance, http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.1-200506271435/eclipse-SDK-3.1-linux-gtk.tar.gz&amp;r=1 will fetch you the
 
Eclipse SDK 3.1 for Linux from a random mirror site without asking you which one.
 
  
 
Remember to allow our mirrors at least 24 hours to sync up before using a transparent mirror redirect.  
 
Remember to allow our mirrors at least 24 hours to sync up before using a transparent mirror redirect.  
  
 
===Use PHP on my website?===
 
===Use PHP on my website?===
PHP support is available on www.eclipse.org only. Simply commit files with the .php file extension to your website's CVS repository.
+
PHP support is available on www.eclipse.org only. Simply commit files with the .php file extension to your website's repository.
 
Although some projects host PHP files on download.eclipse.org, we do not encourage or recommend it.
 
Although some projects host PHP files on download.eclipse.org, we do not encourage or recommend it.
  
Line 50: Line 41:
 
There are many, many other security and PHP best-practices.  These are just the basics.
 
There are many, many other security and PHP best-practices.  These are just the basics.
  
==CVS==
+
==SSH==
===Connect to Eclipse CVS?===
+
===Shells===
Please see [http://dev.eclipse.org/cvshowto.html this page].
+
* Shell access on eclipse.org servers is not supported. See reasons on https://www.eclipse.org/lists/eclipse.org-committers/msg01075.html
 
+
===Connect to Eclipse CVS when PSERVER and/or EXTSSH are firewalled?===
+
Please see the Proxy configuration on [[CVS Howto|this page]].
+
 
+
===Delete files from CVS?===
+
Although you can use SSH and a terminal to delete files in your CVS repository, we recommend you open a Bugzilla bug, in Community CVS,
+
requesting the files that need to be deleted.
+
 
+
===Manage UNIX groups for CVS access?===
+
The unix groups are essentially webmaster tools used to manage commit rights to CVS
+
repositories and to the downloads area.
+
For each project (Eclipse-Foundation-sanctioned project, such as Eclipse Platform, DSDP-DD, Mylar,
+
CDT, etc) we typically create three groups:
+
 
+
* '''project-dev:''' the group of accounts that can commit to the project's code repository
+
* '''project-home:''' the group of accounts that can commit to the project'ss website
+
* '''projectadmin:''' those who can store files in the downloads area.
+
 
+
For some projects, having all committers in one group with commit rights across the entire
+
project is not adequate when some committers must be limited to a specific set of modules. 
+
In these cases, we create project-module groups that allow specific committers to only
+
commit to that portion of CVS.
+
  
 
==Bugzilla==
 
==Bugzilla==
 
===Create a new Component/Version/Milestone/Target?===
 
===Create a new Component/Version/Milestone/Target?===
[http://wiki.eclipse.org/index.php/Webmaster_FAQ#I_need_to_add.2Fremove.change_a_version.2Fmilestone.2Fcomponent_in_Bugzilla._How_do_I_do_this.3F Please see the documentation here]
+
For the Eclipse.org forge, you can use the Bugzilla Manager tool.  More info is [https://wiki.eclipse.org/index.php/Webmaster_FAQ#I_need_to_add.2Fremove.change_a_version.2Fmilestone.2Fcomponent_in_Bugzilla._How_do_I_do_this.3F documented here].
 +
 
 +
For other forges, Bugzilla changes can be requested via a Bugzilla [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Working%20Groups bug against the corresponding Working Group].
  
 
==Downloads==
 
==Downloads==
===Put files on the download server?===
+
=== Upload files to the download server? ===
Downloadable files must be placed in the downloads area (~/downloads, or /home/data/httpd/download.eclipse.org) so they can be mirrored to our mirror sites worldwide.
+
  
'''Please ensure only pertinent, current files are in the downloads area''', as we cannot store an eternity of nightly, integration and stable builds. Production releases can be kept forever; however, we ask that you move archived releases to archive.eclipse.org (see below).
+
Downloadable files must be placed in the downloads area (~/downloads, or /home/data/httpd/download.eclipse.org) so they can be mirrored to our mirror sites worldwide. '''Please ensure only pertinent, current files are in the downloads area''', as we cannot store an eternity of nightly, integration and stable builds. Production releases can be kept forever; however, we ask that you move archived releases to archive.eclipse.org (see below).  
  
To transfer your files, use an SCP (committers,release engineers) or SFTP (build engineers only) client and connect to dev.eclipse.org (or build.eclipse.org) using your committer account. Transfer files to your project's directory in the downloads area (Typically ~/downloads/toplevel/yourproject). Your project's
+
To upload your files:
downloads directory is typically communicated to the Project Lead upon project provisioning.  '''Please ensure that the file permissions include world-readable (664; rw-rw-r--) and directory permissions allow for world-executable (775, rwxrwxr-x).'''
+
*Use [[Jenkins]] to upload your files, see [https://wiki.eclipse.org/Jenkins#How_do_I_deploy_artifacts_to_download.eclipse.org.3F How do I deploy artifacts to download.eclipse.org?].  (Formerly, SFTP or SCP client (in SFTP mode) was used to connect to build.eclipse.org using your committer account, however this is no longer supported).
 +
*'''Please ensure that the file permissions include world-readable (664; rw-rw-r--) and directory permissions allow for world-executable (775, rwxrwxr-x).'''  
  
Large projects with frequent builds may find it more convenient to use RSYNC over SSH.
+
*Although you can link directly to download.eclipse.org/yourfile.zip, you can also use the Find a Mirror script (info below). Using this script allows you to view download statistics and allows users to pick a nearby mirror site for their download.
  
Once your files are on the download.eclipse.org server, they are immediately available to the general public. However, for release builds, we ask that you wait at least four hours for our mirror sites to fetch the new files before linking to them. It typically takes a day or two for all the mirror
+
Once your files are on the download.eclipse.org server, they are immediately available to the general public. However, for release builds, we ask that you wait at least four hours for our mirror sites to fetch the new files before linking to them. It typically takes a day or two for all the mirror sites to synchronize with us and get new files.  
sites to synchronize with us and get new files.
+
 
+
To make your downloads available to the general public, please '''do not link directly to download.eclipse.org'''. Instead, use the Find a Mirror script (info below). Using this script allows you to view download statistics and allows users to pick a nearby mirror site for their download.
+
  
 
Please note that although we tolerate PHP, HTML and JPG/GIF files on download.eclipse.org, we encourage you to put such files on www.eclipse.org. Those files are not mirrored to public mirror servers.
 
Please note that although we tolerate PHP, HTML and JPG/GIF files on download.eclipse.org, we encourage you to put such files on www.eclipse.org. Those files are not mirrored to public mirror servers.
  
To save disk space on our mirror servers, and to reclaim some quota space, we recommend you move old release builds to archive.eclipse.org.
+
'''SYMLINKS''' are not supported. We cannot ensure that all our mirror servers support and honour symlinks. For that reason, please avoid the usage of symlinks.
  
 
===Move files to archive.eclipse.org?===
 
===Move files to archive.eclipse.org?===
  
Because our mirror sites don't have as much disk space for Eclipse files as we do, we have created an http://archive.eclipse.org site for you to
+
Because our mirror sites don't have as much disk space for Eclipse files as we do, we have created an https://archive.eclipse.org site for you to
 
store older release builds.
 
store older release builds.
  
The archive.eclipse.org structure is similar to that of download.eclipse.org.  To move your files, we recommend using the SSH prompt as below. If you are
+
The archive.eclipse.org structure is similar to that of download.eclipse.org.  To move your files, we recommend using a job on your project's Jenkins instance. Alternatively, you can navigate to https://download.eclipse.org/path/to/your/project. From download.eclipse.org, authenticated committers can Archive files and folders (the archive process maintains the directory structure). From https://archive.eclipse.org/path/to/your/project files and folders can be permanently deleted.
not comfortable with the SSH prompt, you can ask WebMaster to move the files for you.
+
  
    ssh yourcommitterid@dev.eclipse.org
+
Some folders contain an index file - such as index.html, which will be shown instead of the directory contents. Append /listing to the URL and the contents will be shown. https://download.eclipse.org/path/to/your/projectdirectory/listing
    mv ~/downloads/your/project/oldrelease/ /home/data/httpd/archive.eclipse.org/your/project/oldrelease/
+
  
'''Note''': if you preserve the exact path and filename from download.eclipse.org to archive.eclipse.org, you don't need to change your links if your links use the Find a Mirror script.
+
'''Note''': if you preserve the exact path and filename from download.eclipse.org to archive.eclipse.org, you don't need to change your links (although it is recommended). This works for p2 repos, direct links to https://download.eclipse.org and if your links use the Find a Mirror script.
  
 
This link will work if /path/to/a/file.zip is on download.eclipse.org, or if it gets moved to the same place on archive.eclipse.org
 
This link will work if /path/to/a/file.zip is on download.eclipse.org, or if it gets moved to the same place on archive.eclipse.org
   http://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip
+
   https://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip
  
===See which mirror sites are mirroring my files?===
+
'''P2 repositories''': P2 repositories are not normally accessed via the mirror selection script. Therefore, extra treatment is required when the move should be made transparently without affecting users who may still have the original URL.  
You can use the Find a Mirror script to see which mirror sites have your files. The Find a Mirror script is: http://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip
+
  
'''Parameters:'''
+
[[Equinox/p2/p2.mirrorsURL#Moving_a_repo_to_archive.eclipse.org]] has a discussion how to achieve this (''work in progress'').
 +
 
 +
===Use mirror sites/see which mirrors are mirroring my files?===
 +
Link to your download files like this:
 +
 
 +
    Acceptable: https://download.eclipse.org/path/to/a/file.zip   
 +
 
 +
    Preferred: https://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip
 +
 
 +
'''Parameters for above script:'''
 
* '''file''' (Required): specify the filename, relative to the downloads home, starting with a "/". This file must exist in the downloads area.  Although you can specify a directory name, your mirror list will be more accurate if you specify a file.
 
* '''file''' (Required): specify the filename, relative to the downloads home, starting with a "/". This file must exist in the downloads area.  Although you can specify a directory name, your mirror list will be more accurate if you specify a file.
 
* '''format''' (Optional): specify html (default) or xml.  Useful for building the mirrors.xml for Update sites.
 
* '''format''' (Optional): specify html (default) or xml.  Useful for building the mirrors.xml for Update sites.
 
* '''protocol''' (Optional): ftp or http: list only ftp or http mirrors only (both are the default)
 
* '''protocol''' (Optional): ftp or http: list only ftp or http mirrors only (both are the default)
* '''r''' (Optional): specify 1 to automatically redirect to the best mirror (the one that would normally be at the top) without asking the user to choose.
+
* '''r''' (DEPRECATED): specify 1 to automatically redirect to the best mirror (the one that would normally be at the top) without asking the user to choose.
* '''nf''' (Optional): specify 1 to get an actual 404 Not Found error if the file doesn't exist (instead of a lovely page saying so).
+
* '''nf''' (DERECATED): specify 1 to get an actual 404 Not Found error if the file doesn't exist (instead of a lovely page saying so).
  
 
The script will examine the Last Modified timestamp of the given file and return only those mirrors that have synchronized with Eclipse.org after that time.
 
The script will examine the Last Modified timestamp of the given file and return only those mirrors that have synchronized with Eclipse.org after that time.
Line 130: Line 104:
 
Examples:
 
Examples:
 
     All mirrors of the Lepido project, in XML format:
 
     All mirrors of the Lepido project, in XML format:
     http://www.eclipse.org/downloads/download.php?file=/technology/lepido/M1/content.jar&amp;format=xml
+
     https://www.eclipse.org/downloads/download.php?file=/technology/lepido/M1/content.jar&amp;format=xml
  
 
     Get a file from a random mirror, without prompting
 
     Get a file from a random mirror, without prompting
     http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.1-200506271435/eclipse-SDK-3.1-win32.zip&amp;r=1
+
     https://download.eclipse.org/eclipse/downloads/drops/R-3.1-200506271435/eclipse-SDK-3.1-win32.zip
  
  
Line 140: Line 114:
 
therefore it's typically more costly (in terms of bandwidth) to mirror them than to support the few client downloads they generate.
 
therefore it's typically more costly (in terms of bandwidth) to mirror them than to support the few client downloads they generate.
 
At time of writing, our exclusion list is:  
 
At time of writing, our exclusion list is:  
*drops/I*
+
*.nfs*
*drops/N*
+
* apitools/
*drops/M*
+
* apidocs/
*webtools/committers/
+
* archive/
*callisto/staging/
+
* archives/
*callisto/testUpdates/
+
* /athena
*eclipse/testUpdates/
+
* builds/N*
*eclipse/updates/3.2milestones
+
* drops/I*
*dev/TPTP*
+
* drops/N*
*tools/cdt/builds/*
+
* drops/M*
*modeling/gmf/downloads/drops/B*
+
* *.jpg
 +
* *.gif
 +
* callisto/*
 +
* compilelogs/
 +
* eclipse.org-common/
 +
* eclipse/testUpdates*
 +
* eclipse/updates/3.2milestones
 +
* /eclipse/updates/3.6-I-builds/
 +
* *eclipse/updates/*-X*
 +
* *eclipse/updates/*-Y*
 +
* dev/TPTP*
 +
* /tools/cdt/builds
 +
* modeling/gmf/downloads/drops/B*
 
* *drops/*/N*
 
* *drops/*/N*
 
* *drops/*/I*
 
* *drops/*/I*
 +
* *javadoc/
 +
* *javadocs/
 +
* linuxtools/N*
 +
* *nightly*
 +
* *Nightly*
 +
* *staging*
 +
* /webtools/downloads/drops/*/M*
 +
* performance/
 +
* /releases/staging
 +
* /releases/europa
 +
* testresults/
 +
* /rt/eclipselink/nightly*
 +
* /technology/babel/update-site*
 +
* /technology/cosmos
 +
* /technology/ohf
 +
* /technology/tigerstripe
 +
* testcompilelogs/
 +
* testResults/
 +
* /tools/downloads
 +
* /tools/orbit/committers
 +
* */N202*
 +
* */I202*
 +
* */I.I202*
 +
* */I-*
 +
* */N-*
 +
* *integration*/
 +
* xref/
 +
* */M20*
 +
* /rt/eclipselink/maven.repo*
  
 
===Use the Find a Mirror script?===
 
===Use the Find a Mirror script?===
Line 158: Line 173:
  
 
===Enable mirrors / use mirrorsURL for my p2 repo?===
 
===Enable mirrors / use mirrorsURL for my p2 repo?===
our artifacts.xml (jar) should have a p2.mirrorsURL property.  Here is a an example from http://download.eclipse.org/eclipse/updates/3.6/R-3.6.2-201102101200/artifacts.jar
+
 
 +
Your artifacts.xml (jar) should have a p2.mirrorsURL property.  Here is a an example from https://download.eclipse.org/eclipse/updates/3.6/R-3.6.2-201102101200/artifacts.jar
  
 
     <repository name='&quot;Eclipse Project Test Site&quot;' type='org.eclipse.equinox.p2.artifact.repository.simpleRepository' version='1'>
 
     <repository name='&quot;Eclipse Project Test Site&quot;' type='org.eclipse.equinox.p2.artifact.repository.simpleRepository' version='1'>
Line 165: Line 181:
 
         <property name='p2.timestamp' value='1297373227427'/>
 
         <property name='p2.timestamp' value='1297373227427'/>
 
         <property name='publishPackFilesAsSiblings' value='true'/>
 
         <property name='publishPackFilesAsSiblings' value='true'/>
         <property name='p2.mirrorsURL' value='http://www.eclipse.org/downloads/download.php?file=/eclipse/updates/3.6/R-3.6.2-201102101200&amp;format=xml'/>
+
         <property name='p2.mirrorsURL' value='https://www.eclipse.org/downloads/download.php?file=/eclipse/updates/3.6/R-3.6.2-201102101200&amp;format=xml'/>
 
       </properties>
 
       </properties>
 +
 +
A more detailed description can be found at [[Equinox/p2/p2.mirrorsURL]].
 +
 +
Ideally, '''everyone''', for all p2 repositories, should use this property, since even if not mirrored currently, it does not hurt anything in that case, and you never know when your repository might become mirrored. In fact, failure to use this property can result in too many requests for jar files coming directly to 'download.eclipse.org' and greatly slow down the network and use too much bandwidth. If this happens for your project (or repository) measures may be taken to automatically redirect all such requests somewhere else, which often does not work well; for examples, see {{bug|368826}}.
 +
 +
===Include a p2.index file at p2 repository site?===
 +
 +
A little documented aide to p2 is to include a special file named "p2.index" at your p2 repository URL site. Every well-behaved, well-optimized p2 repository should have one. This is especially important for composite repository sites as it can save several unsuccessful round trips to download server looking for files that do not exist. For "how to" instructions, see the [[Equinox/p2/p2_index| p2 wiki]]. For history and deeper technical discussion, see {{bug|347448}}.
  
 
===See download statistics?===
 
===See download statistics?===
The Find a Mirror script tracks download requests once the user has picked a mirror site (or the main Eclipse download site).  You can also view download stats for files downloaded via p2 if you [[Equinox p2 download stats|enable your p2 repository for download statistics]].  To view these statistics, use the Live Download Statistics tool (Portal > Project Committer > Tools for all Committers).
+
The Find a Mirror script tracks download requests once the user has picked a mirror site (or the main Eclipse download site).  You can also view download stats for files downloaded via p2 if you [[Equinox p2 download stats|enable your p2 repository for download statistics]].  To view these statistics, use the Live Download Statistics tool (Portal > Project Committer > Tools for all Committers). Download statistics are not available for direct downloads.
  
 
For more information, please see the [[Project Download Stats]] page.
 
For more information, please see the [[Project Download Stats]] page.
  
===View my disk space quota?===
+
===Sign my Jar/plugins/Windows exe/macOS App files?===
Because the downloads content is mirrored worldwide, Eclipse.org imposes disk space quotas to not overburden our mirror sites. There are no quotas on mail, CVS or www.eclipse.org website content. New projects are configured with quotas. If this is insufficient, we can increase the quota to suit your needs. However, before increasing a quota, we will make sure that your downloads area doesn't contain old or stale files. We appreciate you keeping the downloads areas as lean and clean as possible.
+
The Eclipse Foundation allows committers to sign JAR and some executable files on its behalf. Signing is done from any of the Jenkins servers. There are three ways to sign:
  
You can view your project's download.eclipse.org disk usage and quota by logging into the [http://portal.eclipse.org Portal] > [tools] for all Committers > Disk space and quotas.
+
==== CBI Maven signing plugin ====
  
===Increase my disk space quota?===
+
Using the CBI Maven Plugins the signing process can be directly performed at the end of a Maven Tycho build.
Before requesting your quota be increased, please delete any old files that are no longer required, and move older release builds
+
to archive.eclipse.org (instructions above).  If you are confident that your download.eclipse.org footprint is as small as it can be
+
and that you're still running out of space, simply send an e-mail to the WebMaster with your request, stating which project you're on.
+
  
===Sign my plugins/ZIP files?===
+
{{important|Maven Profiles|As signing an Eclipse project is only available from an Eclipse Jenkins server, a common practice is to place the CBI Maven signing plugins in a dedicated profile and enable that profile only in the Jenkins job. This way you can still run your Maven Tycho builds locally without signing. See [https://maven.apache.org/guides/introduction/introduction-to-profiles.html Maven - Introduction to Build Profiles]
The Eclipse Foundation will allow one or two committers on each project to sign JAR and ZIP files on its behalf. Signing is done on the build.eclipse.org server, using your
+
CVS userid and an SSH command line.  To sign, simply get your PMC or Project Lead to contact the webmaster@eclipse.org to indicate the committers (max. 2) that should have
+
signing privilege. Typically, the release engineers, build teams or whoever puts the files on download.eclipse.org should be the signers.
+
  
==Builds==
+
    <profiles>
 +
      <profile>
 +
        <id>sign</id>
 +
        <build>
 +
          <plugins>
 +
            ...
 +
          </plugins>
 +
        </build>
 +
      </profile>
 +
    </profiles>
  
===Access/use the Eclipse Build Server?===
+
The profile can then be activated in the Jenkins build via the -P argument.
[[Image:Build_infra_layout.png|thumb|Build and Hudson storage layout]]
+
}}
  
Committers can use the build.eclipse.org server to run builds and tests for their project.
+
==== JAR signing ====
Unlike the other eclipse.org servers, committers are permitted to run software on this server,
+
and to maintain running software in the background. If you need to run cron jobs, please contact
+
the webmaster, stating the time and frequency at which these jobs are to run, and for how long
+
they typically run.
+
'''Server details:'''
+
host: build.eclipse.org
+
username: use your committer account
+
server: Intel Dual-Quad Xeon E5540 @ 2.53GHz, 24G RAM
+
architecture: x86_64
+
  
You can use an SSH client to connect to the server.  Here are some directories that are of interest:
+
Ensure that all created JAR files are correctly signed by using the [https://www.eclipse.org/cbi/sitedocs/eclipse-jarsigner-plugin/plugin-info.html eclipse-jarsigner-plugin]
'''/cvsroot'''  -> the CVS repositories, connected to eclipse.org via a Gigabit connection. Your Build account cannot write to these files directly
+
  
'''/home/data/httpd/download.eclipse.org''' -> the download.eclipse.org root, connected to eclipse.org via Gigabit connection.
+
    <plugin>
 +
      <groupId>org.eclipse.cbi.maven.plugins</groupId>
 +
      <artifactId>eclipse-jarsigner-plugin</artifactId>
 +
      <version>${cbi-version}</version>
 +
      <executions>
 +
        <execution>
 +
          <id>sign</id>
 +
          <phase>verify</phase>
 +
          <goals>
 +
            <goal>sign</goal>
 +
          </goals>
 +
        </execution>
 +
      </executions>
 +
    </plugin>
  
'''/shared''' -> a shared disk to store your build files and applications. Please note, however, that we do not maintain backups of
+
==== Windows signing ====
this directory. This path is structured like the downloads area, and is accessible via http://build.eclipse.org/ (That URL redirects to the Eclipse homepage, but browsing to a specific project URL will work: http://build.eclipse.org/technology/,  http://build.eclipse.org/tools/ etc.)
+
  
'''/shared/common''' -> a common location to store applications. Ant and JDK 5.0 are located there.
+
To sign the Windows executables use the [https://www.eclipse.org/cbi/sitedocs/eclipse-winsigner-plugin/plugin-info.html eclipse-winsigner-plugin]
  
If you have any questions, please contact the webmaster.
+
    <plugin>
 +
      <groupId>org.eclipse.cbi.maven.plugins</groupId>
 +
      <artifactId>eclipse-winsigner-plugin</artifactId>
 +
      <version>${cbi-version}</version>
 +
      <executions>
 +
        <execution>
 +
          <id>sign</id>
 +
          <goals>
 +
            <goal>sign</goal>
 +
          </goals>
 +
          <phase>package</phase>
 +
          <configuration>
 +
            <signFiles>
 +
              <signFile>${project.build.directory}/products/${product-folder}/win32/win32/x86_64/eclipse.exe</signFile>
 +
              <signFile>${project.build.directory}/products/${product-folder}/win32/win32/x86_64/eclipsec.exe</signFile>
 +
            </signFiles>
 +
          </configuration>
 +
        </execution>
 +
      </executions>
 +
    </plugin>
  
===Access/request Hudson services===
+
==== macOS signing ====
  
Please see the [[Hudson]] document.
+
To sign the macOS executables use the [https://www.eclipse.org/cbi/sitedocs/eclipse-macsigner-plugin/plugin-info.html eclipse-macsigner-plugin]
 +
 
 +
    <plugin>
 +
      <groupId>org.eclipse.cbi.maven.plugins</groupId>
 +
      <artifactId>eclipse-macsigner-plugin</artifactId>
 +
      <version>${cbi-version}</version>
 +
      <executions>
 +
        <execution>
 +
          <id>sign</id>
 +
          <goals>
 +
            <goal>sign</goal>
 +
          </goals>
 +
          <phase>package</phase>
 +
          <configuration>
 +
            <signFiles>
 +
              <signFile>${project.build.directory}/products/${product-folder}/macosx/cocoa/x86_64/Eclipse.app</signFile>
 +
            </signFiles>
 +
            <timeoutMillis>300000</timeoutMillis> <!-- 5 min -->
 +
            <continueOnFail>${macSigner.forceContinue}</continueOnFail>
 +
            <entitlements>${project.basedir}/application.entitlement</entitlements>
 +
          </configuration>
 +
        </execution>
 +
      </executions>
 +
    </plugin>
 +
 
 +
{{important|Entitlements|The security guidelines for macOS application development requires the definition of [https://developer.apple.com/library/archive/documentation/Miscellaneous/Reference/EntitlementKeyReference/Chapters/AboutEntitlements.html Entitlements] to grant an executable permission to use a service or technology. The entitlements used by the Eclipse Platform are defined [https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/tree/master/eclipse.platform.releng.tychoeclipsebuilder/entitlement here]}}
 +
 
 +
==== macOS DMG file creation ====
 +
 
 +
macOS applications are typically published as .dmg files, which are containers that serve as installers with additional security information to avoid that the application gets tampered. To create a DMG file the [https://www.eclipse.org/cbi/sitedocs/eclipse-dmg-packager/plugin-info.html eclipse-dmg-packager] can be used.
 +
 
 +
    <plugin>
 +
      <groupId>org.eclipse.cbi.maven.plugins</groupId>
 +
      <artifactId>eclipse-dmg-packager</artifactId>
 +
      <version>${cbi-version}</version>
 +
      <executions>
 +
        <execution>
 +
          <goals>
 +
            <goal>package-dmg</goal>
 +
          </goals>
 +
          <phase>integration-test</phase>
 +
          <configuration>
 +
            &lt;source&gt;${project.build.directory}/products/${product-id}-macosx.cocoa.x86_64.tar.gz&lt;/source&gt;
 +
            <continueOnFail>true</continueOnFail>
 +
            <timeoutMillis>600000</timeoutMillis> <!-- 10 min -->
 +
            <continueOnFail>${macSigner.forceContinue}</continueOnFail>
 +
            <sign>true</sign>
 +
          </configuration>
 +
        </execution>
 +
      </executions>
 +
    </plugin>
 +
 
 +
==== macOS Notarization ====
 +
 
 +
Since macOS Catalina macOS software that is published outside the AppStore needs to be [https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution notarized], so the Gatekeeper gets information about trusting the software or not.
 +
 
 +
As of now the notarization is not available as Tycho plugin. Therefore the macos-notarization-service webservice needs to be used in the Jenkins job similar to the following snippet:
 +
 
 +
    PRODUCT_ID=...
 +
    BUILD_DIR="${WORKSPACE}/${PRODUCT_ID}/target/products/"
 +
    DMG=${PRODUCT_ID}-macosx.cocoa.x86_64.dmg
 +
   
 +
    pushd $BUILD_DIR
 +
   
 +
    PRIMARY_BUNDLE_ID="app-bundle"
 +
   
 +
    RESPONSE=$(curl -s -X POST -F file=@${DMG} -F 'options={"primaryBundleId": "'${PRIMARY_BUNDLE_ID}'", "staple": true};type=application/json' https://cbi.eclipse.org/macos/xcrun/notarize)
 +
     
 +
    UUID=$(echo $RESPONSE | grep -Po '"uuid"\s*:\s*"\K[^"]+')
 +
    STATUS=$(echo $RESPONSE | grep -Po '"status"\s*:\s*"\K[^"]+')
 +
   
 +
    while [[ ${STATUS} == 'IN_PROGRESS' ]]; do
 +
      sleep 1m
 +
      RESPONSE=$(curl -s https://cbi.eclipse.org/macos/xcrun/${UUID}/status)
 +
      STATUS=$(echo $RESPONSE | grep -Po '"status"\s*:\s*"\K[^"]+')
 +
    done
 +
   
 +
    if [[ ${STATUS} != 'COMPLETE' ]]; then
 +
      echo "Notarization failed: ${RESPONSE}"
 +
      exit 1
 +
    fi
 +
   
 +
    rm "${DMG}"
 +
   
 +
    curl -JO https://cbi.eclipse.org/macos/xcrun/${UUID}/download
 +
    popd
 +
 
 +
A more detailed script is the [https://git.eclipse.org/c/oomph/org.eclipse.oomph.git/tree/releng/org.eclipse.oomph.releng/hudson/repackage.sh#n91 Oomph script].
 +
 
 +
{{important|CBI Maven Plugins Version|For correct signing needed for notarization (including for example hardened runtime) at least the CBI Plugins version 1.1.8-SNAPSHOT needs to be used. Ensure to configure the correct pluginRepository to be able to consume that version
 +
    <pluginRepositories>
 +
      <pluginRepository>
 +
        <id>cbi</id>
 +
        <url>https://repo.eclipse.org/content/repositories/cbi-releases/</url>
 +
      </pluginRepository>
 +
      <pluginRepository>
 +
        <id>cbi-snapshots</id>
 +
        <url>https://repo.eclipse.org/content/repositories/cbi-snapshots/</url>
 +
        <snapshots>
 +
          <enabled>true</enabled>
 +
        </snapshots>
 +
      </pluginRepository>
 +
    </pluginRepositories>
 +
 
 +
Once 1.1.8 is released, the SNAPSHOTS repository is not needed anymore.
 +
}}
 +
 
 +
{{important|Eclipse Platform Version|The macOS notarization will only succeed if the dmg file signing is matching certain criteria (e.g. hardened runtime). The first Eclipse Platform version that includes the Eclipse Launcher and native libraries that match those criterias are included in '''2019-09'''. For any previous versions the notarization will fail.}}
 +
 
 +
For further information on the CBI Maven Plugins have a look at: https://www.eclipse.org/cbi/sitedocs/
 +
 
 +
Note that these plugins use the web services in the background.
 +
 
 +
==== Web service ====
 +
Using a web POST method, individual JAR files can be signed from any of the internal Jenkins servers with this service:
 +
 
 +
    https://cbi.eclipse.org/jarsigner/sign
 +
 
 +
The output of that service will be the signed file.  '''Please note''' that the web service does not pack or process jar files.  You must condition/pack them yourself '''prior''' to signing if you wish to do so.
 +
 
 +
{{important|Resigning Jarsigner|The web service '''always resigns already signed jars'''. The maven jar signer plugin lets you specify a [https://www.eclipse.org/cbi/sitedocs/eclipse-jarsigner-plugin/sign-mojo.html#resigningStrategy strategy to avoid submitting already signed jar to the webservice]. If you use the webservice directly, you need to do deal with it by yourself. You can see how the re-signing strategies are defined by looking at the code of the
 +
[https://github.com/eclipse-cbi/org.eclipse.cbi/blob/main/maven-plugins/eclipse-jarsigner-plugin/src/main/java/org/eclipse/cbi/maven/plugins/jarsigner/JarResigner.java JarResigner]}}
 +
 
 +
    # JAR FILES: Submit unsigned-jar.jar and save signed output to signedfile.jar
 +
    curl -o signedfile.jar -F file=@unsigned-jar.jar https://cbi.eclipse.org/jarsigner/sign
 +
 
 +
    # WINDOWS EXE: Submit Windows unsigned.exe and save signed output to signed.exe
 +
    curl -o signed.exe -F file=@unsigned.exe https://cbi.eclipse.org/authenticode/sign
 +
 
 +
    # WINDOWS MSI: Submit Windows unsigned.msi and save signed output to signed.msi
 +
    curl -o signed.msi -F file=@unsigned.msi https://cbi.eclipse.org/authenticode/sign
 +
 
 +
    # MAC: Submit unsigned and save signed output to signed.zip
 +
    # Note: You must zip your entire *.app directory for example: zip -r unsigned.zip Eclipse.app
 +
    curl -o signed.zip -F file=@unsigned.zip https://cbi.eclipse.org/macos/codesign/sign
 +
 
 +
    # If you need to set entitlements on your app / binary (see https://developer.apple.com/documentation/security/hardened_runtime?preferredLanguage=occ for details),
 +
    # add an `entitlements` part to the request like below
 +
    curl -o signed.zip -F file=@unsigned.zip -F entitlements=@file.entitlements https://cbi.eclipse.org/macos/codesign/sign
 +
 
 +
Using the webservice is equally easy from Ant. Note that ${filename} cannot be a path. Input and output file name can be the same.
 +
 
 +
    <exec dir="${dirname}" executable="curl">
 +
      <arg value="--output"/>
 +
      <arg value="${filename}"/>
 +
      <arg value="--form"/>
 +
      <arg value="file=@${filename}"/>
 +
      <arg value="--silent"/>
 +
      <arg value="--show-error"/>
 +
      <arg value="--fail"/>
 +
      <arg value="https://cbi.eclipse.org/jarsigner/sign"/>
 +
    </exec>
 +
 
 +
{{important|Version of Jarsigner|The web service only signs with Java 8 version of jarsigner.}}
 +
 
 +
Using the web service to sign Mac and Windows applications is also easy from Tycho, see
 +
* [https://www.eclipse.org/cbi/sitedocs/eclipse-macsigner-plugin/plugin-info.html eclipse-macsigner-plugin]
 +
* [https://www.eclipse.org/cbi/sitedocs/eclipse-winsigner-plugin/sign-mojo.html eclipse-winsigner-plugin]
 +
* [https://web.archive.org/web/20161105100753/http://www.codetrails.com:80/blog/sign-your-eclipse-project Sign your eclipse project] (codetrails.com/archive.org)
 +
* [https://dev.eclipse.org/mhonarc/lists/cbi-dev/msg01640.html OS X application signing] (cbi-dev mailing list)
 +
 
 +
==== What about GPG signing? ====
 +
 
 +
JAR signing of the bundles and GPG-signing of the Maven artifacts are two different steps. Once a jar has been "jar-signed", you may or may not GPG sign the corresponding Maven artifact (.jar + .pom file) so as it can be deployed on Central. As you hinted, JAR signing has to be done before the GPG signing, since doing it the other way around would break the GPG signature.
 +
 
 +
So you first have to sign your JAR file with the Eclipse Fdn certificate, either using the Maven plugin from CBI, the command line utility, or the signing web service – see above.
 +
Once you have your signed JAR, you can GPG sign it and stage it on Central like this:
 +
    mvn gpg:sign-and-deploy-file  \
 +
        -DpomFile=target/myapp-1.0.pom  \
 +
        -Dfile=target/myapp-1.0.jar  \
 +
        -Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2/  \
 +
        -DrepositoryId=sonatype_oss
 +
 
 +
==== Publish to Maven Central ====
 +
 
 +
To deploy to Maven Central from your JIPP, you'll need webmaster's assistance to
 +
* Create a project specific account at Sonatype OSSRH
 +
* Generate a GPG keypair for your JIPP user
 +
* Configure your JIPP to GPG sign and upload artifacts
 +
 
 +
It takes a bit of time but afterwards, you will only be required to use a dedicated Maven settings on your JIPP.
 +
 
 +
To get started, please file a bug against https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues asking for your JIPP to be configured to let you publish to Maven central (don't forget the name of your Eclipse project).
 +
 
 +
If you want to publish jars from already released p2 repositories, consider using the strategy adopted by the Eclipse Platform. More info: [[Platform-releng/Publish To Maven Central]]
 +
 
 +
==Builds==
 +
 
 +
===Access/request Jenkins services===
 +
 
 +
Please see the [[Jenkins]] document.
 +
 
 +
== Code Quality Analysis ==
 +
 
 +
* [[FindBugs]]
 +
* [[Sonar]]
 +
* JDT :), please consider enabling [https://help.eclipse.org/topic/org.eclipse.jdt.doc.user/reference/preferences/java/compiler/ref-preferences-errors-warnings.htm?cp=1_4_2_0_3_1 compiler warnings] beyond the defaults. The JDT help also contains a start of a section on [https://help.eclipse.org/topic/org.eclipse.jdt.doc.user/tasks/task-improve_code_quality.htm?cp=1_3_9 improving code quality].
  
 
==Mailing Lists==
 
==Mailing Lists==
Line 237: Line 481:
 
===Create a new page in the Eclipse Wiki===
 
===Create a new page in the Eclipse Wiki===
 
To create a new page, simply type the page name at the end of "/" in the URL. The name can contain spaces. For instance,
 
To create a new page, simply type the page name at the end of "/" in the URL. The name can contain spaces. For instance,
http://wiki.eclipse.org/Some_Page will allow you to create and edit this new page.
+
https://wiki.eclipse.org/Some_Page will allow you to create and edit this new page.
  
 
==Eclipse Servers==
 
==Eclipse Servers==
  
 
Eclipse Foundation [[IT SLA]]
 
Eclipse Foundation [[IT SLA]]
 
When you become committer, your default shell allows only CVS and SVN commands. 
 
If you need a 'real' shell for dealing with distribution files or working with automated builds, you'll need to have your project lead or the project PMC file a bug requesting the upgrade.
 
 
  
 
''This page is moderated by the EMO''
 
''This page is moderated by the EMO''
 
[[Category:Development_Resources]]
 
[[Category:Development_Resources]]
 
[[Category:How to Contribute]]
 
[[Category:How to Contribute]]

Latest revision as of 09:46, 21 March 2022

< Development Resources

Website

How do I setup my project website?

Project websites are hosted in a git repository separate from the actual project code. You can browse project website repositories using cGit. Once the webmaster adds a space for your project, files you commit to the website repository are automatically published to www.eclipse.org/xyz, where xyz is your project's short name. You are free to use HTML and PHP on your website.
Hosting a project website is normally done when the project proposal has been approved. If you suspect your files are not being checked out to the www.eclipse.org website, simply commit a small change to one file. This is usually enough to trigger a website refresh.

How do I use the Solstice theme?

Please see this document for information on using Solstice.

Use a database for my website?

We currently do not offer projects with database support.

I need to put a large file on my website. How should I do this?

Large (1 MB+) ZIP and JAR files must be put in the downloads area, using the Find A Mirror script to link to them. However, small files (less than 1 MB) can be put on the www.eclipse.org/yourproject website directly without causing too much harm.

Remember to allow our mirrors at least 24 hours to sync up before using a transparent mirror redirect.

Use PHP on my website?

PHP support is available on www.eclipse.org only. Simply commit files with the .php file extension to your website's repository. Although some projects host PHP files on download.eclipse.org, we do not encourage or recommend it.

Eclipse.org is a high-traffic website. Please make sure your PHP code is optimized to run in this type of environment. See the next item.

Optimize my PHP code for large-scale use?

Eclipse.org is a high-traffic website. To improve PHP's functionality, we have set very liberal limits on how many resources PHP can consume. However. if if your project is very popular, bad PHP code can slow the entire site down.

Of course, we could harden PHP to protect our website, but that would cut some functionality. Some tips for you:

  • Never call the web service to include/open files - include("http://www.eclipse.org/somefile.html") and fopen("http://localhost/somefile.xml") are very costly to run, because they call the web service, and can lead to eclipse.org Denial-Of-Servicing itself under heavy load.
  • Never include/open remote files - include("http://www.someothersite.org/somefile.html") is forbidden, as someone could launch a Denial-Of-Service attack against a remote site. We don't allow you to establish remote connections from eclipse.org servers other than the build server.
  • Sanitize your incoming parameters - include($parameter) is particularly dangerous if $parameter is not sanitized. Someone could freely surf the web anonymously, hiding behind eclipse.org servers, or they could use your page to access local files, or launch Denial-Of-Service attacks against remote servers.
  • Cache aggregated, processor-intensive data - SQL aggregations, file system scans, Bugzilla lists can (and should) be cached to avoid redundant processor- and disk-intensive operations. For instance, scanning through download.eclipse.org directories to display the size of a build could be useful, but doesn't need to happen for each website visitor. Cache the results of this operation to a file, and update the file if the file is older than 12 hours.

There are many, many other security and PHP best-practices. These are just the basics.

SSH

Shells

Bugzilla

Create a new Component/Version/Milestone/Target?

For the Eclipse.org forge, you can use the Bugzilla Manager tool. More info is documented here.

For other forges, Bugzilla changes can be requested via a Bugzilla bug against the corresponding Working Group.

Downloads

Upload files to the download server?

Downloadable files must be placed in the downloads area (~/downloads, or /home/data/httpd/download.eclipse.org) so they can be mirrored to our mirror sites worldwide. Please ensure only pertinent, current files are in the downloads area, as we cannot store an eternity of nightly, integration and stable builds. Production releases can be kept forever; however, we ask that you move archived releases to archive.eclipse.org (see below).

To upload your files:

  • Use Jenkins to upload your files, see How do I deploy artifacts to download.eclipse.org?. (Formerly, SFTP or SCP client (in SFTP mode) was used to connect to build.eclipse.org using your committer account, however this is no longer supported).
  • Please ensure that the file permissions include world-readable (664; rw-rw-r--) and directory permissions allow for world-executable (775, rwxrwxr-x).
  • Although you can link directly to download.eclipse.org/yourfile.zip, you can also use the Find a Mirror script (info below). Using this script allows you to view download statistics and allows users to pick a nearby mirror site for their download.

Once your files are on the download.eclipse.org server, they are immediately available to the general public. However, for release builds, we ask that you wait at least four hours for our mirror sites to fetch the new files before linking to them. It typically takes a day or two for all the mirror sites to synchronize with us and get new files.

Please note that although we tolerate PHP, HTML and JPG/GIF files on download.eclipse.org, we encourage you to put such files on www.eclipse.org. Those files are not mirrored to public mirror servers.

SYMLINKS are not supported. We cannot ensure that all our mirror servers support and honour symlinks. For that reason, please avoid the usage of symlinks.

Move files to archive.eclipse.org?

Because our mirror sites don't have as much disk space for Eclipse files as we do, we have created an https://archive.eclipse.org site for you to store older release builds.

The archive.eclipse.org structure is similar to that of download.eclipse.org. To move your files, we recommend using a job on your project's Jenkins instance. Alternatively, you can navigate to https://download.eclipse.org/path/to/your/project. From download.eclipse.org, authenticated committers can Archive files and folders (the archive process maintains the directory structure). From https://archive.eclipse.org/path/to/your/project files and folders can be permanently deleted.

Some folders contain an index file - such as index.html, which will be shown instead of the directory contents. Append /listing to the URL and the contents will be shown. https://download.eclipse.org/path/to/your/projectdirectory/listing

Note: if you preserve the exact path and filename from download.eclipse.org to archive.eclipse.org, you don't need to change your links (although it is recommended). This works for p2 repos, direct links to https://download.eclipse.org and if your links use the Find a Mirror script.

This link will work if /path/to/a/file.zip is on download.eclipse.org, or if it gets moved to the same place on archive.eclipse.org

  https://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip

P2 repositories: P2 repositories are not normally accessed via the mirror selection script. Therefore, extra treatment is required when the move should be made transparently without affecting users who may still have the original URL.

Equinox/p2/p2.mirrorsURL#Moving_a_repo_to_archive.eclipse.org has a discussion how to achieve this (work in progress).

Use mirror sites/see which mirrors are mirroring my files?

Link to your download files like this:

   Acceptable: https://download.eclipse.org/path/to/a/file.zip    
   Preferred: https://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip

Parameters for above script:

  • file (Required): specify the filename, relative to the downloads home, starting with a "/". This file must exist in the downloads area. Although you can specify a directory name, your mirror list will be more accurate if you specify a file.
  • format (Optional): specify html (default) or xml. Useful for building the mirrors.xml for Update sites.
  • protocol (Optional): ftp or http: list only ftp or http mirrors only (both are the default)
  • r (DEPRECATED): specify 1 to automatically redirect to the best mirror (the one that would normally be at the top) without asking the user to choose.
  • nf (DERECATED): specify 1 to get an actual 404 Not Found error if the file doesn't exist (instead of a lovely page saying so).

The script will examine the Last Modified timestamp of the given file and return only those mirrors that have synchronized with Eclipse.org after that time.

Examples:

   All mirrors of the Lepido project, in XML format:
   https://www.eclipse.org/downloads/download.php?file=/technology/lepido/M1/content.jar&format=xml
   Get a file from a random mirror, without prompting
   https://download.eclipse.org/eclipse/downloads/drops/R-3.1-200506271435/eclipse-SDK-3.1-win32.zip


PLEASE NOTE: We have a list of excluded file patterns -- files that are *not* sent to our mirrors. Nightly and Integration builds are typically very large and don't get many downloads, therefore it's typically more costly (in terms of bandwidth) to mirror them than to support the few client downloads they generate. At time of writing, our exclusion list is:

  • .nfs*
  • apitools/
  • apidocs/
  • archive/
  • archives/
  • /athena
  • builds/N*
  • drops/I*
  • drops/N*
  • drops/M*
  • *.jpg
  • *.gif
  • callisto/*
  • compilelogs/
  • eclipse.org-common/
  • eclipse/testUpdates*
  • eclipse/updates/3.2milestones
  • /eclipse/updates/3.6-I-builds/
  • *eclipse/updates/*-X*
  • *eclipse/updates/*-Y*
  • dev/TPTP*
  • /tools/cdt/builds
  • modeling/gmf/downloads/drops/B*
  • *drops/*/N*
  • *drops/*/I*
  • *javadoc/
  • *javadocs/
  • linuxtools/N*
  • *nightly*
  • *Nightly*
  • *staging*
  • /webtools/downloads/drops/*/M*
  • performance/
  • /releases/staging
  • /releases/europa
  • testresults/
  • /rt/eclipselink/nightly*
  • /technology/babel/update-site*
  • /technology/cosmos
  • /technology/ohf
  • /technology/tigerstripe
  • testcompilelogs/
  • testResults/
  • /tools/downloads
  • /tools/orbit/committers
  • */N202*
  • */I202*
  • */I.I202*
  • */I-*
  • */N-*
  • *integration*/
  • xref/
  • */M20*
  • /rt/eclipselink/maven.repo*

Use the Find a Mirror script?

See the section above.

Enable mirrors / use mirrorsURL for my p2 repo?

Your artifacts.xml (jar) should have a p2.mirrorsURL property. Here is a an example from https://download.eclipse.org/eclipse/updates/3.6/R-3.6.2-201102101200/artifacts.jar

   <repository name='"Eclipse Project Test Site"' type='org.eclipse.equinox.p2.artifact.repository.simpleRepository' version='1'>
     <properties size='4'>
       <property name='p2.compressed' value='true'/>
       <property name='p2.timestamp' value='1297373227427'/>
       <property name='publishPackFilesAsSiblings' value='true'/>
       <property name='p2.mirrorsURL' value='https://www.eclipse.org/downloads/download.php?file=/eclipse/updates/3.6/R-3.6.2-201102101200&format=xml'/>
     </properties>

A more detailed description can be found at Equinox/p2/p2.mirrorsURL.

Ideally, everyone, for all p2 repositories, should use this property, since even if not mirrored currently, it does not hurt anything in that case, and you never know when your repository might become mirrored. In fact, failure to use this property can result in too many requests for jar files coming directly to 'download.eclipse.org' and greatly slow down the network and use too much bandwidth. If this happens for your project (or repository) measures may be taken to automatically redirect all such requests somewhere else, which often does not work well; for examples, see bug 368826.

Include a p2.index file at p2 repository site?

A little documented aide to p2 is to include a special file named "p2.index" at your p2 repository URL site. Every well-behaved, well-optimized p2 repository should have one. This is especially important for composite repository sites as it can save several unsuccessful round trips to download server looking for files that do not exist. For "how to" instructions, see the p2 wiki. For history and deeper technical discussion, see bug 347448.

See download statistics?

The Find a Mirror script tracks download requests once the user has picked a mirror site (or the main Eclipse download site). You can also view download stats for files downloaded via p2 if you enable your p2 repository for download statistics. To view these statistics, use the Live Download Statistics tool (Portal > Project Committer > Tools for all Committers). Download statistics are not available for direct downloads.

For more information, please see the Project Download Stats page.

Sign my Jar/plugins/Windows exe/macOS App files?

The Eclipse Foundation allows committers to sign JAR and some executable files on its behalf. Signing is done from any of the Jenkins servers. There are three ways to sign:

CBI Maven signing plugin

Using the CBI Maven Plugins the signing process can be directly performed at the end of a Maven Tycho build.

Important.png
Maven Profiles
As signing an Eclipse project is only available from an Eclipse Jenkins server, a common practice is to place the CBI Maven signing plugins in a dedicated profile and enable that profile only in the Jenkins job. This way you can still run your Maven Tycho builds locally without signing. See Maven - Introduction to Build Profiles
   <profiles>
     <profile>
       <id>sign</id>
       <build>
         <plugins>
           ...
         </plugins>
       </build>
     </profile>
   </profiles> 
The profile can then be activated in the Jenkins build via the -P argument.


JAR signing

Ensure that all created JAR files are correctly signed by using the eclipse-jarsigner-plugin

   <plugin>
     <groupId>org.eclipse.cbi.maven.plugins</groupId>
     <artifactId>eclipse-jarsigner-plugin</artifactId>
     <version>${cbi-version}</version>
     <executions>
       <execution>
         <id>sign</id>
         <phase>verify</phase>
         <goals>
           <goal>sign</goal>
         </goals>
       </execution>
     </executions>
   </plugin>

Windows signing

To sign the Windows executables use the eclipse-winsigner-plugin

   <plugin>
     <groupId>org.eclipse.cbi.maven.plugins</groupId>
     <artifactId>eclipse-winsigner-plugin</artifactId>
     <version>${cbi-version}</version>
     <executions>
       <execution>
         <id>sign</id>
         <goals>
           <goal>sign</goal>
         </goals>
         <phase>package</phase>
         <configuration>
           <signFiles>
             <signFile>${project.build.directory}/products/${product-folder}/win32/win32/x86_64/eclipse.exe</signFile>
             <signFile>${project.build.directory}/products/${product-folder}/win32/win32/x86_64/eclipsec.exe</signFile>
           </signFiles>
         </configuration>
       </execution>
     </executions>
   </plugin>

macOS signing

To sign the macOS executables use the eclipse-macsigner-plugin

   <plugin>
     <groupId>org.eclipse.cbi.maven.plugins</groupId>
     <artifactId>eclipse-macsigner-plugin</artifactId>
     <version>${cbi-version}</version>
     <executions>
       <execution>
         <id>sign</id>
         <goals>
           <goal>sign</goal>
         </goals>
         <phase>package</phase>
         <configuration>
           <signFiles>
             <signFile>${project.build.directory}/products/${product-folder}/macosx/cocoa/x86_64/Eclipse.app</signFile>
           </signFiles>
           <timeoutMillis>300000</timeoutMillis> 
           <continueOnFail>${macSigner.forceContinue}</continueOnFail>
           <entitlements>${project.basedir}/application.entitlement</entitlements>
         </configuration>
       </execution>
     </executions>
   </plugin>
Important.png
Entitlements
The security guidelines for macOS application development requires the definition of Entitlements to grant an executable permission to use a service or technology. The entitlements used by the Eclipse Platform are defined here


macOS DMG file creation

macOS applications are typically published as .dmg files, which are containers that serve as installers with additional security information to avoid that the application gets tampered. To create a DMG file the eclipse-dmg-packager can be used.

   <plugin>
     <groupId>org.eclipse.cbi.maven.plugins</groupId>
     <artifactId>eclipse-dmg-packager</artifactId>
     <version>${cbi-version}</version>
     <executions>
       <execution>
         <goals>
           <goal>package-dmg</goal>
         </goals>
         <phase>integration-test</phase>
         <configuration>
           <source>${project.build.directory}/products/${product-id}-macosx.cocoa.x86_64.tar.gz</source>
           <continueOnFail>true</continueOnFail>
           <timeoutMillis>600000</timeoutMillis> 
           <continueOnFail>${macSigner.forceContinue}</continueOnFail>
           <sign>true</sign>
         </configuration>
       </execution>
     </executions>
   </plugin>

macOS Notarization

Since macOS Catalina macOS software that is published outside the AppStore needs to be notarized, so the Gatekeeper gets information about trusting the software or not.

As of now the notarization is not available as Tycho plugin. Therefore the macos-notarization-service webservice needs to be used in the Jenkins job similar to the following snippet:

   PRODUCT_ID=...
   BUILD_DIR="${WORKSPACE}/${PRODUCT_ID}/target/products/"
   DMG=${PRODUCT_ID}-macosx.cocoa.x86_64.dmg
   
   pushd $BUILD_DIR
   
   PRIMARY_BUNDLE_ID="app-bundle"
   
   RESPONSE=$(curl -s -X POST -F file=@${DMG} -F 'options={"primaryBundleId": "'${PRIMARY_BUNDLE_ID}'", "staple": true};type=application/json' https://cbi.eclipse.org/macos/xcrun/notarize)
     
   UUID=$(echo $RESPONSE | grep -Po '"uuid"\s*:\s*"\K[^"]+')
   STATUS=$(echo $RESPONSE | grep -Po '"status"\s*:\s*"\K[^"]+')
   
   while [[ ${STATUS} == 'IN_PROGRESS' ]]; do
     sleep 1m
     RESPONSE=$(curl -s https://cbi.eclipse.org/macos/xcrun/${UUID}/status)
     STATUS=$(echo $RESPONSE | grep -Po '"status"\s*:\s*"\K[^"]+')
   done
   
   if [[ ${STATUS} != 'COMPLETE' ]]; then
     echo "Notarization failed: ${RESPONSE}"
     exit 1
   fi
   
   rm "${DMG}"
   
   curl -JO https://cbi.eclipse.org/macos/xcrun/${UUID}/download
   popd

A more detailed script is the Oomph script.

Important.png
CBI Maven Plugins Version
For correct signing needed for notarization (including for example hardened runtime) at least the CBI Plugins version 1.1.8-SNAPSHOT needs to be used. Ensure to configure the correct pluginRepository to be able to consume that version
   <pluginRepositories>
     <pluginRepository>
       <id>cbi</id>
       <url>https://repo.eclipse.org/content/repositories/cbi-releases/</url>
     </pluginRepository>
     <pluginRepository>
       <id>cbi-snapshots</id>
       <url>https://repo.eclipse.org/content/repositories/cbi-snapshots/</url>
       <snapshots>
         <enabled>true</enabled>
       </snapshots>
     </pluginRepository>
   </pluginRepositories>
Once 1.1.8 is released, the SNAPSHOTS repository is not needed anymore.


Important.png
Eclipse Platform Version
The macOS notarization will only succeed if the dmg file signing is matching certain criteria (e.g. hardened runtime). The first Eclipse Platform version that includes the Eclipse Launcher and native libraries that match those criterias are included in 2019-09. For any previous versions the notarization will fail.


For further information on the CBI Maven Plugins have a look at: https://www.eclipse.org/cbi/sitedocs/

Note that these plugins use the web services in the background.

Web service

Using a web POST method, individual JAR files can be signed from any of the internal Jenkins servers with this service:

   https://cbi.eclipse.org/jarsigner/sign

The output of that service will be the signed file. Please note that the web service does not pack or process jar files. You must condition/pack them yourself prior to signing if you wish to do so.

Important.png
Resigning Jarsigner
The web service always resigns already signed jars. The maven jar signer plugin lets you specify a strategy to avoid submitting already signed jar to the webservice. If you use the webservice directly, you need to do deal with it by yourself. You can see how the re-signing strategies are defined by looking at the code of the JarResigner


   # JAR FILES: Submit unsigned-jar.jar and save signed output to signedfile.jar
   curl -o signedfile.jar -F file=@unsigned-jar.jar https://cbi.eclipse.org/jarsigner/sign
   # WINDOWS EXE: Submit Windows unsigned.exe and save signed output to signed.exe
   curl -o signed.exe -F file=@unsigned.exe https://cbi.eclipse.org/authenticode/sign
   # WINDOWS MSI: Submit Windows unsigned.msi and save signed output to signed.msi
   curl -o signed.msi -F file=@unsigned.msi https://cbi.eclipse.org/authenticode/sign
   # MAC: Submit unsigned and save signed output to signed.zip
   # Note: You must zip your entire *.app directory for example: zip -r unsigned.zip Eclipse.app
   curl -o signed.zip -F file=@unsigned.zip https://cbi.eclipse.org/macos/codesign/sign
   # If you need to set entitlements on your app / binary (see https://developer.apple.com/documentation/security/hardened_runtime?preferredLanguage=occ for details), 
   # add an `entitlements` part to the request like below
   curl -o signed.zip -F file=@unsigned.zip -F entitlements=@file.entitlements https://cbi.eclipse.org/macos/codesign/sign

Using the webservice is equally easy from Ant. Note that ${filename} cannot be a path. Input and output file name can be the same.

   <exec dir="${dirname}" executable="curl">
     <arg value="--output"/>
     <arg value="${filename}"/>
     <arg value="--form"/>
     <arg value="file=@${filename}"/>
     <arg value="--silent"/>
     <arg value="--show-error"/>
     <arg value="--fail"/>
     <arg value="https://cbi.eclipse.org/jarsigner/sign"/>
   </exec>
Important.png
Version of Jarsigner
The web service only signs with Java 8 version of jarsigner.


Using the web service to sign Mac and Windows applications is also easy from Tycho, see

What about GPG signing?

JAR signing of the bundles and GPG-signing of the Maven artifacts are two different steps. Once a jar has been "jar-signed", you may or may not GPG sign the corresponding Maven artifact (.jar + .pom file) so as it can be deployed on Central. As you hinted, JAR signing has to be done before the GPG signing, since doing it the other way around would break the GPG signature.

So you first have to sign your JAR file with the Eclipse Fdn certificate, either using the Maven plugin from CBI, the command line utility, or the signing web service – see above. Once you have your signed JAR, you can GPG sign it and stage it on Central like this:

   mvn gpg:sign-and-deploy-file   \
       -DpomFile=target/myapp-1.0.pom  \
       -Dfile=target/myapp-1.0.jar  \
       -Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2/  \
       -DrepositoryId=sonatype_oss

Publish to Maven Central

To deploy to Maven Central from your JIPP, you'll need webmaster's assistance to

  • Create a project specific account at Sonatype OSSRH
  • Generate a GPG keypair for your JIPP user
  • Configure your JIPP to GPG sign and upload artifacts

It takes a bit of time but afterwards, you will only be required to use a dedicated Maven settings on your JIPP.

To get started, please file a bug against https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues asking for your JIPP to be configured to let you publish to Maven central (don't forget the name of your Eclipse project).

If you want to publish jars from already released p2 repositories, consider using the strategy adopted by the Eclipse Platform. More info: Platform-releng/Publish To Maven Central

Builds

Access/request Jenkins services

Please see the Jenkins document.

Code Quality Analysis

Mailing Lists

Setup a new mailing list?

Because Mailing Lists are subject to SPAM and can adversely affect eclipse.org performance (imaging sending 200 e-mails to a list that contains 3000 members), proper care is taken in configuring each list. New mailing lists are set up by the WebMaster for this reason. Also, the webmaster creates an HTML view (called mailing list archives) of mailing list postings for archive and search purposes.

View list members?

Because mailing lists contain private information, such as a member's e-mail address, name and surname, we cannot publicly display this information. However, the PMC or Project Lead can become the list administrator, which would allow you to view the membership information for your lists. The PMC/Project lead can inquire about list administration to the WebMaster, stating which lists they would like to manage.


Eclipse Wiki

Create a new page in the Eclipse Wiki

To create a new page, simply type the page name at the end of "/" in the URL. The name can contain spaces. For instance, https://wiki.eclipse.org/Some_Page will allow you to create and edit this new page.

Eclipse Servers

Eclipse Foundation IT SLA

This page is moderated by the EMO

Back to the top