Difference between revisions of "CBI/How to check integrity of downloads from the Eclipse Foundation"

Revision as of 12:50, 15 May 2016

How to check the integrity of downloads from the Eclipse Foundation

There are several methods to verify the integrity of downloads from Eclipse.org. They all involve getting a "checksum" and then verifying that checksum matches the checksum computed from the artifact that was downloaded.

This page focuses on downloading artifacts obtained from the Eclipse Foundation's "select a mirror page". Most general users would want to use that for releases or milestones.

This common method to verify integrity -- which applies to most downloads available from "eclipse.org" -- is to obtain the checksum from the "Select a Mirror" page. Typically there are links (currently near the left of the page) that are labeled "MD5", "SHA1", and SHA512. If verifying "visually", we recommend SHA1, since it is short enough to be feasible to verify "by hand", and still a fair algorithm for confirming download integrity. Note: an important aspect of this type of verification is that you get the checksum from an "https" connection (not "http") directly from an eclipse.org URL, as that ensures it really is coming from "eclipse.org" and not someone "in the middle".

We recommend verifying the checksums programmatically and using SHA512 checksums as that is a more advanced algorithm that better avoids collisions and tampering. Collisions or tampering would be extremely unlikely, but, if you want to be positive, use the longer checksums. If you just want to be sure your network didn't hiccup in the middle of a download and lose a few bytes SHA1 would be adequate. (We do not recommend MD5. It is being provided for compatibility for those that might still depend on it, but the trend is to move away from MD5.)

Example

If you want to download and verify the Windows (64 bit) zip file from the Eclipse Project's site, at

http://download.eclipse.org/eclipse/downloads/drops4/R-4.5.2-201602121500/

then when you click on "eclipse-SDK-4.5.2-win32-x86_64.zip" you are taken to the "select a mirror" page.

After selecting a mirror, you would end up with a downloaded file named

eclipse-SDK-4.5.2-win32-x86_64.zip

But first, while on the "select a mirror" page, if you click on the SHA1 link (or "button") you will see a "popup" that says

72867f34bfdacd8f4fd1de52d749ae8f96951ef1 eclipse-SDK-4.5.2-win32-x86_64.zip

If you click on the SHA512 link the pop up would say

894886f68ae4838c0c0ab8a80ae95dbc4da06dc6f4b9e4cfa6db7be52c1f8cce5b5c39a00871d0a32deb1e70718cffbdb56c647afa41fde95a40d74591be7308 eclipse-SDK-4.5.2-win32-x86_64.zip

It is best to copy and paste (one of) those checksums to a text file and save to the location as where the artifact will be downloaded and, for this example name it say to 'eclipse-SDK-4.5.2-win32-x86_64.zip.sha512'.

Once you have saved the checksum and downloaded the artifact:

1. If verifying "visually", by hand, simply invoke the following command and ensure the results obtained are the same as you see from the "pop-up" on the https connection to the "select a mirror" page.

 sha1sum eclipse-SDK-4.5.2-win32-x86_64.zip
 
 returns 
 
 72867f34bfdacd8f4fd1de52d749ae8f96951ef1  eclipse-SDK-4.5.2-win32-x86_64.zip

2. To verify programmatically with the SHA515 checksum.

If you saved the copy and pasted the checksum value to a file named 'eclipse-SDK-4.5.2-win32-x86_64.zip.sha512' and saved to the same directory as the artifact 'eclipse-SDK-4.5.2-win32-x86_64.zip' then

sha512sum --check eclipse-SDK-4.5.2-win32-x86_64.zip.sha512

returns

eclipse-SDK-4.5.2-win32-x86_64.zip: OK

Note: when copy and pasting and saving the checksum value, there must be no extra blank lines in the file, and you may need to add an extra space or the '*' (asterisk) character inserted before the filename. (See bug 493439)

Variations for operating systems

There are some variations on the above, depending on the operating system you use.

The Linux tools are part of the GNU Coreutils package and are installed automatically on most distributions of Linux.

On Windows, you may need to download and install equivalent utilities such as those from Cygwin or other packages that can be found by searching the internet. Note that a tool that is suitable may depend on which version of Windows you are using. For example, it has been said that Power Shell, in Windows 10, has Get-FileHash.

On MaxOSX, the utilities are typically installed but the name and arguments may differ. For example,

 shasum -a 512 -c eclipse-SDK-4.5.2-win32-x86_64.zip.sha512

For a cross-platform solution, there is always Ant. To help you get started with an Ant scripts, see the Ant's checksum task, or, an even better example, is their "get-and-checksum" macro example (which would need some modification to suit your needs).

Or for some other cross-platform solutions, institutions or power users may want to use Java methods directly to compute checksums or improve automation. It has also been said the GNU Utilities can be built to work on a number of platforms.

Improving automation

Automation can be improved either with command script files or tools such as Ant -- Ant also can be used to make a "cross-platform" method of achieving these tasks. The key to the automation is knowing the URLs to use to "fetch the artifact" (from a mirror) and "fetch the checksum" (directly from eclipse.org) without having to click on a web page.

To download an artifact from a system-selected mirror add '&r=1' to the end of the normal mirror URL. Continuing the above example, one method to get 'eclipse-SDK-4.5.2-win32-x86_64.zip' is to use 'wget' with options and a URL similar to the following:

  wget -O eclipse-SDK-4.5.2-win32-x86_64.zip "http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops4/R-4.5.2-201602121500/eclipse-SDK-4.5.2-win32-x86_64.zip&r=1"
  

Then to get the checksum, use the 'sums.php' file, instead of the 'download.php' file in the URL (and change the file it is saved in, of course). For example,

  wget -O eclipse-SDK-4.5.2-win32-x86_64.zip.sha512 "https://www.eclipse.org/downloads/sums.php?file=/eclipse/downloads/drops4/R-4.5.2-201602121500/eclipse-SDK-4.5.2-win32-x86_64.zip"
  

By default, 'sums.php' returns the SHA512 checksum. If you want SHA1 instead of SHA512 then you can add "&type=sha1" to the URL, such as

  wget -O eclipse-SDK-4.5.2-win32-x86_64.zip.sha1 "https://www.eclipse.org/downloads/sums.php?file=/eclipse/downloads/drops4/R-4.5.2-201602121500/eclipse-SDK-4.5.2-win32-x86_64.zip&type=sha1"
  

Note that for artifact retrieval "http://" is used (to avoid wasting computing power on encrypting large files) but for checksum retrieval "https://" is used (to help ensure you are accessing the site that you think you are).

In the above example of getting SHA1, the parameter "&type=sha1" must be lowercase and there are no error message for invalid values -- SHA512 will be returned if the type is not recognized as a supported type. (See bug 493456).

Occasionally, as with any internet work, the request for the checksum may fail. Typically some "loop" is required to "retry" for few seconds to help make sure it is be obtained.

Also, note, the above "sums.php" method only works for files that are mirrored by the Eclipse Foundation. For example, the N-builds and even I-builds from the Eclipse Platform Project are not typically mirrored (see bug 489521. But, for the general user getting milestones or releases the methods described here should work well.

Committers should read the Downloads section of the IT Infrastructure Documentation to understand how to make use of the mirroring system at Eclipse.org.