Jump to: navigation, search

Difference between revisions of "Orion/Running the tests"

(Setting up)
(Running tests)
 
(29 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
== Client tests ==
 
== Client tests ==
=== Running a test ===
+
Orion's client tests are written in JavaScript using the [http://visionmedia.github.io/mocha/ <tt>mocha</tt> test framework]. Assertions are provided by [http://chaijs.com/ <tt>chai</tt>]. The tests run in a web browser. This section explains how Orion developers should write unit tests.
* Tests are located under the <code>js-test</code> folders.
+
* A test is defined by an HTML file, typically named <code>test.html</code>.  
+
* To run a test, load its <code>test.html</code> file in your browser. (Example: <code>http://[your_orion_server]/js-tests/serviceRegistry/test.html</code> )
+
** Test output is printed to the JavaScript console.
+
  
<!--
+
=== Organizing tests ===
=== Writing ===
+
* Every source bundle in the Orion client repo (that is, every child folder of <tt>bundles/</tt>) should have exactly 1 test page. This makes it easy for developers to run all the bundle's tests in one shot.
To create a new test:
+
* Tests should be divided into files organized by functional components. Each component gets a separate file.
* Create a subdirectory of <code>js-tests</code> for your component.
+
** For example, the JavaScript bundle has separate test files dealing with JS content assist, JS validation, JS parsing, etc.
* Create a <code>test.html</code> file. Using RequireJS, your file should load these libraries:
+
* All the test artifacts for a bundle should go under the <tt>{bundle}/web/js-tests/</tt> folder.
** <code>orion/plugin</code>
+
 
** <code>orion/test</code>
+
=== Running tests ===
** Your <code>testcase.js</code> file (see next step).
+
* All the test pages in the product are listed in [https://github.com/eclipse/orion.client/blob/master/releng/org.eclipse.orion.client.releng/test/package.json#L27-L32 this file].
Example test.html:
+
* To run the tests for a bundle, simply load its test page file in a web browser.
  <script src="../../orion/requirejs/require.js"></script>
+
** For example, the test page for the Web Tools bundle is [https://github.com/eclipse/orion.client/blob/master/bundles/org.eclipse.orion.client.webtools/web/js-tests/webtools/webtoolsMochaTests.html <tt>webtoolsMochaTests.html</tt>]. Therefore, to run the tests for the Web Tools bundle, you would navigate to <tt>http://[some_orion_server]/js-tests/webtools/webtoolsMochaTests.html</tt>.
  <script>
+
 
  window.onload = function() {
+
=== Writing tests ===
     require(['orion/plugin', 'orion/test', './testcase'], function(plugin, runner, testcase) {
+
First read [http://visionmedia.github.io/mocha/#getting-started Mocha: Getting Started] for the basics of how tests are structured.
      runner.run(testcase);
+
 
 +
To create a test file for a component, start with this skeleton and add more tests as necessary:
 +
<source lang="javascript" enclose="div" line="1">
 +
// js-tests/foo/myNewTestFile.js
 +
define([
 +
    "chai/chai",
 +
    "mocha/mocha",
 +
], function(chai) {
 +
     var assert = chai.assert;
 +
 
 +
    describe("component X", function() {
 +
        it("should do Y", function() {
 +
            assert.equal(1, 1, "Hopefully 1 == 1");
 +
        });
 +
        it("should do Z", function() {
 +
            // Make more assertions here
 +
        });
 
     });
 
     });
  };
+
});
  </script>
+
</source>
* Create a <code>testcase.js</code> file.
+
When you have a lot of tests, use nested <tt>describe()</tt>s to create a logical hierarchy among them.
** This file is a RequireJS module that exports a TestCase object.
+
 
** Any properties of the TestCase object whose name begins with <code>"test"</code> will be executed.
+
==== Promises in tests ====
** TestCase objects can be nested.
+
If your component performs asynchronous operations using promises, you may find it convenient to return a promise from the <tt>it()</tt> function, and use the promise's resolve() to pass or reject() to fail the test. This offers a cleaner alternative to mocha's <tt>done()</tt> callback when several async steps are involved. If your test rejects, it should provide an instance of Error as the value. Never call <tt>reject(null)</tt> or <tt>reject(undefined)</tt>, or Mocha will count it as a pass!
** A test method may optionally return a [http://dojotoolkit.org/documentation/tutorials/1.6/promises/ promise], which indicates the method will complete asynchronously when the promises resolves.
+
<source lang="javascript" enclose="div" line="1" highlight="4,6,10">
Example testcase.js:
+
// js-tests/foo/myNewTestFile.js
  define(['orion/assert'], function(assert) {
+
define([
     var tests = {};
+
    "chai/chai",
     tests.testOne = function() {
+
    "orion/Deferred",
      assert.strictEqual('a', String.fromCharCode(97), 'a is a');
+
    "mocha/mocha",
     };
+
], function(chai, Deferred) {
    return tests;
+
     it("should do Y asynchronously with a promise", function() {
  });
+
        var deferred = new Deferred();
* At this point, load the <code>test.html</code> file in your browser and you should see the test results, like so:
+
        setTimeout(function() {
[[Image:Test-run.png]]
+
            deferred.reject(new Error("This test should fail"));
* To register a test so it will be run automatically during the Orion build process, add a line to the <code>jsTestSuite.js</code> file, like this:
+
        }, 0);
   OrionTestCase("My awesome test case", "/js-tests/mycomponent/test.html");
+
        return deferred;
-->
+
     });
 +
});
 +
</source>
 +
 
 +
==== Adding to the bundle's test page ====
 +
After creating a new test file, you must add it to the parent bundle's test page. To do this, simply edit the test page and add a dependency in the <tt>require([ .. ])</tt> that goes around the <tt>mocha.run()</tt> call:
 +
 
 +
<source lang="javascript" enclose="div" line="1" highlight="8">
 +
require(['mocha/sauce'], function(mocha) {
 +
    mocha.setup('bdd');
 +
    require([
 +
        'js-tests/javascript/astManagerTests',
 +
        'js-tests/javascript/contentAssistTests',
 +
        'js-tests/javascript/finderTests',
 +
        'js-tests/javascript/occurrencesTests',
 +
        'js-tests/foo/myNewTestFile',
 +
    ], function(){
 +
        mocha.run();
 +
     });
 +
});
 +
</source>
 +
 
 +
Reload the test page in a web browser, and your tests will be executed!
 +
 
 +
==== Writing a new bundle's test page ====
 +
If you find yourself creating an entirely new bundle, then you'll have to create a test page for it. Start with this skeleton, and change as needed (pay close attention to the relative paths, in particular).
 +
 
 +
<source lang="javascript" enclose="div" line="1">
 +
<!DOCTYPE html>
 +
<html>
 +
<head>
 +
    <title>My New Bundle Tests</title>
 +
    <link rel="stylesheet" href="../../mocha/mocha.css" />
 +
    <script src="../../requirejs/require.js"></script>
 +
    <script>
 +
        /*jslint amd:true*/
 +
        require({
 +
            baseUrl: "../../"
 +
        });
 +
        require(["mocha/sauce"], function(mochaSauce) {
 +
            mochaSauce.setup("bdd");
 +
            require([
 +
                /* List this bundle's test files here as dependencies */
 +
            ], function(){
 +
                mochaSauce.run();
 +
            });
 +
        });
 +
    </script>
 +
</head>
 +
<body>
 +
    <h3>My New Bundle Tests</h3>
 +
    <div id="mocha"></div>
 +
</body>
 +
</html>
 +
</source>
 +
 
 +
Note that we use the exports of <tt>mocha/sauce</tt> instead of the regular mocha global: this is necessary for build-time test integration. See [[#Integrating tests with the build|Integrating tests with the build]] for more details.
 +
 
 +
=== Integrating tests with the build ===
 +
To run a bundle's tests in the Orion nightly build, you must perform the following steps:
 +
# '''Important!''' Ensure that the bundle's test page loads the [https://github.com/eclipse/orion.client/blob/master/bundles/org.eclipse.orion.client.core/web/mocha/sauce.js Sauce-enabled version of mocha defined in <tt>sauce.js</tt>]. Do '''not''' use the typical mocha global <tt>window.mocha</tt>.
 +
#* To verify this, load the test page in a browser. You should see some XML looking like <tt>&lt;testcase name="whatever"…</tt> printed to the developer console: this means your test is correctly Sauce-enabled. If you don't see a pile of XML, then your test page is '''wrong''' and will break the build.
 +
# Open the [https://github.com/eclipse/orion.client/blob/master/releng/org.eclipse.orion.client.releng/test/package.json <tt>package.json</tt>] file and add your test page's URL to the <tt>urls</tt> array.
 +
# After the next build, your test results should appear on the [https://hudson.eclipse.org/orion/job/orion-client/lastCompletedBuild/testReport/%28root%29/ orion-client page on Hudson].
 +
 
 +
For details about the build-time unit test architecture, see [[Orion/Releng Builds#JavaScript_tests|Releng Builds]].
 +
 
 +
== Node.js Server Tests ==
 +
 
 +
Our Node server unit tests are written against the [http://visionmedia.github.com/mocha/ Mocha] test framework, and run under Node.js. Developers are expected to run these tests before releasing changes to the server, and before publishing the minified [https://npmjs.org/package/orion orion package] to npm.
 +
 
 +
=== Running the tests ===
 +
 
 +
From the <code>org.eclipse.orion.client/modules/orionode</code> directory, just run the command:
 +
  npm test
 +
 
 +
This will invoke Mocha and produce console output showing which tests passed and failed.
 +
 
 +
If you want to pass custom arguments to Mocha, you'll need to invoke it explicitly like this:
 +
   ./node_modules/mocha/bin/mocha [debug] [options] [files]
 +
 
 +
To make this easier, you can install Mocha as a global npm package (<code>npm install mocha -g</code>), and then invoke it as simply <code>mocha</code> from a command shell.
 +
 
 +
=== Writing more tests ===
 +
When you're prototyping a new feature, writing unit tests for it is always a good idea. Here's how to write a test:
 +
# Create a new file <code>my_tests.js</code> in the <code>org.eclipse.orion.client/modules/orionode/test/</code> directory.
 +
# Write your tests in the file. Here are two resources to help you get started:
 +
#* [http://visionmedia.github.com/mocha/ Mocha reference]: the general structure of a test.
 +
#* [http://visionmedia.github.com/superagent/ Superagent reference]: how to write concise assertions for testing HTTP.
 +
# [[# Running the tests|Run the tests]].
 +
#* You don't have to register your new tests with the framework; it will discover anything in the <code>test/</code> directory automatically.
 +
 
 +
Helper data or classes should go in <code>test/support/</code>.
 +
 
 +
== JVM Server tests ==
  
== Server tests ==
 
 
=== Setting up ===
 
=== Setting up ===
 
# Set up your Eclipse IDE as explained in [[Orion/Getting the source]].
 
# Set up your Eclipse IDE as explained in [[Orion/Getting the source]].

Latest revision as of 14:51, 10 July 2014

Client tests

Orion's client tests are written in JavaScript using the mocha test framework. Assertions are provided by chai. The tests run in a web browser. This section explains how Orion developers should write unit tests.

Organizing tests

  • Every source bundle in the Orion client repo (that is, every child folder of bundles/) should have exactly 1 test page. This makes it easy for developers to run all the bundle's tests in one shot.
  • Tests should be divided into files organized by functional components. Each component gets a separate file.
    • For example, the JavaScript bundle has separate test files dealing with JS content assist, JS validation, JS parsing, etc.
  • All the test artifacts for a bundle should go under the {bundle}/web/js-tests/ folder.

Running tests

  • All the test pages in the product are listed in this file.
  • To run the tests for a bundle, simply load its test page file in a web browser.
    • For example, the test page for the Web Tools bundle is webtoolsMochaTests.html. Therefore, to run the tests for the Web Tools bundle, you would navigate to http://[some_orion_server]/js-tests/webtools/webtoolsMochaTests.html.

Writing tests

First read Mocha: Getting Started for the basics of how tests are structured.

To create a test file for a component, start with this skeleton and add more tests as necessary:

  1. // js-tests/foo/myNewTestFile.js
  2. define([
  3.     "chai/chai",
  4.     "mocha/mocha",
  5. ], function(chai) {
  6.     var assert = chai.assert;
  7.  
  8.     describe("component X", function() {
  9.         it("should do Y", function() {
  10.             assert.equal(1, 1, "Hopefully 1 == 1");
  11.         });
  12.         it("should do Z", function() {
  13.             // Make more assertions here
  14.         });
  15.     });
  16. });

When you have a lot of tests, use nested describe()s to create a logical hierarchy among them.

Promises in tests

If your component performs asynchronous operations using promises, you may find it convenient to return a promise from the it() function, and use the promise's resolve() to pass or reject() to fail the test. This offers a cleaner alternative to mocha's done() callback when several async steps are involved. If your test rejects, it should provide an instance of Error as the value. Never call reject(null) or reject(undefined), or Mocha will count it as a pass!

  1. // js-tests/foo/myNewTestFile.js
  2. define([
  3.     "chai/chai",
  4.     "orion/Deferred",
  5.     "mocha/mocha",
  6. ], function(chai, Deferred) {
  7.     it("should do Y asynchronously with a promise", function() {
  8.         var deferred = new Deferred();
  9.         setTimeout(function() {
  10.             deferred.reject(new Error("This test should fail"));
  11.         }, 0);
  12.         return deferred;
  13.     });
  14. });

Adding to the bundle's test page

After creating a new test file, you must add it to the parent bundle's test page. To do this, simply edit the test page and add a dependency in the require([ .. ]) that goes around the mocha.run() call:

  1. require(['mocha/sauce'], function(mocha) {
  2.     mocha.setup('bdd');
  3.     require([
  4.         'js-tests/javascript/astManagerTests',
  5.         'js-tests/javascript/contentAssistTests',
  6.         'js-tests/javascript/finderTests',
  7.         'js-tests/javascript/occurrencesTests',
  8.         'js-tests/foo/myNewTestFile',
  9.     ], function(){
  10.         mocha.run();
  11.     });
  12. });

Reload the test page in a web browser, and your tests will be executed!

Writing a new bundle's test page

If you find yourself creating an entirely new bundle, then you'll have to create a test page for it. Start with this skeleton, and change as needed (pay close attention to the relative paths, in particular).

  1. <!DOCTYPE html>
  2. <html>
  3. <head>
  4.     <title>My New Bundle Tests</title>
  5.     <link rel="stylesheet" href="../../mocha/mocha.css" />
  6.     <script src="../../requirejs/require.js"></script>
  7.     <script>
  8.         /*jslint amd:true*/
  9.         require({
  10.             baseUrl: "../../"
  11.         });
  12.         require(["mocha/sauce"], function(mochaSauce) {
  13.             mochaSauce.setup("bdd");
  14.             require([
  15.                 /* List this bundle's test files here as dependencies */
  16.             ], function(){
  17.                 mochaSauce.run();
  18.             });
  19.         });
  20.     </script>
  21. </head>
  22. <body>
  23.     <h3>My New Bundle Tests</h3>
  24.     <div id="mocha"></div>
  25. </body>
  26. </html>

Note that we use the exports of mocha/sauce instead of the regular mocha global: this is necessary for build-time test integration. See Integrating tests with the build for more details.

Integrating tests with the build

To run a bundle's tests in the Orion nightly build, you must perform the following steps:

  1. Important! Ensure that the bundle's test page loads the Sauce-enabled version of mocha defined in sauce.js. Do not use the typical mocha global window.mocha.
    • To verify this, load the test page in a browser. You should see some XML looking like <testcase name="whatever"… printed to the developer console: this means your test is correctly Sauce-enabled. If you don't see a pile of XML, then your test page is wrong and will break the build.
  2. Open the package.json file and add your test page's URL to the urls array.
  3. After the next build, your test results should appear on the orion-client page on Hudson.

For details about the build-time unit test architecture, see Releng Builds.

Node.js Server Tests

Our Node server unit tests are written against the Mocha test framework, and run under Node.js. Developers are expected to run these tests before releasing changes to the server, and before publishing the minified orion package to npm.

Running the tests

From the org.eclipse.orion.client/modules/orionode directory, just run the command:

 npm test

This will invoke Mocha and produce console output showing which tests passed and failed.

If you want to pass custom arguments to Mocha, you'll need to invoke it explicitly like this:

 ./node_modules/mocha/bin/mocha [debug] [options] [files]

To make this easier, you can install Mocha as a global npm package (npm install mocha -g), and then invoke it as simply mocha from a command shell.

Writing more tests

When you're prototyping a new feature, writing unit tests for it is always a good idea. Here's how to write a test:

  1. Create a new file my_tests.js in the org.eclipse.orion.client/modules/orionode/test/ directory.
  2. Write your tests in the file. Here are two resources to help you get started:
  3. Run the tests.
    • You don't have to register your new tests with the framework; it will discover anything in the test/ directory automatically.

Helper data or classes should go in test/support/.

JVM Server tests

Setting up

  1. Set up your Eclipse IDE as explained in Orion/Getting the source.
  2. Make sure you have imported the test projects into your workspace, and they're open:
    • org.eclipse.orion.server.tests
    • org.eclipse.orion.server.tests.feature
  3. The test projects have additional dependencies over the rest of the Orion source code. The next 3 steps explain how to satisfy them.
  4. Add the plugins from your Eclipse SDK to your target platform:
    1. Go to Preferences > Target Platform, select your target definition and click Edit.
    2. Click Add... > Installation, then type ${eclipse_home} in the Location field.
    3. Click Finish.
    4. Now there should be 2 locations shown in your target definition. Click Finish.
      Orion-target-result.png
  5. Now we will checkout the remaining test dependencies from CVS. Download this file:
    File:OrionServerTestDepdendencies.psf.
  6. In your Eclipse IDE, go to File -> Import -> Team -> Team Project Set, select OrionServerTestDependencies.psf, and click Finish.
  7. At this point you should have no Java compilation errors. You can now run the tests.

Manually checking out the dependent projects

If the previous section fails for some reason, here's the list of projects from the .psf file. You can check them out manually if need be.

Projects from the Eclipse Platform CVS repository (/cvsroot/eclipse):

  • org.eclipse.core.runtime.compatibility
  • org.eclipse.core.runtime.compatibility.auth
  • org.eclipse.core.tests.harness
  • org.eclipse.core.tests.resources
  • org.eclipse.test.performance
  • org.eclipse.test.performance.data
  • org.eclipse.test.performance.win32

Projects from the Orbit CVS repository (/cvsroot/tools):

  • javax.mail.glassfish (Branch v1_4_1)
  • org.antlr.runtime (Branch v3_2_0)
  • org.hamcrest.core (Branch v1_1)
  • org.junit (Branch v3_8_2)
  • org.junit4 (Branch v4_8_2)

Running

  1. Go to the org.eclipse.orion.server.tests project.
  2. Open the launchConfigurations folder, right-click All Server Tests.launch and choose Run As > All Server Tests.
  3. The JUnit view will open and display the test results.

To run just a subset of the tests, edit the launch configuration (Run > Run Configurations).