Writing unit tests for your plugin

Overview

Hudson comes with a test harness built around JUnit to make test development simpler. This harness provides the following features:

1. Starts an embedded servlet container so that the test code can exercise user interaction through HTTP and assert based on the outcome.
2. HtmlUnit with a bit of enhancements allows you to productively test HTTP/UI interaction.
3. Prepares and tears down a fresh Hudson instance for each test case. So each test method will run in a fresh environment, isolated from other tests.
4. Test code can also directly access Hudson object model. This allows tests to assert directly against the internal state of Hudson, as well as perform some operations directly, as opposed to doing so only through the HTML scraping.
5. Declarative annotations to specify the environment in which a test will be run. For example, your test method can request that Hudson shall be started with a certain HUDSON_HOME contents.
6. Declarative annotations to maintain association between tests and bugs/discussions.

Example

The following code shows a very simple test case. Your test will extend from HudsonTestCase instead of standard TestCase. HudsonTestCase defines additional code that provides some of those features outlined above.

Each test method will start with a fresh temporary installation of Hudson. The test1 method doesn't request any particular Hudson dataset to seed Hudson, so it will start from an empty installation.

The test then proceed to create a new project and set it up. As you can see, the code directly talks to the in-memory Hudson object model (There is the protected 'hudson' variable defined in HudsonTestCase that you can use for access.) While you can do the same by emulating the user HTTP interaction through HtmlUnit, this way is often a convenient way to prepare an environment for the code that you want to test.

The test code in this example then switch to HtmlUnit to emulate the UI interaction and verify that we get results that we expect.

When a test completes, the temporary Hudson installation will be destroyed.

import org.jvnet.hudson.test.HudsonTestCase;
import org.apache.commons.io.FileUtils;
import hudson.model.*;
import hudson.tasks.Shell;

public class AppTest extends HudsonTestCase
{
    public void test1() throws Exception {
        FreeStyleProject project = createFreeStyleProject();
        project.getBuildersList().add(new Shell("echo hello"));

        FreeStyleBuild build = project.scheduleBuild2(0).get();
        System.out.println(build.getDisplayName()+" completed");

        // TODO: change this to use HtmlUnit
        String s = FileUtils.readFileToString(build.getLogFile());
        assertTrue(s.contains("+ echo hello"));
    }
}

How to set environment variables

When preparing you virtual test environment, you may wish to simulate Hudson environment variables that can be set on the Hudson configuration page. Adding environment variables to a Hudson instance before a test is simple, as the example below demonstrates.

public void setEnvironmentVariables() throws IOException {
EnvironmentVariablesNodeProperty prop = new EnvironmentVariablesNodeProperty();
EnvVars envVars = prop.getEnvVars();
envVars.put("sampleEnvVarKey", "sampleEnvVarValue");
super.hudson.getGlobalNodeProperties().add(prop);
}

Debugging

Your IDE would most likely have an ability of selecting a single JUnit test and execute it in the debugger. Otherwise you can run mvn -Dmaven.surefire.debug -Dtest=hudson.SomeTest test to run a test with the debugger.

To debug slaves launched by Hudson, set -Dorg.jvnet.hudson.test.HudsonTestCase.slaveDebugPort=PORT to the system property, or from your test case set HudsonTestCase.SLAVE_DEBUG_PORT to a non-0 value.

Various Test Techniques

Mocking

Sometimes you want to have quick tests which don't starts up a 'full' Hudson instance as HudsonTestCase does as this can take some time. In that case you shouldn't have your test classes extend HudsonTestCase.

As creating most Hudson core classes without a Hudson instance is unfortunately not easy;mocking can come in handy. One excellent mocking framework is e.g. Mockito. For example, if you want to mock a build with a certain result you could do:

AbstractBuild build = Mockito.mock(AbstractBuild.class);
Mockito.when(build.getResult()).thenReturn(Result.FAILURE);

HTML scraping

If you'd like to test the HTML generated by Hudson, XPath test is often convenient.

HtmlPage page = wc.goTo("computer/test/");
HtmlElement navbar = root.getElementById("left-top-nav");
assertEquals(1,navbar.selectNodes(".//a").size());

Submitting forms

Submitting a form through HtmlUnit in Hudson is a bit tricker than it should be. This is because Structured Form Submission is done in JavaScript at the form submission time, and the submit button is usually decorated by YUI, which internally converts <input type="submit"> to <button>.

So you'd have to look for the corresponding HtmlButton element, then use that to call the submit method, like this:

HtmlPage configPage = new WebClient().goTo("configure");
HtmlForm form = configPage.getFormByName("config");
form.submit((HtmlButton)last(form.getHtmlElementsByTagName("button")));

Dealing with problems in JavaScript

When JavaScript throws an exception and causes a test to fail, it will often print a long chain of nested exceptions. Notice that the stack trace includes synthesized stack frames for JavaScript, which is different from the actual Java execution stack.

The original HtmlUnit doesn't really do a good job of chaining all exceptions together, so we are patching HtmlUnit to make sure it retains the full stack trace leading up to the root cause. If you found a case where this chain is broken, please file a bug.

If you set a break point in Java code, and if your execution suspends while its directly/indirectly invoked through JavaScript, you can use HudsonTestCase.jsDebugger to introspect JavaScript call stack and its local variables. This is often very useful in identifying where in JavaScript things went wrong.

Configuration round-trip testing

One of the very useful test idioms for Builder, Publisher, and anything that has configuration forms is the round-trip testing. This test goes like this:

1. Programmatically construct a fully populated instance
2. Request a configuration page via HtmlUnit
3. Submit the config page without making any changes
4. Verify that you still have the identically configured instance

FreeStyleProject p = createFreeStyleProject();
YourBuilder before = new YourBuilder("a","b",true,100);
p.getBuildersList().add(before);

submit(createWebClient().getPage(p,"configure").getFormByName("config"));

YourBuilder after = p.getBuildersList().get(YourBuilder.class);

assertEqualBeans(before,after,"prop1,prop2,prop3,...");

This test ensures that your configuration page is properly pre-populated with the current setting of your model object, and it also makes sure that the submitted values are correctly reflected on the constructed model object. To be really sure, do this twice with different actual values — for example, you should try a non-null string and null string, true and false, etc., to exhaust representative cases.

Web page assertions

HtmlUnit has a WebAssert class that can be used for simple assertions on HTML pages.

To assert that the System configuration page contains the CVS SCM configuration entry:

HtmlPage page = new WebClient().goTo("configure");
WebAssert.assertElementPresent(page, "hudson-scm-CVSSCM");

Doing things differently in JavaScript when it runs as unit test

JavaScript in Hudson can test whether it's running in the unit test or not by checking the global isRunAsTest variable defined in hudson-behavior.js, which is included in all the pages. This can be used to disable some ajax operations, for example. Obviously, this has to be used with caution so that tests will continue to test the real thing as much as possible.

TestCase as an RootAction

Instance of the test case being executed is added to Hudson's URL space as /self because HudsonTestCase is itself a RootAction. Among other things, this enbles your test class to define Jelly views, and invoke it like createWebClient().goTo("self/myview").

Custom builder

You can extend TestBuilder to write a one-off builder that can coordinate with your test. This is often convenient to stage things up for testing your Publisher, for example by placing files in the workspace, etc.

FreeStyleProject project = createFreeStyleProject();
project.getBuildersList().add(new TestBuilder() {
    public boolean perform(AbstractBuild<?, ?> build, Launcher launcher,
        BuildListener listener) throws InterruptedException, IOException {
        build.getWorkspace().child("abc.txt").write("hello","UTF-8");
        return true;
    }
});

project.scheduleBuild2(0);
buildStarted.block(); // wait for the build to really start

OneShotEvent is also often an useful companion so that the thread that runs your test method and the thread that runs the build can coordinate — for example, the following program have the main thread block until a build starts.

final OneShotEvent buildStarted = new OneShotEvent();

FreeStyleProject project = createFreeStyleProject();
project.getBuildersList().add(new TestBuilder() {
    public boolean perform(AbstractBuild<?, ?> build, Launcher launcher,
        BuildListener listener) throws InterruptedException, IOException {
        buildStarted.signal();
        ...
        return true;
    }
});

project.scheduleBuild2(0);
buildStarted.block(); // wait for the build to really start

Registering Extensions during tests

During the test, one might want to register extensions just during that particular test, for example to assist the test scenario. You can do this by defining such extension as a nested type of your test case class and put TestExtension instead of Extension.

It lets you tie an extension to just one test method, or all test methods on the same class.

Test harness annotations

There are several annotations in the Hudson test framework.

Informational annotations

@Bug(1234)

Issue tracker number for the test.

@For(FooBar.class)

Production classes that tests are related to. Useful when the relationship between the test class name and the test target class is not obvious.

@Url(http://internet.org)

URL to the web page indicating a problem related to this test case.

@Email(http://....)

URL to the e-mail archive. Look for the e-mail in http://www.nabble.com/Hudson-users-f16872.html or http://www.nabble.com/Hudson-dev-f25543.html Test environment annotations

@Recipe(SetupClass)

The specified class will be used to set up the test environment using HudsonTestCase.

@LocalData

Runs a test case with a data set local to test method or the test class.

This recipe allows your test case to start with the preset HUDSON_HOME data loaded either from your test method or from the test class. For example, if the test method if org.acme.FooTest.testBar(), then you can have your test data in one of the following places in resources folder (typically src/test/resources):

• Under org/acme/FooTest/testBar directory (that is, you'll have org/acme/FooTest/testBar/config.xml), in the same layout as in the real HUDSON_HOME directory.
• In org/acme/FooTest/testBar.zip as a zip file.
• Under org/acme/FooTest directory (that is, you'll have org/acme/FooTest/config.xml), in the same layout as in the real HUDSON_HOME directory.
• In org/acme/FooTest.zip as a zip file.

Search is performed in this specific order. The fall back mechanism allows you to write one test class that interacts with different aspects of the same data set, by associating the dataset with a test class, or have a data set local to a specific test method.

The choice of zip and directory depends on the nature of the test data, as well as the size of it.

@PresetData(SecurityPreset)

Runs a test case with one of the preset HUDSON_HOME data set:

   NO_ANONYMOUS_READACCESS - Secured Hudson that has no anonymous read access. Any logged in user can do anything.
   ANONYMOUS_READONLY - Secured Hudson where anonymous user is read-only, and any logged in user has a full access.

@WithPlugin(NameOfPlugin)

Installs the specified plugin before launching Hudson in the test. For now, this has to be one of the plugins statically available in resources "/plugins/NAME".

TODO

• Host test harness javadoc and link from this article.
• define HtmlPage subclass to define more convenience methods.