Making your plugin behave in distributed Hudson

If you are just getting started with writing a Hudson plugin, then this is not something you need to worry about right now. Come back to this page when you have something working.

Distribution Architecture of Hudson

Hudson uses a mechanism simliar to distributed agents to perform distributed computing. That is, the thread that's running on the master can send closures to remote machines, then get the result back when that closure finishes computation.

For example, the following code, taken from hudson.main.ProcessCache, shows one such closure implementation.

    private static class GetSystemProperties implements Callable<Properties,RuntimeException> {
        public Properties call() {
            return System.getProperties();
        }
        private static final long serialVersionUID = 1L;
    }

Closures implements Callable, which is parameterized on both the return type and the exception type that it can throw.

You can dispatch this closure to a slave by calling Channel.call like this:

Properties systemProperties = channel.call(new GetSystemProperties());

Java serialization is used to send the closure and the return value back and forth. That is, the following things happen when the above statement is executed:

1. (local) the closure is serialized (and thus everything that the closure references gets serialized)
2. (local) serialized byte image will be sent to the remote JVM
3. (remote) the closure gets deserialized.
4. (remote) the closure is executed
5. (remote) the return value (upon a normal completion) or the exception (upon an abnormal completion) are serialized
6. (remote) serialized byte image will be sent back to the local JVM
7. (local) return value or exception is deserialized, and the Channel.call returns.

Behind the scene, the remoting framework takes care of class file transmissions, exception chaining, and other low-level stuff.

More on Closures

More commonly, closures are written as anonymous classes. The above example can be written more concisely as:

Properties systemProperties = channel.call(new Callable<Properties,RuntimeException>(){
    public Properties call() {
        return System.getProperties();
    }
});

This syntax is particularly beneficial when a closure needs to bring a lot of parameters to the remote JVM. The downside is that you may end up bringing more than you needed, in particular the implicit this object of the enclosing class.

Implicit Remoting

Since distributed computing is a complex topic, Hudson has several key abstractions in place to make this aspect of Hudson somewhat transparent.

FilePath

For simple plugins, doing remoting at the file access layer is the easiest and the most transparent way to achieve distribution-safe code. For this reason, Hudson introduces the FilePath class to perform file access (instead of java.io.File.) Unlike File, FilePath can point to any file/directory in the master or any of the slaves, and methods defined on this class can work correctly when files that it refers to is on a remote machine.

Launcher

Launcher is another abstraction for an implicit distribution. This class plays a similar role to java.lang.ProcessBuilder, except that it can launch a process on a remote JVM.

Performance consideration

Because of the pervasive use of FilePath and Launcher throughout in Hudson core, sometimes plugin developers don't even notice that their code behave correctly in the distributed environment.

However, one should note that these simple approach may have performance penalty — often you can achieve better performance by moving the code to where the data is (which is what a closure gives you), instead of moving the data to where the code is (which is what the implicit data remoting in FilePath gives you.)

So a well-written Hudson plugin should use explicit closures to perform a block of task remotely, and only send the summary back to the master, instead of moving the large data over the network.

A good example of this can be seen in the JUnitResultArchiver class, which performs XML parsing on the remote machine and just sends back the resulting objects to the master.

TBD

• Asynchronous calls
• FilePath.act
• Pipes
• Remotable objects, like TaskListener, FilePath.
• Exporting a proxy
• your suggestions welcome