Spawning processes from build

Sometimes you'd like to spawn a process from a build that lives longer than the build itself. For example, maybe a part of the build is to launch a new application server with the result of the build. When you do this, you often experience a problem where the build doesn't terminate; you'll see that shell script/ant/maven terminates as expected, but Hudson just insists on waiting, as if it didn't notice that the build is over.

Starting Hudson 1.136, Hudson detects this situation and instead of causing infinite block, it will just print out a warning and let you get going. But you should still understand what's causing this.

Why?

The reason this problem happens is because of file descriptor leak and how they are inherited from one process to another. Hudson and the child process are connected by three pipes (stdin/stdout/stderr.) This allows Hudson to capture the output from the child process. Since the child process may write a lot of data to the pipe and quit immediately after that, Hudson needs to make sure that it drained the pipes before it considers the build to be over. Hudson does this by waiting for EOF.

When a process terminates for whatever reasons, the operating system closes all the file descriptors it owned. So even if the process didn't close stdout/stderr, Hudson will nevertheless get EOF.

The complication happens when those file descriptors are inherited to other processes. Let's say the child process forks another process to the background. The background process (AKA daemon) inherits all the file descriptors of the parent, including the writing side of the stdout/stderr pipes that connect the child process and Hudson. If the daemon forgets to close them, Hudson won't get EOF for pipes even when the child process exits, because daemon still have those descriptors open. That's how this problem happens.

A good daemon program closes all file descriptors to avoid problems like this, but often there are bad ones that don't follow the rule.

Work around

On Unix, you can use a wrapper like this to make the daemon behave. On Windows, the 'at' command can be used to launch a process in the background. See the example below:

<scriptdef name="get-next-minute" language="beanshell">
  <attribute name="property" />

  date = new java.text.SimpleDateFormat("HH:mm")
    .format(new Date(System.currentTimeMillis() + 60000));
  project.setProperty(attributes.get("property"), date);
</scriptdef>

<get-next-minute property="next-minute" />
<exec executable="at">
  <arg value="${next-minute}" />
  <arg value="/interactive" />
  <arg value="${jboss.home}\bin\run.bat" />
</exec>

Another similar workaround on Windows is to use a wrapper script and launch your program through it.

<exec executable="cscript.exe">
   <env key="ANTRUN_TITLE" value="Title for Window" />  <!-- optional -->
   <env key="ANTRUN_OUTPUT" value="output.log" />  <!-- optional -->
   <arg value="//NoLogo" />
   <arg value="antRunAsync.js" />  <!-- this script -->
   <arg value="real executable" />
</exec>

Another workaraund for Windows XP and later is to shedule permanent task and force running it from the ant script. Once run the command:

C:\>SCHTASKS /Create /RU SYSTEM /SC ONSTART /TN Tomcat /TR "C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\startup.bat"

Note, that ONSTART can be replaced with ONCE if you do not want to keep Tomcat running. Add the following code to your ant script:

<exec executable="SCHTASKS">
    <arg value="/Run"/>
    <arg value="/TN"/>
    <arg value="Tomcat"/>
</exec>

Another possibility that we can consider is to do something in Hudson.