Jump to: navigation, search

Difference between revisions of "Hudson-ci/features/Restart Within Hudson"

m (Restart Option)
(Example)
 
(37 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
Configuration changes that require Hudson restart to take effect should provide a <code>Restart</code> link or button.
 
Configuration changes that require Hudson restart to take effect should provide a <code>Restart</code> link or button.
  
Unfortunately, the CLI restart or soft-restart methods, which call <code>Hidson.restart</code> and <code>Hudson.safeRestart</code>, respectively, don't handle many of the necessary use cases by default. A Hudson restart link based on these methods would too often fail. It is one thing to leave an opening for plugins to implement, but quite another to depend on the kindness of plugins for basic Hudson operations. The default Lifecycle implementation needs to be in some sense universal, even if it requires the cooperation of system administrators.
+
Unfortunately, the CLI restart or soft-restart methods, which call <code>Hudson.restart</code> and <code>Hudson.safeRestart</code>, respectively, don't handle many of the necessary use cases by default. A Hudson restart link based on these methods would too often fail. It is one thing to leave an opening for plugins to implement, but quite another to depend on the kindness of plugins for basic Hudson operations. The default Lifecycle implementation needs to be in some sense universal, even if it requires the cooperation of system administrators.
  
 
==Requirements==
 
==Requirements==
Line 15: Line 15:
 
==Design Approach==
 
==Design Approach==
  
Two design approaches are being actively pursued.  
+
There are two ways to provide a correct restart. The first is to add a plugin that extends <code>hudson.lifecycle.Lifecycle</code> and add the <code>hudson.lifecycle</code> Java property to the command line that launches Hudson. This option was also available in previous releases. The second is by using a new feature, Restart Command, that allows restart with OS scripts. We illustrate it with the simplest possible Lifecycle and supporting Hudson launch script.
  
The first, Restart Command, allows a system administrator to specify a command or provide a script during Hudson startup. This is most suited to Hudson running in containers like Tomcat, etc. which provide easy application restart from the command line. The second, Soft Restart, is more general and more likely to cause problems; it involves re-invoking a subset of the startup sequence to create new instances of Hudson classes, plugins, etc. rooted in a different classloader.
+
The Restart Command allows a system administrator to specify a command or provide a script during Hudson startup.
  
===Restart Command===
+
===Just Exit Lifecycle===
 +
 
 +
The Just Exit Lifecycle Plugin implements a Lifecycle whose restart method just calls <code>System.exit(0)</code>. About as trivial a plugin as one can imagine, but it is enough to enable the Restart button in the plugin manager. (It is may not be a good idea to use this when running Hudson in a server with multiple applications in the same JVM, but there aren't many other good ideas for restart in this environment, suggesting the environment, itself, may not be a good idea.)
 +
 
 +
====Supporting Shell Script====
 +
 
 +
Obviously, Just Exit only does the stop part of restart. To do the start part, we recommend always launching Hudson with a script that will automatically restart it when it exits, for example, when it is stopped by the <code>kill</code> command. For example,
 +
<pre>
 +
#!/bin/bash
 +
export HUDSON_WAR=<path-to-Hudson-WAR>
 +
export HUDSON_HOME=<path-to-Hudson-home>
 +
if [ -e "$HUDSON_HOME/plugins/just-exit-lifecycle.hpi" ]
 +
then
 +
  export LIFECYCLE="-Dhudson.lifecycle=org.hudsonci.plugins.justexit.JustExitLifecycle"
 +
fi
 +
 
 +
while :
 +
do
 +
 
 +
    if [ "x$1" == "x-d" ]
 +
    then
 +
        java $LIFECYCLE -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000 -jar $HUDSON_WAR &
 +
    else
 +
        java $LIFECYCLE -jar $HUDSON_WAR &
 +
    fi
 +
 
 +
    trap "break" SIGINT SIGTERM
 +
    # receiving trapped signal makes wait return
 +
    # with exit status >= 128, then trap executes
 +
    wait ${!}
 +
    sleep 1
 +
    echo "Restarting Hudson"
 +
 
 +
done
 +
echo "Killing child process"
 +
pkill -P "$$"
 +
exit
 +
</pre>
 +
When the script is run, it executes Hudson in a loop. Thus, it implements the 'start' part of restart. Hudson will restart if it exits or is killed. (To actually kill Hudson, kill the script that launched it.)
 +
 
 +
This technique is simple and can be adapted to a failsafe environment by ensuring the alternate server doesn't start up during the restart.
 +
 
 +
==Restart Command==
  
 
Correct restart is a multi-dimensional problem, different for each OS, service implementation and container (Tomcat, Jetty, GlassFish, etc.). The Lifecycle extension point is not well suited for multi-dimensional invocation, e.g., the OS is X AND the container is Y and (the container does not support single application restart OR the application name/war file location in the container is Z). While certain tricks, like replacing the Hudson WAR file, work to restart the application in many containers, to cover all the possibilites, Lifecycle would need an extension for every possible combination.
 
Correct restart is a multi-dimensional problem, different for each OS, service implementation and container (Tomcat, Jetty, GlassFish, etc.). The Lifecycle extension point is not well suited for multi-dimensional invocation, e.g., the OS is X AND the container is Y and (the container does not support single application restart OR the application name/war file location in the container is Z). While certain tricks, like replacing the Hudson WAR file, work to restart the application in many containers, to cover all the possibilites, Lifecycle would need an extension for every possible combination.
Line 27: Line 69:
 
Two new ways are provided for the system administrator to configure correct restart:
 
Two new ways are provided for the system administrator to configure correct restart:
  
* by providing a <code>hudson-restart</code> file in the $HUDSON_HOME provisioned for the Hudson instance and
+
* by providing a <code>hudson-restart</code> file in the $HUDSON_HOME provisioned for the Hudson instance or
 
* by specifying a restart command on the command line when Hudson is invoked.
 
* by specifying a restart command on the command line when Hudson is invoked.
  
Line 50: Line 92:
 
The script should be specific to the environment in which the HUDSON_HOME is used.
 
The script should be specific to the environment in which the HUDSON_HOME is used.
  
===Restart Option===
+
===Restart System Property===
  
Even if a restart script is present in HUDSON_HOME, a Hudson admin may change the restart command to do something else, effectively ignoring the script. This may be overridden by the command line switch:
+
Even if a restart script is present in HUDSON_HOME, a system administrator may change the restart command to do something else, effectively ignoring the script, using the command line option:
  
  --restartCommand=value
+
  -Dhudson.restart=value
  
The value of the <code>restartCommand</code> option will be used to initialize the restart command. The correct order of initialization is:
+
The value of the <code>hudson.restart</code> system property will be used to initialize the restart command. The correct order of initialization is:
  
*If <code>--restartCommand</code> is specified, use that.
+
*If <code>-Dhudson.restart</code> is specified, use that.
 
*Otherwise, if a restart script is present, use a command that invokes the script.
 
*Otherwise, if a restart script is present, use a command that invokes the script.
 
*Otherwise, the restart command is not specified.
 
*Otherwise, the restart command is not specified.
 +
 +
===Implementation===
 +
 +
A new lifecycle, <code>hudson.lifecycle.RestartCommandLifecycle</code>, will be added to Hudson.
 +
 +
<code>hudson.lifecycle.Lifecycle</code> will be modified to use the <code>RestartCommandLifecycle</code> for restart if:
 +
 +
* a restart script or command has been specified, and
 +
* the <code>hudson.lifecycle</code> system property is not defined.
 +
 +
The implementation will log a warning message if <code>hudson.lifecycle</code> has been specified and a restart script or command have been provided.
 +
 +
  
 
==Restart Links or Buttons==
 
==Restart Links or Buttons==
Line 66: Line 121:
 
Restart links should not be shown unless <code>Lifecycle.get().canRestart()</code> returns <code>true</code>. Otherwise, a message like "Restart required for changes to take effect" should be displayed.
 
Restart links should not be shown unless <code>Lifecycle.get().canRestart()</code> returns <code>true</code>. Otherwise, a message like "Restart required for changes to take effect" should be displayed.
  
A restart link should show the single word <b>Restart</b> and should call <code>Hudson.softRestart</code>.
+
A restart link should show the single word <b>Restart</b> and should call <code>Hudson.safeRestart</code>.
  
 
===Plugin Manager===
 
===Plugin Manager===
  
 
The Plugin Manager will show a restart link if a) a plugin is updated or a plugin is loaded that requires restart, and b) if <code>Lifecycle.get().canRestart()</code> returns <code>true</code>.
 
The Plugin Manager will show a restart link if a) a plugin is updated or a plugin is loaded that requires restart, and b) if <code>Lifecycle.get().canRestart()</code> returns <code>true</code>.

Latest revision as of 22:19, 20 August 2013

Restart Within Hudson

Configuration changes that require Hudson restart to take effect should provide a Restart link or button.

Unfortunately, the CLI restart or soft-restart methods, which call Hudson.restart and Hudson.safeRestart, respectively, don't handle many of the necessary use cases by default. A Hudson restart link based on these methods would too often fail. It is one thing to leave an opening for plugins to implement, but quite another to depend on the kindness of plugins for basic Hudson operations. The default Lifecycle implementation needs to be in some sense universal, even if it requires the cooperation of system administrators.

Requirements

In the following, "correct restart" means restart specifically tailored to the environment in which the Hudson instance is running. "System administrator" is a person who provisions and deploys the Hudson instance. "Hudson admin" is a person with admin privileges in Hudson. Sometimes these are the same people; sometimes not.

  • The current Lifecycle API must be preserved, for compatibility with existing plugin extensions. The new default mechanism will only be invoked if the hudson.lifecycle system property is not specified.
  • It must be possible for a system administrator to preconfigure Hudson for correct restart. This is particularly important for "no-admin" uses of Hudson.
  • Restart must require Hudson admin privileges (ACL.SYSTEM).

Design Approach

There are two ways to provide a correct restart. The first is to add a plugin that extends hudson.lifecycle.Lifecycle and add the hudson.lifecycle Java property to the command line that launches Hudson. This option was also available in previous releases. The second is by using a new feature, Restart Command, that allows restart with OS scripts. We illustrate it with the simplest possible Lifecycle and supporting Hudson launch script.

The Restart Command allows a system administrator to specify a command or provide a script during Hudson startup.

Just Exit Lifecycle

The Just Exit Lifecycle Plugin implements a Lifecycle whose restart method just calls System.exit(0). About as trivial a plugin as one can imagine, but it is enough to enable the Restart button in the plugin manager. (It is may not be a good idea to use this when running Hudson in a server with multiple applications in the same JVM, but there aren't many other good ideas for restart in this environment, suggesting the environment, itself, may not be a good idea.)

Supporting Shell Script

Obviously, Just Exit only does the stop part of restart. To do the start part, we recommend always launching Hudson with a script that will automatically restart it when it exits, for example, when it is stopped by the kill command. For example,

#!/bin/bash
export HUDSON_WAR=<path-to-Hudson-WAR>
export HUDSON_HOME=<path-to-Hudson-home>
if [ -e "$HUDSON_HOME/plugins/just-exit-lifecycle.hpi" ]
then
  export LIFECYCLE="-Dhudson.lifecycle=org.hudsonci.plugins.justexit.JustExitLifecycle"
fi

while :
do

    if [ "x$1" == "x-d" ]
    then
        java $LIFECYCLE -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000 -jar $HUDSON_WAR &
    else
        java $LIFECYCLE -jar $HUDSON_WAR &
    fi

    trap "break" SIGINT SIGTERM
    # receiving trapped signal makes wait return
    # with exit status >= 128, then trap executes
    wait ${!}
    sleep 1
    echo "Restarting Hudson"

done
echo "Killing child process"
pkill -P "$$"
exit

When the script is run, it executes Hudson in a loop. Thus, it implements the 'start' part of restart. Hudson will restart if it exits or is killed. (To actually kill Hudson, kill the script that launched it.)

This technique is simple and can be adapted to a failsafe environment by ensuring the alternate server doesn't start up during the restart.

Restart Command

Correct restart is a multi-dimensional problem, different for each OS, service implementation and container (Tomcat, Jetty, GlassFish, etc.). The Lifecycle extension point is not well suited for multi-dimensional invocation, e.g., the OS is X AND the container is Y and (the container does not support single application restart OR the application name/war file location in the container is Z). While certain tricks, like replacing the Hudson WAR file, work to restart the application in many containers, to cover all the possibilites, Lifecycle would need an extension for every possible combination.

Yet, every system administrator already knows or can easily discover a command or script to restart any running Hudson instance. The best and most likely to be correct restart mechanism would leverage those commands/scripts and not try to replace it with Java code. The feature described below provides a generic Lifecycle that does.

Two new ways are provided for the system administrator to configure correct restart:

  • by providing a hudson-restart file in the $HUDSON_HOME provisioned for the Hudson instance or
  • by specifying a restart command on the command line when Hudson is invoked.

Command

In a nutshell, restart within Hudson will invoke a system administrator--supplied command.

Since the command is presumed to restart Hudson, it may not return at all. If it does return, it is expected to return successfully. If the restart command fails, restart will fail.

If the restart command is not specified, the existing default Lifecycle mechanism will be invoked.

If the hudson.lifecycle system property is specified, the restart command, if any, will not be used.

Default Restart Script

To allow system administrators to pre-configure Hudson for correct restart, if a restart script is present in HUDSON_HOME the initial value of the restart command will be set to invoke the script. The script must be named hudson_restart[.extension].

The script or program may have an extension, e.g., .bat on Windows or .sh, .bash, .py, etc. on Unix, but it doesn't matter what it is; in all cases, it will be invoked as a program.

The script must be executable.

The script should be specific to the environment in which the HUDSON_HOME is used.

Restart System Property

Even if a restart script is present in HUDSON_HOME, a system administrator may change the restart command to do something else, effectively ignoring the script, using the command line option:

-Dhudson.restart=value

The value of the hudson.restart system property will be used to initialize the restart command. The correct order of initialization is:

  • If -Dhudson.restart is specified, use that.
  • Otherwise, if a restart script is present, use a command that invokes the script.
  • Otherwise, the restart command is not specified.

Implementation

A new lifecycle, hudson.lifecycle.RestartCommandLifecycle, will be added to Hudson.

hudson.lifecycle.Lifecycle will be modified to use the RestartCommandLifecycle for restart if:

  • a restart script or command has been specified, and
  • the hudson.lifecycle system property is not defined.

The implementation will log a warning message if hudson.lifecycle has been specified and a restart script or command have been provided.


Restart Links or Buttons

Restart links should not be shown unless Lifecycle.get().canRestart() returns true. Otherwise, a message like "Restart required for changes to take effect" should be displayed.

A restart link should show the single word Restart and should call Hudson.safeRestart.

Plugin Manager

The Plugin Manager will show a restart link if a) a plugin is updated or a plugin is loaded that requires restart, and b) if Lifecycle.get().canRestart() returns true.