Jump to: navigation, search

Difference between revisions of "Jetty/Feature/JVM NIO Bug"

< Jetty‎ | Feature
(New page: {{Jetty Feature |introduction= The Sun JVM contains several NIO bugs that appear to mostly affect linux servers: * [http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933 6403933] * [h...)
 
Line 19: Line 19:
 
== Triggering Work Arounds ==
 
== Triggering Work Arounds ==
 
The JVM bugs workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, but before the expected timeout.  The simple fixes (cancelling keys) always apply.  The more draconian fixes are only attempted if the number of triggers exceeds a threshhold within a period, controlled by the following system parameters:
 
The JVM bugs workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, but before the expected timeout.  The simple fixes (cancelling keys) always apply.  The more draconian fixes are only attempted if the number of triggers exceeds a threshhold within a period, controlled by the following system parameters:
* org.mortbay.io.nio.JVMBUG_THRESHHOLD, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
+
* <tt>org.mortbay.io.nio.JVMBUG_THRESHHOLD</tt>, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
* org.mortbay.io.nio.MONITOR_PERIOD, defaults to 1000 and is the period over which the threshhold applies.
+
* <tt>org.mortbay.io.nio.MONITOR_PERIOD</tt> defaults to 1000 and is the period over which the threshhold applies.
  
 
== Cancelling Keys ==
 
== Cancelling Keys ==
Jetty has an optimization that dispatch connections are not removed from the select set, as this is an expensive operation. Instead, their interested operations are set to zero, so the registered key should be ignored.  However this is one of the conditions that triggers
+
Jetty has an optimization that dispatch connections are not removed from the select set, as this is an expensive operation. Instead, their interested operations are set to zero, so the registered key should be ignored.  However this is one of the conditions that triggers the JVM bugs, so if jetty sees the selector returning with no keys selected, it will cancel all keys with 0 interested operations.
 
+
  
 
== Recreating Selector ==
 
== Recreating Selector ==
 +
If cancelling keys is unable to avoid the JVM bugs, then when the <tt>JVMBUG_THRESHOLD</tt> is reached, jetty will discard the entire select set and create a new one.
  
 
== Injecting Delay ==
 
== Injecting Delay ==
 +
In some circumstances, even a newly created select set quickly suffers from the same problems.  If Jetty detects that the <tt>JVMBUG_THREADHOLD</tt> has been exceeded, then it reacts by inserting pauses in the selecting thread. The duration of the pause is controlled by the <tt>org.mortbay.io.nio.BUSY_PAUSE</tt> system parameter and defaults to 50ms.
 +
 +
The pause allows the acceptor thread to stop calling select in a busy loop and allows the dispatched threads to proceed with handling any selected connections.  At 50ms, the worst case is that this delay adds 25ms latency to a request.  However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.
  
 
== Busy key ==
 
== Busy key ==
 +
If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection.  If the system parameter <tt>org.mortbay.io.nio.BUSY_KEY</tt> is set to a positive number, then that connection will be closed if the key is selected in excess of the busy key value in any one period. The default is -1. 
  
 
== Busy selector ==
 
== Busy selector ==
 
+
As a final catch all measure, the jetty selector puts a maximum limit on the number of selects per second per selector.  This is set with the <tt>org.mortbay.io.nio.MAX_SELECTS</tt> system parameter and defaults to 15000. If this value is exceeded in a monitor period, then delays are injected into the select loop.
 
+
  
  
 
}}
 
}}

Revision as of 20:44, 9 February 2010



Introduction

The Sun JVM contains several NIO bugs that appear to mostly affect linux servers:

These bugs should be fixed in java 7, but jetty includes several work arounds to deal with these problems. If jetty has to employ these work arounds, you will see messages in your logs like:

  JVM BUG(s) - cancelled keys 16 times
  JVM BUG(s) - recreating selector 4 times, cancelled keys 32 times
  JVM BUG(s) - injecting delay 8 times

Feature

Triggering Work Arounds

The JVM bugs workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, but before the expected timeout. The simple fixes (cancelling keys) always apply. The more draconian fixes are only attempted if the number of triggers exceeds a threshhold within a period, controlled by the following system parameters:

  • org.mortbay.io.nio.JVMBUG_THRESHHOLD, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
  • org.mortbay.io.nio.MONITOR_PERIOD defaults to 1000 and is the period over which the threshhold applies.

Cancelling Keys

Jetty has an optimization that dispatch connections are not removed from the select set, as this is an expensive operation. Instead, their interested operations are set to zero, so the registered key should be ignored. However this is one of the conditions that triggers the JVM bugs, so if jetty sees the selector returning with no keys selected, it will cancel all keys with 0 interested operations.

Recreating Selector

If cancelling keys is unable to avoid the JVM bugs, then when the JVMBUG_THRESHOLD is reached, jetty will discard the entire select set and create a new one.

Injecting Delay

In some circumstances, even a newly created select set quickly suffers from the same problems. If Jetty detects that the JVMBUG_THREADHOLD has been exceeded, then it reacts by inserting pauses in the selecting thread. The duration of the pause is controlled by the org.mortbay.io.nio.BUSY_PAUSE system parameter and defaults to 50ms.

The pause allows the acceptor thread to stop calling select in a busy loop and allows the dispatched threads to proceed with handling any selected connections. At 50ms, the worst case is that this delay adds 25ms latency to a request. However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.

Busy key

If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection. If the system parameter org.mortbay.io.nio.BUSY_KEY is set to a positive number, then that connection will be closed if the key is selected in excess of the busy key value in any one period. The default is -1.

Busy selector

As a final catch all measure, the jetty selector puts a maximum limit on the number of selects per second per selector. This is set with the org.mortbay.io.nio.MAX_SELECTS system parameter and defaults to 15000. If this value is exceeded in a monitor period, then delays are injected into the select loop.