Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Jetty/Feature/JVM NIO Bug"

< Jetty‎ | Feature
m
Line 1: Line 1:
 
{{Jetty Feature
 
{{Jetty Feature
 
|introduction=
 
|introduction=
The Sun JVM contains several NIO bugs that appear to mostly affect linux servers:
+
The Sun JVM contains several NIO bugs that appear to mostly affect Linux servers:
 
* [http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933 6403933]
 
* [http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933 6403933]
 
* [http://bugs.sun.com/view_bug.do?bug_id=6693490 6693490]
 
* [http://bugs.sun.com/view_bug.do?bug_id=6693490 6693490]
  
These bugs should be fixed in java 7, but jetty includes several work arounds to
+
Java 7 should fix these bugs. In the meantime, Jetty includes several work arounds that deal with these problems. When Jetty employs these work arounds, you see messages in your logs like the following:
deal with these problems.   If jetty has to employ these work arounds, you will
+
see messages in your logs like:
+
  
 
<pre>
 
<pre>
Line 18: Line 16:
  
 
== Triggering Work Arounds ==
 
== Triggering Work Arounds ==
The JVM bugs workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, but before the expected timeout.   The simple fixes (cancelling keys) always apply. The more draconian fixes are only attempted if the number of triggers exceeds a threshhold within a period, controlled by the following system parameters:
+
The workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, and before the expected timeout. The simple fixes (cancelling keys) always apply. Jetty attempts the more draconian fixes only if the number of triggers exceeds a threshhold within a set time period. Use the following system parameters to set these values:
 
* <tt>org.mortbay.io.nio.JVMBUG_THRESHHOLD</tt>, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
 
* <tt>org.mortbay.io.nio.JVMBUG_THRESHHOLD</tt>, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
 
* <tt>org.mortbay.io.nio.MONITOR_PERIOD</tt> defaults to 1000 and is the period over which the threshhold applies.
 
* <tt>org.mortbay.io.nio.MONITOR_PERIOD</tt> defaults to 1000 and is the period over which the threshhold applies.
  
 
== Cancelling Keys ==
 
== Cancelling Keys ==
Jetty has an optimization that dispatch connections are not removed from the select set, as this is an expensive operation. Instead, their interested operations are set to zero, so the registered key should be ignored. However this is one of the conditions that triggers the JVM bugs, so if jetty sees the selector returning with no keys selected, it will cancel all keys with 0 interested operations.
+
Jetty does not remove dispatch connections from the select set, as this is an expensive operation. Instead, Jetty sets the interested operations to zero, so the registered key should be ignored. However this is one of the conditions that triggers the JVM bugs, so if Jetty sees the selector returning with no keys selected, it cancels all keys with 0 interested operations.
  
 
== Recreating Selector ==
 
== Recreating Selector ==
If cancelling keys is unable to avoid the JVM bugs, then when the <tt>JVMBUG_THRESHOLD</tt> is reached, jetty will discard the entire select set and create a new one.
+
If cancelling keys does not avoid the JVM bugs, when the <tt>JVMBUG_THRESHOLD</tt> is reached, Jetty discards the entire select set and creates a new one.
  
 
== Injecting Delay ==
 
== Injecting Delay ==
In some circumstances, even a newly created select set quickly suffers from the same problems.  If Jetty detects that the <tt>JVMBUG_THREADHOLD</tt> has been exceeded, then it reacts by inserting pauses in the selecting thread. The duration of the pause is controlled by the <tt>org.mortbay.io.nio.BUSY_PAUSE</tt> system parameter and defaults to 50ms.
+
In some circumstances, even a newly created select set quickly suffers from the same problems.  If Jetty detects that the <tt>JVMBUG_THREADHOLD</tt> has been exceeded, it reacts by inserting pauses in the selecting thread. The<tt>org.mortbay.io.nio.BUSY_PAUSE</tt> system parameter controls the duration of the pause; it defaults to 50ms.
  
The pause allows the acceptor thread to stop calling select in a busy loop and allows the dispatched threads to proceed with handling any selected connections.  At 50ms, the worst case is that this delay adds 25ms latency to a request.  However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.
+
The pause allows the acceptor thread to stop calling select in a busy loop, and allows the dispatched threads to proceed with handling any selected connections.  At 50ms, the worst case is that this delay adds 25ms latency to a request.  However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.
  
 
== Busy key ==
 
== Busy key ==
If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection.  If the system parameter <tt>org.mortbay.io.nio.BUSY_KEY</tt> is set to a positive number, then that connection will be closed if the key is selected in excess of the busy key value in any one period. The default is -1.   
+
If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection.  If you set the system parameter <tt>org.mortbay.io.nio.BUSY_KEY</tt> to a positive number, Jetty closes the connection if the key is selected in excess of the busy key value in any one period. The default is -1.   
  
 
== Busy selector ==
 
== Busy selector ==
As a final catch all measure, the jetty selector puts a maximum limit on the number of selects per second per selector.  This is set with the <tt>org.mortbay.io.nio.MAX_SELECTS</tt> system parameter and defaults to 15000. If this value is exceeded in a monitor period, then delays are injected into the select loop.
+
As a final catch all measure, the Jetty selector puts a maximum limit on the number of selects per second per selector.  You set this value in the <tt>org.mortbay.io.nio.MAX_SELECTS</tt> system parameter; it defaults to 15000. If this value is exceeded in a monitor period, Jetty injects delays into the select loop.
  
  
 
}}
 
}}

Revision as of 16:27, 12 September 2011



Introduction

The Sun JVM contains several NIO bugs that appear to mostly affect Linux servers:

Java 7 should fix these bugs. In the meantime, Jetty includes several work arounds that deal with these problems. When Jetty employs these work arounds, you see messages in your logs like the following:

  JVM BUG(s) - cancelled keys 16 times
  JVM BUG(s) - recreating selector 4 times, cancelled keys 32 times
  JVM BUG(s) - injecting delay 8 times

Feature

Triggering Work Arounds

The workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, and before the expected timeout. The simple fixes (cancelling keys) always apply. Jetty attempts the more draconian fixes only if the number of triggers exceeds a threshhold within a set time period. Use the following system parameters to set these values:

  • org.mortbay.io.nio.JVMBUG_THRESHHOLD, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
  • org.mortbay.io.nio.MONITOR_PERIOD defaults to 1000 and is the period over which the threshhold applies.

Cancelling Keys

Jetty does not remove dispatch connections from the select set, as this is an expensive operation. Instead, Jetty sets the interested operations to zero, so the registered key should be ignored. However this is one of the conditions that triggers the JVM bugs, so if Jetty sees the selector returning with no keys selected, it cancels all keys with 0 interested operations.

Recreating Selector

If cancelling keys does not avoid the JVM bugs, when the JVMBUG_THRESHOLD is reached, Jetty discards the entire select set and creates a new one.

Injecting Delay

In some circumstances, even a newly created select set quickly suffers from the same problems. If Jetty detects that the JVMBUG_THREADHOLD has been exceeded, it reacts by inserting pauses in the selecting thread. Theorg.mortbay.io.nio.BUSY_PAUSE system parameter controls the duration of the pause; it defaults to 50ms.

The pause allows the acceptor thread to stop calling select in a busy loop, and allows the dispatched threads to proceed with handling any selected connections. At 50ms, the worst case is that this delay adds 25ms latency to a request. However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.

Busy key

If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection. If you set the system parameter org.mortbay.io.nio.BUSY_KEY to a positive number, Jetty closes the connection if the key is selected in excess of the busy key value in any one period. The default is -1.

Busy selector

As a final catch all measure, the Jetty selector puts a maximum limit on the number of selects per second per selector. You set this value in the org.mortbay.io.nio.MAX_SELECTS system parameter; it defaults to 15000. If this value is exceeded in a monitor period, Jetty injects delays into the select loop.

Back to the top