Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Jetty/Feature/JVM NIO Bug"
m |
m (s/THREADHOLD/THREASHOLD/) |
||
| (One intermediate revision by one other user not shown) | |||
| Line 1: | Line 1: | ||
{{Jetty Feature | {{Jetty Feature | ||
|introduction= | |introduction= | ||
| + | |||
| + | {{Jetty Deprecated}} | ||
| + | |||
The Sun JVM contains several NIO bugs that appear to mostly affect Linux servers: | The Sun JVM contains several NIO bugs that appear to mostly affect Linux servers: | ||
* [http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933 6403933] | * [http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933 6403933] | ||
| Line 27: | Line 30: | ||
== Injecting Delay == | == Injecting Delay == | ||
| − | In some circumstances, even a newly created select set quickly suffers from the same problems. If Jetty detects that the <tt> | + | In some circumstances, even a newly created select set quickly suffers from the same problems. If Jetty detects that the <tt>JVMBUG_THREASHOLD</tt> has been exceeded, it reacts by inserting pauses in the selecting thread. The<tt>org.mortbay.io.nio.BUSY_PAUSE</tt> system parameter controls the duration of the pause; it defaults to 50ms. |
The pause allows the acceptor thread to stop calling select in a busy loop, and allows the dispatched threads to proceed with handling any selected connections. At 50ms, the worst case is that this delay adds 25ms latency to a request. However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay. | The pause allows the acceptor thread to stop calling select in a busy loop, and allows the dispatched threads to proceed with handling any selected connections. At 50ms, the worst case is that this delay adds 25ms latency to a request. However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay. | ||
Latest revision as of 12:11, 20 May 2014
Contents
Introduction
If you feel this should be brought back please file a bug.
The Sun JVM contains several NIO bugs that appear to mostly affect Linux servers:
Java 7 should fix these bugs. In the meantime, Jetty includes several work arounds that deal with these problems. When Jetty employs these work arounds, you see messages in your logs like the following:
JVM BUG(s) - cancelled keys 16 times JVM BUG(s) - recreating selector 4 times, cancelled keys 32 times JVM BUG(s) - injecting delay 8 times
Feature
Triggering Work Arounds
The workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, and before the expected timeout. The simple fixes (cancelling keys) always apply. Jetty attempts the more draconian fixes only if the number of triggers exceeds a threshhold within a set time period. Use the following system parameters to set these values:
- org.mortbay.io.nio.JVMBUG_THRESHHOLD, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
- org.mortbay.io.nio.MONITOR_PERIOD defaults to 1000 and is the period over which the threshhold applies.
Cancelling Keys
Jetty does not remove dispatch connections from the select set, as this is an expensive operation. Instead, Jetty sets the interested operations to zero, so the registered key should be ignored. However this is one of the conditions that triggers the JVM bugs, so if Jetty sees the selector returning with no keys selected, it cancels all keys with 0 interested operations.
Recreating Selector
If cancelling keys does not avoid the JVM bugs, when the JVMBUG_THRESHOLD is reached, Jetty discards the entire select set and creates a new one.
Injecting Delay
In some circumstances, even a newly created select set quickly suffers from the same problems. If Jetty detects that the JVMBUG_THREASHOLD has been exceeded, it reacts by inserting pauses in the selecting thread. Theorg.mortbay.io.nio.BUSY_PAUSE system parameter controls the duration of the pause; it defaults to 50ms.
The pause allows the acceptor thread to stop calling select in a busy loop, and allows the dispatched threads to proceed with handling any selected connections. At 50ms, the worst case is that this delay adds 25ms latency to a request. However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.
Busy key
If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection. If you set the system parameter org.mortbay.io.nio.BUSY_KEY to a positive number, Jetty closes the connection if the key is selected in excess of the busy key value in any one period. The default is -1.
Busy selector
As a final catch all measure, the Jetty selector puts a maximum limit on the number of selects per second per selector. You set this value in the org.mortbay.io.nio.MAX_SELECTS system parameter; it defaults to 15000. If this value is exceeded in a monitor period, Jetty injects delays into the select loop.