Jetty/Feature/JVM NIO Bug
The Sun JVM contains several NIO bugs that appear to mostly affect linux servers:
These bugs should be fixed in java 7, but jetty includes several work arounds to deal with these problems. If jetty has to employ these work arounds, you will see messages in your logs like:
JVM BUG(s) - cancelled keys 16 times JVM BUG(s) - recreating selector 4 times, cancelled keys 32 times JVM BUG(s) - injecting delay 8 times
Triggering Work Arounds
The JVM bugs workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, but before the expected timeout. The simple fixes (cancelling keys) always apply. The more draconian fixes are only attempted if the number of triggers exceeds a threshhold within a period, controlled by the following system parameters:
- org.mortbay.io.nio.JVMBUG_THRESHHOLD, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
- org.mortbay.io.nio.MONITOR_PERIOD defaults to 1000 and is the period over which the threshhold applies.
Jetty has an optimization that dispatch connections are not removed from the select set, as this is an expensive operation. Instead, their interested operations are set to zero, so the registered key should be ignored. However this is one of the conditions that triggers the JVM bugs, so if jetty sees the selector returning with no keys selected, it will cancel all keys with 0 interested operations.
If cancelling keys is unable to avoid the JVM bugs, then when the JVMBUG_THRESHOLD is reached, jetty will discard the entire select set and create a new one.
In some circumstances, even a newly created select set quickly suffers from the same problems. If Jetty detects that the JVMBUG_THREADHOLD has been exceeded, then it reacts by inserting pauses in the selecting thread. The duration of the pause is controlled by the org.mortbay.io.nio.BUSY_PAUSE system parameter and defaults to 50ms.
The pause allows the acceptor thread to stop calling select in a busy loop and allows the dispatched threads to proceed with handling any selected connections. At 50ms, the worst case is that this delay adds 25ms latency to a request. However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.
If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection. If the system parameter org.mortbay.io.nio.BUSY_KEY is set to a positive number, then that connection will be closed if the key is selected in excess of the busy key value in any one period. The default is -1.
As a final catch all measure, the jetty selector puts a maximum limit on the number of selects per second per selector. This is set with the org.mortbay.io.nio.MAX_SELECTS system parameter and defaults to 15000. If this value is exceeded in a monitor period, then delays are injected into the select loop.