Jetty/Feature/JVM NIO Bug

Introduction

The Sun JVM contains several NIO bugs that appear to mostly affect Linux servers:

• 6403933
• 6693490

Java 7 should fix these bugs. In the meantime, Jetty includes several work arounds that deal with these problems. When Jetty employs these work arounds, you see messages in your logs like the following:

  JVM BUG(s) - cancelled keys 16 times
  JVM BUG(s) - recreating selector 4 times, cancelled keys 32 times
  JVM BUG(s) - injecting delay 8 times

Triggering Work Arounds

The workarounds trigger whenever the NIO SelectSet returns from a select with zero keys selected, and before the expected timeout. The simple fixes (cancelling keys) always apply. Jetty attempts the more draconian fixes only if the number of triggers exceeds a threshhold within a set time period. Use the following system parameters to set these values:

• org.mortbay.io.nio.JVMBUG_THRESHHOLD, defaults to 512 and is the number of zero select returns that must be exceeded in a period.
• org.mortbay.io.nio.MONITOR_PERIOD defaults to 1000 and is the period over which the threshhold applies.

Cancelling Keys

Jetty does not remove dispatch connections from the select set, as this is an expensive operation. Instead, Jetty sets the interested operations to zero, so the registered key should be ignored. However this is one of the conditions that triggers the JVM bugs, so if Jetty sees the selector returning with no keys selected, it cancels all keys with 0 interested operations.

Recreating Selector

If cancelling keys does not avoid the JVM bugs, when the JVMBUG_THRESHOLD is reached, Jetty discards the entire select set and creates a new one.

Injecting Delay

In some circumstances, even a newly created select set quickly suffers from the same problems. If Jetty detects that the JVMBUG_THREADHOLD has been exceeded, it reacts by inserting pauses in the selecting thread. Theorg.mortbay.io.nio.BUSY_PAUSE system parameter controls the duration of the pause; it defaults to 50ms.

The pause allows the acceptor thread to stop calling select in a busy loop, and allows the dispatched threads to proceed with handling any selected connections. At 50ms, the worst case is that this delay adds 25ms latency to a request. However, in practise these problems only occur on busy servers with jobs in excess of the available CPUs, so this 25ms is probably not much in excess of an expected scheduling delay.

Busy key

If a single select key is repeatedly selected as the only active key, this can indicate a problem with that connection. If you set the system parameter org.mortbay.io.nio.BUSY_KEY to a positive number, Jetty closes the connection if the key is selected in excess of the busy key value in any one period. The default is -1.

Busy selector

As a final catch all measure, the Jetty selector puts a maximum limit on the number of selects per second per selector. You set this value in the org.mortbay.io.nio.MAX_SELECTS system parameter; it defaults to 15000. If this value is exceeded in a monitor period, Jetty injects delays into the select loop.