TM: i can only give some high level input here due to the lack of impl. knowledge. but here are my 2 cents and correct my assumptions if they are wrong.
- As far as i understand, one of the bottle necks is the invocation of the BPEL pipeline for each record adding too much overhead.
- if that is the case: why not grab N records/massages according to config from the queue and stick them into the pipeline at once? (seems like u did that as part of "Send and Receive Queue messages in blocks" but i suggest here that grouping of records is only done on the Receive side.)
- that should be fairly easy to impl. but at the costs that all pipelets would need to actually process the given Id and not just do the first one -- not sure if that is everywhere the case...
- An extremist case where then to just have one listener thread that takes all messages from the queue that fits his config and put it into the pipeline.
- Really nice in this context would be a self regulating manager that adjusts the amount of max. records a listener grabs from the queue as opposed to having to config that. the manager checks the pipeline throughput of records in intervals and with an heuristic adjusts threads and max. records per thread.
DS: Yes Tom, that's what I did. Getting at max N messages from the Queue and pass them to BPEL. This reduced the BPEL invocation overhead. And having multiple Listener worker thread still improves performance.
As there are more variables affecting performance troughput than only the block size N we should stick to a configurable implementation first and check out how it behaves. Later on we can implement a self adjusting Listener as you proposed.