Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Talk:SMILA/Specifications/ProcessingPerformanceDiscussion"

(New page: TM: i can only give some high level input here due to the lack of impl. knowledge. but here are my 2 cents and correct my assumptions if they are wrong. As far as i understand, one of the...)
 
Line 1: Line 1:
 
TM: i can only give some high level input here due to the lack of impl. knowledge. but here are my 2 cents and correct my assumptions if they are wrong.
 
TM: i can only give some high level input here due to the lack of impl. knowledge. but here are my 2 cents and correct my assumptions if they are wrong.
  
As far as i understand, one of the bottle necks is the invocation of the BPEL pipeline for each record adding too much overhead.
+
: As far as i understand, one of the bottle necks is the invocation of the BPEL pipeline for each record adding too much overhead.
 +
:if that is the case: why not grab N records/massages according to config from the queue and stick them into the pipeline at once? (seems like u did that as part of "Send and Receive Queue messages in blocks" but i suggest here that grouping of records is only done on the ''Receive'' side.)
 +
:that should be fairly easy to impl. but at the costs that all pipelets would need to actually process the given Id[] and not just do the first one -- not sure if that is everywhere the case...
 +
:An extremist case where then to just have one listener thread that takes all messages from the queue that fits his config and put it into the pipeline.
  
if that is the case: why not grab N records/massages according to config from the queue and stick them into the pipeline at once?
+
:Really nice in this context would be a self regulating manager that adjusts the amount of max. records a listener grabs from the queue as opposed to having to config that. the manager checks the pipeline throughput of records in intervals and with an heuristic adjusts threads and max. records per thread.
 
+
that should be fairly easy to impl. but at the costs that all pipelets would need to actually process the given Id[] and not just do the first one -- not sure if that is everywhere the case...
+

Revision as of 04:35, 29 June 2009

TM: i can only give some high level input here due to the lack of impl. knowledge. but here are my 2 cents and correct my assumptions if they are wrong.

As far as i understand, one of the bottle necks is the invocation of the BPEL pipeline for each record adding too much overhead.
if that is the case: why not grab N records/massages according to config from the queue and stick them into the pipeline at once? (seems like u did that as part of "Send and Receive Queue messages in blocks" but i suggest here that grouping of records is only done on the Receive side.)
that should be fairly easy to impl. but at the costs that all pipelets would need to actually process the given Id[] and not just do the first one -- not sure if that is everywhere the case...
An extremist case where then to just have one listener thread that takes all messages from the queue that fits his config and put it into the pipeline.
Really nice in this context would be a self regulating manager that adjusts the amount of max. records a listener grabs from the queue as opposed to having to config that. the manager checks the pipeline throughput of records in intervals and with an heuristic adjusts threads and max. records per thread.

Back to the top