Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: for the plan.

Jump to: navigation, search

SMILA/Specifications/Processing Message Resequencer/Full Resequencer

not continued
this concept is not developed further ATM for lack of need and b/c it is basically the same as the Smart Resequencer. ideas there mostly apply here too.

Full Resequencer (FRS)

Synopsis: The Resequencer will update the processing target in the exact order as the crawler or agent adds PRs to connectivity.

The processing will be like so:

  1. the router will feed Q1 with PRs.
    For the resequencer to know the order, a new meta info needs to be added -- the sequence number (SN). it must be generated by the agent or by the agent controller
  2. the processing piplines are as normal, but:
    1. w/o the step of calling the processing target
    2. they add the result to a new queue, Q2
  3. the Resequencer will listen on Q2 and picks up all PRs
    1. starting with the first record: feed consecutive chunks of PRs to the processing target
    2. wait for PRs only a max. amount of time (timeout)
  • no processing target can ask for more and correct result is always possible
  • it is possible to add a note into the index for records ending up in the DLQ, ie. record was not indexed due to processing error.
  • setup is a more complex
  • added overhead due to more steps in the processing chain
  • change to agents or agent controller
  • overkill, b/c there is no need to resequence all PRs, only those with same ID
  • lost PRs will cause the FRS to delay all following PRs up to the timeout

Back to the top