SMILA/Specifications/Processing Message Resequencer/Full Resequencer
Full Resequencer (FRS)
Synopsis: The Resequencer will update the processing target in the exact order as the crawler or agent adds PRs to connectivity.
The processing will be like so:
- the router will feed Q1 with PRs.
For the resequencer to know the order, a new meta info needs to be added -- the sequence number (SN). it must be generated by the agent or by the agent controller
- the processing piplines are as normal, but:
- w/o the step of calling the processing target
- they add the result to a new queue, Q2
- the Resequencer will listen on Q2 and picks up all PRs
- starting with the first record: feed consecutive chunks of PRs to the processing target
- wait for PRs only a max. amount of time (timeout)
- no processing target can ask for more and correct result is always possible
- it is possible to add a note into the index for records ending up in the DLQ, ie. record was not indexed due to processing error.
- setup is a more complex
- added overhead due to more steps in the processing chain
- change to agents or agent controller
- overkill, b/c there is no need to resequence all PRs, only those with same ID
- lost PRs will cause the FRS to delay all following PRs up to the timeout