Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Stardust/Knowledge Base/Integration/Camel/Camel Based Monitoring and Auditing Transient Processes Using SEDA
In this article we are going to explore, how we can use Stardust Camel Applications,Triggers to monitoring and auditing the details of Transient process.
- There is one process which is marked as transient(having only non-interactive activities), so this process will not be persisted in audittrail database.
- For auditing purpose user wants minimum information to be captured.
- Monitoring of Transient process should be asynchronous and should not affect it's execution.
Why Transient Process ?
Before we jump to modeling above use case let's first discuss why do we have decided the process should not be saved in audiTrail and marked it as Transient ? see below process
- As per above process if you see there is one camel trigger which is polling to a directory for some files.
- It picks the file and passed to another camel application "Send File to JMS/FTP server" which does some processing and again sends to FTP or JMS server.
- Since this process is processing the files and files sizes may vary for e.g. 100 MB 200 MB or bigger. And we are storing file content into Process data and this data will persist into auditrail db.
- Storing the big files in process data will result in un-necessary growing database size.
- To avoid above situation we decided, not to persist this process details in database by making this process as Transient:
- So once the process is completed all the process data will be discarded.
As we discussed transient process will not be saved to auditrail database it means we will not be able to identify what all files are processed by which processes ? From an auditing purpose atleast following information should be captured:
- File name of the processed files.
- File processing started by transient process.
- File Processing completed by transient process.
To solve above issue and monitoring the execution of transient process we used camel SEDA component.Because it provides asynchronous SEDA(Staged Event Driven Architecture) behavior:
- Modified camel route in application "Send file to JMS/FTP server" as below:
- Above route will send events to SEDA endpoints before and after completing the file processing.
- Create consumer for SEDA START events
- Add process descriptor to this process
- Create consumer for SEDA END events
- Add below process descriptor
NOTE: If you notice the SEDA endpoints, we use "size"(size=500) parameter to prevent even if consumer stops working then it will not result into your server to go OUT OF MEMORY.
- To execute discussed process download the attached model from here File:SEDA Model.zip
- Deploy the above downloaded model.
- Once the files will be processed by transient process for each processed file you could see below entries in process overview: