Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Stardust/Knowledge Base/Integration/Camel/Email Trigger with Process Attachments"
| (15 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | == Introduction == | + | == Introduction == |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| + | This article discusses an integration scenario where a Stardust process is kicked off by an incoming mail in a POP3 account that is being monitored by Camel. The incoming mail is parsed for certain keywords and the process to be invoked is selected dynamically based on the value of the keyword. Certain process data is also selected dynamically by parsing the mail contents. Any attachments that are part of the mail are converted into Documents and associated with the newly created Process Instance. | ||
| + | All the artifacts discussed in this article can be downloaded from [http://wiki.eclipse.org/images/0/06/Stardust_Integration_CamelEmail.zip here]. | ||
== Process Model == | == Process Model == | ||
| Line 11: | Line 9: | ||
[[Image:Stardust Integration Camel EmailTrigger Model1.png|900x400px]] | [[Image:Stardust Integration Camel EmailTrigger Model1.png|900x400px]] | ||
| + | <br> | ||
| + | |||
| + | The model above contains two processes viz. Purchase and Redemption. For simplicity, both the processes have been designed to be identical. We define two simple primitives viz. "Client" and "MailBody", both of type String. The "Client" data will be populated based on the value of a lookup made from within the Camel route depending on certain process data, while "MailBody" will contain the text of the Email. The first activity named "Wait for Doc Sync" is a Route activity which is set to Hibernate initially. We will discuss the reasons for this decision later in the article. The subsequent activity displays the contents of the data above in the Stardust portal. Note that both the processes need to be able to support "Process Attachments". This can be enabled by changing the process properties. | ||
| + | |||
| + | |||
| + | |||
| + | == Camel Route == | ||
| + | |||
| + | We next look at the details of the Camel route employed.While the default Stardust camel-context.xml file already ships with a few bean definitions and a "do-nothing" route, we will restrict ourselves to only discuss those beans and endpoints used in this integration. | ||
| + | |||
| + | [[Image:Stardust Integration Camel EmailTrigger Context.png]] | ||
| + | |||
| + | <br> | ||
| + | |||
| + | The "ippServiceFactoryAccess" bean belongs to the ISB component and is used within our custom bean to pass ServiceFactory credentials when invoking the Stardust services. We employ two custom beans in this integration viz. "mailProcessDataExtractor" and "ippProcessRouter". As seen above, the Camel Mail component is responsible for monitoring a POP3 inbox. The default polling interval is used here (60s). The "delete" attribute on the Endpoint ensures that the mail is processed only once by the Camel route and deleted upon successful execution of the route (mail deletion may not work on all servers). The Exchange is routed from the Mail endpoint to our custom bean. Note that since no method is explicitly specified, the method with the "@Handler" annotation will be invoked. We will discuss the code in our custom beans later in the article and limit ourselves to a higher level discussion in this section. If the mail happens to contain the right keywords, the mailProcessDataExtractor sets the "ippProcessId" header which identifies the process to be started to the "ipp" endpoint later in the route. This bean also sets a couple of custom headers viz. "dataValue" and "dataID". The "dataID" header contains the Id of the data within our process model that will contain the "dataValue" (in this case "Client"). The "dataValue" is the value to be set for "dataID". The bean parses the mail contents, scans certain keywords and matches these against a couple of property lookup files to determine these values. | ||
| + | |||
| + | If the mail does not happen to contain any valid keyword, the bean is unable to set the custom headers. The route terminates at this point (exit choice above). | ||
| + | |||
| + | If the mail does appear to contain valid keywords, the route then invokes our second custom bean viz. "ippProcessRouter". This bean is responsible for dynamically replacing the placeholder [header.dataID] in the IPP endpoint with the value computed by the mailProcessDataExtractor bean. This is explained in greater detail in the following section. The "ipp:authenticate' endpoint simply sets the proper user credentials to start the process and the subsequent step "ipp:process:start" actually starts off the process and sets the data values expected by the model. Note that at this point, no process attachments have been created due to the fact that we do not have a ProcessInstanceOID to associate the documents with until the process is started. The header "ippProcessInstanceOid" is set on the exchange body after the process is started. We make use of this value by calling the "moveDocumentstoPI" method on the mailProcessDataExtractor bean. This method moves the documents to the correct location under the Document Repository associated with the newly created Process Instance and sets the "PROCESS_ATTACHMENTS" header with a Document List that contains these Documents. The next endpoint in the route "ipp:process:setProperties" maps this header to the "PROCESS_ATTACHMENTS" data path of the process thereby associating the documents with the new Process Instance. At this point the need for a hibernated activity in the process model above becomes clear. We would like to ensure that the documents are associated with the correct Process Instance before we let the process continue any further. The above steps ensure that we achieve this goal. We are now free to complete the awaken the hibernated Route activity and let the next activity in the process execute. This is achieved by the final call to "ipp:process:continue" in the route. | ||
| + | |||
| + | == Custom Beans == | ||
| + | |||
| + | === Properties files === | ||
| + | |||
| + | Let us first take a look at the properties files referrred to by our mailProcessDataExtractor bean. This bean depends on two properties files viz.process.properties and datavalue.properties | ||
| + | |||
| + | ==== Process.properties ==== | ||
| + | |||
| + | This properties file defines the keywords that must be scanned within an email and the corresponding process to start if a keyword is encountered. For example, with the contents below, we specify that for a given row should any of the keywords on the right side be encountered (case insensitive), the process on the left should be started.So, on encountering the word "buy" in the email, the "Purchase" process would be started. Likewise, the word "sell" in a email with start the "Redemption" process. | ||
| + | |||
| + | <source lang="text"> | ||
| + | #comma separated list in the format ProcesID=[list of keywords that invoke this process] | ||
| + | Purchase=Purchase,Buy | ||
| + | Redemption=Sell,Redeem,Sale | ||
| + | </source> | ||
| + | |||
| + | ==== Datavalue.properties ==== | ||
| + | |||
| + | This properties file defines the keywords that must be scanned within an email and the corresponding value of the "dataID" and "dataValue" exchange headers.For example, with the contents below, we specify that for a given row should any of the comma separated keywords on the right side be encountered (case sensitive), the data with ID on the left (separated by ;) should be populated with the normalized form of the keyword (between = signs). So, on encountering the keyword "XYZ" for example, the data with ID "Client" would be set with the value "XYZ Corp". Likewise, if the email contains the keyword "PQR", the "Client" data will contain the value "PQR Inc". | ||
| + | |||
| + | <source lang="text"> | ||
| + | #comma separated list in the format Data,Data Value={variations of name to map to this data value} | ||
| + | Client;1=XYZ Corp=XYZ, XYZCo | ||
| + | Client;2=PQR Inc=PQR, PQRInc | ||
| + | </source> | ||
| + | |||
| + | === MailProcessDataExtractor Bean === | ||
| + | |||
| + | The structure of the MailProcessDataExtractor bean is as seen below: | ||
| + | |||
| + | [[Image:Stardust Integration Camel EmailTrigger MailProcessDataExtractor.png]] | ||
| + | |||
| + | A quick review of the main methods follows. A detailed discussion of this class is beyond the scope of this article and left as an exercise to the reader.<br> <source lang="text"> | ||
| + | get/setServiceFactoryAccess: | ||
| + | Injected by Spring, credentials used to access Stardust services such as DMS and workflow. | ||
| + | getPropertiesFromClasspath: | ||
| + | Load the properties files (discussed above) and called by constructor. | ||
| + | MailProcessDataExtractor: | ||
| + | Invokes the above method and sets Object properties. | ||
| + | processEmailBody: | ||
| + | Processes the Exchange containing the email details, converts attachments to Stardust Documents, | ||
| + | extracts mail body and set various headers | ||
| + | getContentFromMail: | ||
| + | Gets the mail body for mails with attachments or forwards (multipart). Invoked by processEmailBody | ||
| + | setHeadersAndBodyForIpp: | ||
| + | Does the actual text parsing, comparison and setting of headers for Process ID, Data and DataValue. | ||
| + | Also adds the mail text to the exchange message body. Called by processEmailBody. | ||
| + | convertToDocument: | ||
| + | Does the actual work of converting the attachment to a Document. The Document is stored under a | ||
| + | folder called "mail_trigger_attachments" temporarily until it is moved upon creation of the | ||
| + | Process Instance. Invoked by processEmailBody. | ||
| + | moveDocumentstoPI: | ||
| + | Moves the documents to the Process Instance specific folder once the process starts. Invoked directly | ||
| + | by Camel. | ||
| + | </source> | ||
| + | |||
| + | === IPPProcessRouter Bean === | ||
| + | |||
| + | The structure of the IPPProcessRouter bean is as shown below: | ||
| + | |||
| + | [[Image:Stardust Integration Camel EmailTrigger IPPProcessRouter.png]] | ||
| + | |||
| + | This bean contains only one method viz. "routeToIPP". This method is responsible for dynamically modifying the "ipp:process:start" endpoint and replacing the placeholder [header.dataID] with the actual data ID contained in the model and set in the header of the same name. For example, with the properties file defined as above, upon execution of this method, the endpoint <source lang="text"> | ||
| + | <to uri="ipp:process:start?data=[header.dataID]::$simple{header.dataValue},MailBody::$simple{body}" /> | ||
| + | </source> would be replaced with <source lang="text"> | ||
| + | <to uri="ipp:process:start?data=Client::$simple{header.dataValue},MailBody::$simple{body}" /> | ||
| + | </source> since this is the actual data ID value as obtained from the properties file and placed in the header "dataID". This is now a valid endpoint and on invocation, a new Stardust process will be started and the data with ID "Client" (a primitive String as shown in the model above) will be populated with the value in the header "dataValue". | ||
| + | |||
| + | == Seeing it in action<br> == | ||
| + | |||
| + | To prepare the project and execute the route, please follow the following steps: | ||
| + | #If you have direct access to a POP3 mail server, you can use that. If firewall issues prevent access or if you would like to set up your own mail server on a development environment you can use Apache James. Refer to the documentation available on the [http://james.apache.org Apache James site]. | ||
| + | #Set up a mail client to send mails. Mozilla Thunderbird is a popular option. You can configure it easily to connect to your James server. This client also supports mail attachments. You can download it from [http://www.mozilla.org/thunderbird/ here]. | ||
| + | #Send a test mail to yourself from Thunderbird and check that you receive it. This will ensure that there are no connectivity issues. | ||
| + | #Create a "Dynamic Web Application" with the "Stardust Portal" facet included. Add the Java classes and properties files into your project "src" directory. | ||
| + | #Edit the existing camel-context.xml file and replace it with the one provided here. Edit the pop3 account setting as appropriate. You can add additional endpoint properties if you so desire. Refer to the documentation available on the [http://camel.apache.org/mail.html Apache Camel site]. | ||
| + | #Deploy the project to Tomcat and start up the server. | ||
| + | #Deploy the process model. | ||
| + | #Test the route by sending mails to your configured account with/without attachments and with variations of the keywords in the properties files. Open up the Stardust portal and check whether the "Client" and "MailBody" data values are populated properly in the manual activity. Also check if the process attachments show up correctly. An example activity view with the process attachments visible would look like this: | ||
| − | + | [[Image:Stardust_Integration_Camel_EmailTrigger_Portal_New.png]]<br> | |
Latest revision as of 11:36, 14 September 2012
Contents
Introduction
This article discusses an integration scenario where a Stardust process is kicked off by an incoming mail in a POP3 account that is being monitored by Camel. The incoming mail is parsed for certain keywords and the process to be invoked is selected dynamically based on the value of the keyword. Certain process data is also selected dynamically by parsing the mail contents. Any attachments that are part of the mail are converted into Documents and associated with the newly created Process Instance.
All the artifacts discussed in this article can be downloaded from here.
Process Model
The model above contains two processes viz. Purchase and Redemption. For simplicity, both the processes have been designed to be identical. We define two simple primitives viz. "Client" and "MailBody", both of type String. The "Client" data will be populated based on the value of a lookup made from within the Camel route depending on certain process data, while "MailBody" will contain the text of the Email. The first activity named "Wait for Doc Sync" is a Route activity which is set to Hibernate initially. We will discuss the reasons for this decision later in the article. The subsequent activity displays the contents of the data above in the Stardust portal. Note that both the processes need to be able to support "Process Attachments". This can be enabled by changing the process properties.
Camel Route
We next look at the details of the Camel route employed.While the default Stardust camel-context.xml file already ships with a few bean definitions and a "do-nothing" route, we will restrict ourselves to only discuss those beans and endpoints used in this integration.
The "ippServiceFactoryAccess" bean belongs to the ISB component and is used within our custom bean to pass ServiceFactory credentials when invoking the Stardust services. We employ two custom beans in this integration viz. "mailProcessDataExtractor" and "ippProcessRouter". As seen above, the Camel Mail component is responsible for monitoring a POP3 inbox. The default polling interval is used here (60s). The "delete" attribute on the Endpoint ensures that the mail is processed only once by the Camel route and deleted upon successful execution of the route (mail deletion may not work on all servers). The Exchange is routed from the Mail endpoint to our custom bean. Note that since no method is explicitly specified, the method with the "@Handler" annotation will be invoked. We will discuss the code in our custom beans later in the article and limit ourselves to a higher level discussion in this section. If the mail happens to contain the right keywords, the mailProcessDataExtractor sets the "ippProcessId" header which identifies the process to be started to the "ipp" endpoint later in the route. This bean also sets a couple of custom headers viz. "dataValue" and "dataID". The "dataID" header contains the Id of the data within our process model that will contain the "dataValue" (in this case "Client"). The "dataValue" is the value to be set for "dataID". The bean parses the mail contents, scans certain keywords and matches these against a couple of property lookup files to determine these values.
If the mail does not happen to contain any valid keyword, the bean is unable to set the custom headers. The route terminates at this point (exit choice above).
If the mail does appear to contain valid keywords, the route then invokes our second custom bean viz. "ippProcessRouter". This bean is responsible for dynamically replacing the placeholder [header.dataID] in the IPP endpoint with the value computed by the mailProcessDataExtractor bean. This is explained in greater detail in the following section. The "ipp:authenticate' endpoint simply sets the proper user credentials to start the process and the subsequent step "ipp:process:start" actually starts off the process and sets the data values expected by the model. Note that at this point, no process attachments have been created due to the fact that we do not have a ProcessInstanceOID to associate the documents with until the process is started. The header "ippProcessInstanceOid" is set on the exchange body after the process is started. We make use of this value by calling the "moveDocumentstoPI" method on the mailProcessDataExtractor bean. This method moves the documents to the correct location under the Document Repository associated with the newly created Process Instance and sets the "PROCESS_ATTACHMENTS" header with a Document List that contains these Documents. The next endpoint in the route "ipp:process:setProperties" maps this header to the "PROCESS_ATTACHMENTS" data path of the process thereby associating the documents with the new Process Instance. At this point the need for a hibernated activity in the process model above becomes clear. We would like to ensure that the documents are associated with the correct Process Instance before we let the process continue any further. The above steps ensure that we achieve this goal. We are now free to complete the awaken the hibernated Route activity and let the next activity in the process execute. This is achieved by the final call to "ipp:process:continue" in the route.
Custom Beans
Properties files
Let us first take a look at the properties files referrred to by our mailProcessDataExtractor bean. This bean depends on two properties files viz.process.properties and datavalue.properties
Process.properties
This properties file defines the keywords that must be scanned within an email and the corresponding process to start if a keyword is encountered. For example, with the contents below, we specify that for a given row should any of the keywords on the right side be encountered (case insensitive), the process on the left should be started.So, on encountering the word "buy" in the email, the "Purchase" process would be started. Likewise, the word "sell" in a email with start the "Redemption" process.
#comma separated list in the format ProcesID=[list of keywords that invoke this process] Purchase=Purchase,Buy Redemption=Sell,Redeem,Sale
Datavalue.properties
This properties file defines the keywords that must be scanned within an email and the corresponding value of the "dataID" and "dataValue" exchange headers.For example, with the contents below, we specify that for a given row should any of the comma separated keywords on the right side be encountered (case sensitive), the data with ID on the left (separated by ;) should be populated with the normalized form of the keyword (between = signs). So, on encountering the keyword "XYZ" for example, the data with ID "Client" would be set with the value "XYZ Corp". Likewise, if the email contains the keyword "PQR", the "Client" data will contain the value "PQR Inc".
#comma separated list in the format Data,Data Value={variations of name to map to this data value}
Client;1=XYZ Corp=XYZ, XYZCo
Client;2=PQR Inc=PQR, PQRIncMailProcessDataExtractor Bean
The structure of the MailProcessDataExtractor bean is as seen below:
get/setServiceFactoryAccess: Injected by Spring, credentials used to access Stardust services such as DMS and workflow. getPropertiesFromClasspath: Load the properties files (discussed above) and called by constructor. MailProcessDataExtractor: Invokes the above method and sets Object properties. processEmailBody: Processes the Exchange containing the email details, converts attachments to Stardust Documents, extracts mail body and set various headers getContentFromMail: Gets the mail body for mails with attachments or forwards (multipart). Invoked by processEmailBody setHeadersAndBodyForIpp: Does the actual text parsing, comparison and setting of headers for Process ID, Data and DataValue. Also adds the mail text to the exchange message body. Called by processEmailBody. convertToDocument: Does the actual work of converting the attachment to a Document. The Document is stored under a folder called "mail_trigger_attachments" temporarily until it is moved upon creation of the Process Instance. Invoked by processEmailBody. moveDocumentstoPI: Moves the documents to the Process Instance specific folder once the process starts. Invoked directly by Camel.
IPPProcessRouter Bean
The structure of the IPPProcessRouter bean is as shown below:
<to uri="ipp:process:start?data=[header.dataID]::$simple{header.dataValue},MailBody::$simple{body}" /><to uri="ipp:process:start?data=Client::$simple{header.dataValue},MailBody::$simple{body}" />Seeing it in action
To prepare the project and execute the route, please follow the following steps:
- If you have direct access to a POP3 mail server, you can use that. If firewall issues prevent access or if you would like to set up your own mail server on a development environment you can use Apache James. Refer to the documentation available on the Apache James site.
- Set up a mail client to send mails. Mozilla Thunderbird is a popular option. You can configure it easily to connect to your James server. This client also supports mail attachments. You can download it from here.
- Send a test mail to yourself from Thunderbird and check that you receive it. This will ensure that there are no connectivity issues.
- Create a "Dynamic Web Application" with the "Stardust Portal" facet included. Add the Java classes and properties files into your project "src" directory.
- Edit the existing camel-context.xml file and replace it with the one provided here. Edit the pop3 account setting as appropriate. You can add additional endpoint properties if you so desire. Refer to the documentation available on the Apache Camel site.
- Deploy the project to Tomcat and start up the server.
- Deploy the process model.
- Test the route by sending mails to your configured account with/without attachments and with variations of the keywords in the properties files. Open up the Stardust portal and check whether the "Client" and "MailBody" data values are populated properly in the manual activity. Also check if the process attachments show up correctly. An example activity view with the process attachments visible would look like this:
