Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Stardust/Knowledge Base/Performance Tuning/Achieving high Throughput with transient Processes"
(→Abort Process) |
m |
||
(8 intermediate revisions by the same user not shown) | |||
Line 8: | Line 8: | ||
* Significantly higher throughput | * Significantly higher throughput | ||
− | Additional information about Transient Processes is available [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.dev/html/handbooks/concepts/audit-trail/transient-procs.html here]. All artifacts referred to in the following discussion can be downloaded from [ | + | Additional information about Transient Processes is available [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.dev/html/handbooks/concepts/audit-trail/transient-procs.html here]. All artifacts referred to in the following discussion can be downloaded from [https://wiki.eclipse.org/images/f/fc/Stardust_Knowledge_Base_Performance_Tuning_TransientProcesses.zip here]. |
== Example == | == Example == | ||
Line 16: | Line 16: | ||
As shown above, the process consists of a couple of activities each of which invokes a POJO with a couple of In/Out parameters. Ordinarily this data would be persisted to the AuditTrail during process execution and the process history would display the execution sequence if one were to view this information in the Portal. To mark the process as transient we select the corresponding property in the Process Properties dialog box (see Figure 2 below).<br /> | As shown above, the process consists of a couple of activities each of which invokes a POJO with a couple of In/Out parameters. Ordinarily this data would be persisted to the AuditTrail during process execution and the process history would display the execution sequence if one were to view this information in the Portal. To mark the process as transient we select the corresponding property in the Process Properties dialog box (see Figure 2 below).<br /> | ||
[[File:Stardust Knowledge Base Performance Tuning Transient2.png|center|Figure 2: Process Properties for Transient Process]]<br /><br /> | [[File:Stardust Knowledge Base Performance Tuning Transient2.png|center|Figure 2: Process Properties for Transient Process]]<br /><br /> | ||
− | Figure 2: Process Properties for Transient Process<br /> | + | Figure 2: Process Properties for Transient Process<br /><br /> |
To ensure that this property is used by the Stardust engine at runtime we need to set the “SupportTransientProcesses” property in the carnot.properties file. We also need to ensure that the database we use supports sequences (such as Oracle). For this discussion we will use MySQL as our AuditTrail database. To enable support for sequences in MySQL we are required to do the following: | To ensure that this property is used by the Stardust engine at runtime we need to set the “SupportTransientProcesses” property in the carnot.properties file. We also need to ensure that the database we use supports sequences (such as Oracle). For this discussion we will use MySQL as our AuditTrail database. To enable support for sequences in MySQL we are required to do the following: | ||
− | + | ==== Create a compatible Stardust AuditTrail schema ==== | |
Create a MySQL database and run the sysconsole command as follows to create an AuditTrail that supports sequences: | Create a MySQL database and run the sysconsole command as follows to create an AuditTrail that supports sequences: | ||
<source lang="text"> | <source lang="text"> | ||
Line 27: | Line 27: | ||
Note that the dbtype option is set to "mysql_seq". Additional information on creating an AuditTrail on MySQL is available [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.deployment/html/audittraildatabasesetup/mysql.html here]. To learn more about the sysconsole command please refer to the Stardust [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.dev/html/handbooks/operation/commandline/ag-commandline-2.htm documentation]. | Note that the dbtype option is set to "mysql_seq". Additional information on creating an AuditTrail on MySQL is available [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.deployment/html/audittraildatabasesetup/mysql.html here]. To learn more about the sysconsole command please refer to the Stardust [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.dev/html/handbooks/operation/commandline/ag-commandline-2.htm documentation]. | ||
− | + | ==== Set appropriate database properties in carnot.properties ==== | |
Add/Edit the following entries to carnot.properties: | Add/Edit the following entries to carnot.properties: | ||
<source lang="text"> | <source lang="text"> | ||
Line 39: | Line 39: | ||
The URL, Schema, User and Password entries should match the values provided in the sysconsole command above. | The URL, Schema, User and Password entries should match the values provided in the sysconsole command above. | ||
− | + | ==== Provide an implementation of a ClusterSafeObjectProviderHolder ==== | |
+ | This is required by the Stardust engine to share objects internally. A simple implementation is provided below: | ||
[[File:Stardust Knowledge Base Performance Tuning Transient3.png|center|Figure 3: Custom ClusterSafeObjectProvider implementation]] | [[File:Stardust Knowledge Base Performance Tuning Transient3.png|center|Figure 3: Custom ClusterSafeObjectProvider implementation]] | ||
<br /> | <br /> | ||
− | Figure 3: Custom ClusterSafeObjectProvider implementation<br /> | + | Figure 3: Custom ClusterSafeObjectProvider implementation<br /><br /> |
Add the following line to your carnot.properties entry so that the Stardust engine can use your implementation at runtime: | Add the following line to your carnot.properties entry so that the Stardust engine can use your implementation at runtime: | ||
<source lang="text"> | <source lang="text"> | ||
Line 56: | Line 57: | ||
In this section we identify scenarios that cause the Stardust engine to switch from “Transient” to “Immediate” persistence indicating that process data will be written to the AuditTrail. Note that we do not discuss all such scenarios. The reader is encouraged to refer to the documentation and explore further with this background information. | In this section we identify scenarios that cause the Stardust engine to switch from “Transient” to “Immediate” persistence indicating that process data will be written to the AuditTrail. Note that we do not discuss all such scenarios. The reader is encouraged to refer to the documentation and explore further with this background information. | ||
− | === Manual Activity === | + | ==== Manual Activity ==== |
We start off by adding a manual activity to the process flow and study the difference in behavior | We start off by adding a manual activity to the process flow and study the difference in behavior | ||
[[File:Stardust Knowledge Base Performance Tuning Transient6.png|center|Transient Process with Manual Activity]]<br /> | [[File:Stardust Knowledge Base Performance Tuning Transient6.png|center|Transient Process with Manual Activity]]<br /> | ||
Line 68: | Line 69: | ||
Figure 8: Activity_Instance and Process_Instance tables after switching to Immediate Persistence<br /><br /> | Figure 8: Activity_Instance and Process_Instance tables after switching to Immediate Persistence<br /><br /> | ||
− | === Runtime Exception === | + | ==== Runtime Exception ==== |
We execute another process that throws a Runtime Exception (process “Transient Process with Runtime Exception” in attached model) (Figure 9 below).[[File:Stardust Knowledge Base Performance Tuning Transient9.png|center|Transient Process that throws a Runtime Exception]]<br /> | We execute another process that throws a Runtime Exception (process “Transient Process with Runtime Exception” in attached model) (Figure 9 below).[[File:Stardust Knowledge Base Performance Tuning Transient9.png|center|Transient Process that throws a Runtime Exception]]<br /> | ||
Figure 9: Transient Process that throws a Runtime Exception<br /><br /> | Figure 9: Transient Process that throws a Runtime Exception<br /><br /> | ||
Line 77: | Line 78: | ||
[[File:Stardust Knowledge Base Performance Tuning Transient11_1.png|center|Activity Instance and Process Instance tables after Runtime Exception occurs]]<br /> | [[File:Stardust Knowledge Base Performance Tuning Transient11_1.png|center|Activity Instance and Process Instance tables after Runtime Exception occurs]]<br /> | ||
Figure 11: Activity Instance and Process Instance tables after Runtime Exception occurs<br /><br /> | Figure 11: Activity Instance and Process Instance tables after Runtime Exception occurs<br /><br /> | ||
− | === Hibernate Initially === | + | ==== Hibernate Initially ==== |
Next, we execute the process “Transient Process with Hibernate Initially”. The process is identical to the one in Figure 6 above except for the “Hibernate Initially” flag which is set to true for the activity “Call Second POJO” (Figure 12 below). | Next, we execute the process “Transient Process with Hibernate Initially”. The process is identical to the one in Figure 6 above except for the “Hibernate Initially” flag which is set to true for the activity “Call Second POJO” (Figure 12 below). | ||
[[File:Stardust Knowledge Base Performance Tuning Transient12.png|center| | [[File:Stardust Knowledge Base Performance Tuning Transient12.png|center| | ||
Line 90: | Line 91: | ||
Figure 14: Activity Instance and Process Instance tables after Activity Hibernation<br /><br /> | Figure 14: Activity Instance and Process Instance tables after Activity Hibernation<br /><br /> | ||
− | === Abort Process === | + | ==== Abort Process ==== |
We next execute the process “Transient Process with Self Abort”. The "Generate Exception" activity generates a Runtime Exception. An Event Action is set to abort the process when the exception is raised (Figure 15 below). | We next execute the process “Transient Process with Self Abort”. The "Generate Exception" activity generates a Runtime Exception. An Event Action is set to abort the process when the exception is raised (Figure 15 below). | ||
[[File:Stardust Knowledge Base Performance Tuning Transient15.png|center| | [[File:Stardust Knowledge Base Performance Tuning Transient15.png|center| | ||
Line 105: | Line 106: | ||
== Using Hazelcast as the object provider == | == Using Hazelcast as the object provider == | ||
− | Hazelcast can be used as an in-memory cache and a distributed object provider with Stardust. This | + | Hazelcast can be used as an in-memory cache and a distributed object provider with Stardust. This wouldre quire us to replace our simple ClusterSafeObjectProvider implementation (Figure 3) with the one provided by Hazelcast. Additional information on Hazelcast integration is available [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.deployment/html/applicationserversetup/hazel-spring.html here].The following procedure describes the steps required to achieve this:<br /><br /> |
1. Add the following entries to your carnot.properties | 1. Add the following entries to your carnot.properties | ||
<source lang="text"> | <source lang="text"> | ||
− | Infinity.Engine.Caching = true (not required for transient processes but | + | Infinity.Engine.Caching = true (not required for transient processes but enables cross-transaction entity caching). |
Carnot.Engine.Hazelcast.JcaConnectionFactoryProvider=org.eclipse.stardust.engine.spring.integration | Carnot.Engine.Hazelcast.JcaConnectionFactoryProvider=org.eclipse.stardust.engine.spring.integration | ||
.jca.SpringAppContextHazelcastJcaConnectionFactoryProvider | .jca.SpringAppContextHazelcastJcaConnectionFactoryProvider | ||
Line 118: | Line 119: | ||
hazelcast-ra-2.4-stardust01.jar | hazelcast-ra-2.4-stardust01.jar | ||
− | Add a file called hazelcast-context.xml in the same location as your stardust-spring-context.xml file. The contents of this file are provided in the Hazelcast documentation link mentioned above. | + | 3. Add a file called hazelcast-context.xml in the same location as your stardust-spring-context.xml file. The contents of this file are provided in the Hazelcast documentation link mentioned above.<br /> |
+ | |||
With these settings you are now ready to use Hazelcast with Stardust for execution of transient processes. | With these settings you are now ready to use Hazelcast with Stardust for execution of transient processes. | ||
− | |||
== Tuning and Performance Improvements == | == Tuning and Performance Improvements == | ||
− | In this section we highlight a few techniques to improve the performance of the Stardust engine.More information on tuning and performance optimization is available here. If we do not intend to use the Stardust non-query APIs (like WorkflowService for example) against the transient Process Instance and its related persistents we can choose not to expose the In-Memory storage by setting the | + | In this section we highlight a few techniques to improve the performance of the Stardust engine. More information on tuning and performance optimization is available [https://infinity.sungard.com/documentation/stardust/1.0/topic/org.eclipse.stardust.docs.dev/html/handbooks/operation/tuning/tuning-transient-proc.html here]. If we do not intend to use the Stardust non-query APIs (like WorkflowService for example) against the transient Process Instance and its related persistents we can choose not to expose the In-Memory storage by setting the following property in carnot.properties |
<source lang="text"> | <source lang="text"> | ||
− | |||
Carnot.Engine.Tuning.TransientProcesses.ExposeInMemStorage = false | Carnot.Engine.Tuning.TransientProcesses.ExposeInMemStorage = false | ||
</source> | </source> | ||
− | We can also configure a fast caching sequence generator to optimize sequence fetching with | + | We can also configure a fast caching sequence generator to optimize sequence fetching with minimal database communication. The following carnot.properties need to be enabled for this: |
<source lang="text"> | <source lang="text"> | ||
Carnot.Engine.Tuning.SequenceBatchSize = 5000 | Carnot.Engine.Tuning.SequenceBatchSize = 5000 |
Latest revision as of 02:55, 11 December 2013
Contents
Introduction
Traditionally, BPM systems have been employed in designing Human-centric or Case Management type process flows. Such processes rely on the Audit Trail capabilities of the underlying BPM solution. While Stardust already provides good support for such processes, it is also capable of supporting integration workflows. Here, it acts as a routing and transformation engine to orchestrate a process flow. In such cases the need for low latency and high throughput takes precedence over AuditTrail data persistence.
Transient Processes provide the following advantages:
- Very little communication with the database.
- All runtime process data is kept in memory for the lifetime of the Process Instance.
- Optimized for low latency
- Significantly higher throughput
Additional information about Transient Processes is available here. All artifacts referred to in the following discussion can be downloaded from here.
Example
We use the following example to start off our discussion on Transient Processes (see Figure 1 below).
Figure 1: Example Transient Process
As shown above, the process consists of a couple of activities each of which invokes a POJO with a couple of In/Out parameters. Ordinarily this data would be persisted to the AuditTrail during process execution and the process history would display the execution sequence if one were to view this information in the Portal. To mark the process as transient we select the corresponding property in the Process Properties dialog box (see Figure 2 below).
Figure 2: Process Properties for Transient Process
To ensure that this property is used by the Stardust engine at runtime we need to set the “SupportTransientProcesses” property in the carnot.properties file. We also need to ensure that the database we use supports sequences (such as Oracle). For this discussion we will use MySQL as our AuditTrail database. To enable support for sequences in MySQL we are required to do the following:
Create a compatible Stardust AuditTrail schema
Create a MySQL database and run the sysconsole command as follows to create an AuditTrail that supports sequences:
sysconsole -v -r com.mysql.jdbc.Driver -dbschema <schema_name> -t mysql_seq -l jdbc:mysql://<servername>:<port>/<dbname> -d <username>-s <password> -p sysop createschema
Note that the dbtype option is set to "mysql_seq". Additional information on creating an AuditTrail on MySQL is available here. To learn more about the sysconsole command please refer to the Stardust documentation.
Set appropriate database properties in carnot.properties
Add/Edit the following entries to carnot.properties:
AuditTrail.Type = MYSQL_SEQ AuditTrail.DriverClass = com.mysql.jdbc.Driver AuditTrail.URL = jdbc:mysql://<servername>:<port>/<dbname> AuditTrail.Schema = <dbname> AuditTrail.User = <username> AuditTrail.Password = <password>
The URL, Schema, User and Password entries should match the values provided in the sysconsole command above.
Provide an implementation of a ClusterSafeObjectProviderHolder
This is required by the Stardust engine to share objects internally. A simple implementation is provided below:
Figure 3: Custom ClusterSafeObjectProvider implementation
Add the following line to your carnot.properties entry so that the Stardust engine can use your implementation at runtime:
Carnot.Engine.Tuning.SupportTransientProcesses.ClusterSafeObjectProvider = com.test.CustomObjectProvider
We now have a basic setup in place to support Transient Processes. Let us go ahead and deploy the process model described above. Since the process has a manual trigger we execute it from the Portal a few times and verify from the logs that the process has executed. Sample output from a process run is shown below:
Figure 4: Sample log output for a process run transiently
The log file indicates that Process Instance 101 was executed successfully and Activity Instances 101 and 102 were completed in this process context. However we cannot locate this process when we search for it in the Portal (Figure 5 below) or query the AuditTrail process execution tables (activity_instance, process_instance etc.).
Figure 5: Process Information is not available in the Portal
AuditTrail Persistence Scenarios
In this section we identify scenarios that cause the Stardust engine to switch from “Transient” to “Immediate” persistence indicating that process data will be written to the AuditTrail. Note that we do not discuss all such scenarios. The reader is encouraged to refer to the documentation and explore further with this background information.
Manual Activity
We start off by adding a manual activity to the process flow and study the difference in behavior
Figure 6: Transient Process with Manual Activity
The log file output reveals that the Process Instance did execute until the manual activity at which point it switched the persistence mode from “Transient” to “Immediate” (Figure 7 below).
Figure 7: Server log indicating switch from Transient to Immediate persistence for Manual Activity
The associated tables in the AuditTrail are also populated with data pertaining to this Process Instances and the activities executed thus far (the first application activity and the manual activity following it).
Figure 8: Activity_Instance and Process_Instance tables after switching to Immediate Persistence
Runtime Exception
We execute another process that throws a Runtime Exception (process “Transient Process with Runtime Exception” in attached model) (Figure 9 below).Figure 9: Transient Process that throws a Runtime Exception
Figure 10: Server log indicating switch from Transient to Immediate persistence for Runtime Exception
The associated tables in the AuditTrail are also populated with data pertaining to this Process Instance and the activities executed thus far (the three application activities).
Figure 11: Activity Instance and Process Instance tables after Runtime Exception occurs
Hibernate Initially
Next, we execute the process “Transient Process with Hibernate Initially”. The process is identical to the one in Figure 6 above except for the “Hibernate Initially” flag which is set to true for the activity “Call Second POJO” (Figure 12 below).
Figure 12: The “Hibernate Initially” flag is set to true for Activity “Call Second POJO”
As expected the log file output reveals that the persistence mode has changed to “Immediate” (Figure 13 below).
Figure 13: Server log indicating switch from Transient to Immediate persistence for Hibernated activity
The associated tables in the AuditTrail are also populated with data pertaining to this Process Instance and the activities executed within it (the two application activities).
Figure 14: Activity Instance and Process Instance tables after Activity Hibernation
Abort Process
We next execute the process “Transient Process with Self Abort”. The "Generate Exception" activity generates a Runtime Exception. An Event Action is set to abort the process when the exception is raised (Figure 15 below).
Figure 15: Event Action set to Abort Process on Activity Generate Exception
The log file indicates that the persistence mode has changed to “Immediate” after the occurrence of the Runtime Exception (Figure 16 below). The Event Action causes the activity and its associated process instance to be aborted.
Figure 16: Server log indicating switch from Transient to Immediate persistence for Process Abortion
The associated tables in the AuditTrail are also populated with data pertaining to this Process Instance and the activities executed within it.
Figure 17: Activity Instance and Process Instance tables after Process Abortion
The attached Process Model contains a few more scenarios. The reader is also encouraged to try these out to gain a better understanding of situations where persistence mode changes can occur. Additional scenarios covering SubProcesses, Splits, JMS, Mail applications etc., are not covered in this article.
Using Hazelcast as the object provider
Hazelcast can be used as an in-memory cache and a distributed object provider with Stardust. This wouldre quire us to replace our simple ClusterSafeObjectProvider implementation (Figure 3) with the one provided by Hazelcast. Additional information on Hazelcast integration is available here.The following procedure describes the steps required to achieve this:
1. Add the following entries to your carnot.properties
Infinity.Engine.Caching = true (not required for transient processes but enables cross-transaction entity caching). Carnot.Engine.Hazelcast.JcaConnectionFactoryProvider=org.eclipse.stardust.engine.spring.integration .jca.SpringAppContextHazelcastJcaConnectionFactoryProvider (comment out the ClusterSafeObjectProvider propertymentioned above if added)
2. Add the following jars to the classpath of your web application:
hazelcast-2.4.jar hazelcast-ra-2.4-stardust01.jar
3. Add a file called hazelcast-context.xml in the same location as your stardust-spring-context.xml file. The contents of this file are provided in the Hazelcast documentation link mentioned above.
With these settings you are now ready to use Hazelcast with Stardust for execution of transient processes.
Tuning and Performance Improvements
In this section we highlight a few techniques to improve the performance of the Stardust engine. More information on tuning and performance optimization is available here. If we do not intend to use the Stardust non-query APIs (like WorkflowService for example) against the transient Process Instance and its related persistents we can choose not to expose the In-Memory storage by setting the following property in carnot.properties
Carnot.Engine.Tuning.TransientProcesses.ExposeInMemStorage = false
We can also configure a fast caching sequence generator to optimize sequence fetching with minimal database communication. The following carnot.properties need to be enabled for this:
Carnot.Engine.Tuning.SequenceBatchSize = 5000 AuditTrail.SequenceGenerator = org.eclipse.stardust.engine.core.persistence.jdbc.sequence.FastCachingSequenceGenerator
The FastCachingSequenceGenerator for MySQL can only be used if the AuditTrail has been generated with database type “mysql_seq” (see documentation above). This option creates the required database sequences and/or function to generate sequences. For MySQL a function called “next_sequence_value_for” is created in the AuditTrail schema. The user needs to ensure that the value added to the last_insert_id (default 1) matches the SequenceBatchSize value in carnot.properties.