Difference between revisions of "Stardust/Knowledge Base/Deployment/Setting up Stardust in a clustered Tomcat environment"

Latest revision as of 10:36, 25 June 2013

Introduction

When we talk about clustering, we could be referring to clustering of databases (like an Oracle RAC setup), middle tier clustering (with application servers), distributed JMS (multiple brokers and distributed queues with failover) and lastly CMS clustering (like having a clustered Jackrabbit setup). Regardless, the clustering option chosen must make logical sense even if it leaves single points of failure. For example, having multiple Tomcat nodes with the Stardust web app running on each with its own Jackrabbit repository is logically inconsistent since a failover will leave the user without access to the documents he could see through the original node. Likewise, if he logs out and logs in again and is now associated with a different Tomcat instance he would still want to be able to see the documents that were available before.

The setup described here is for a multiple Tomcat, single Jackrabbit (with clustering configuration – as described later), single DB and single MQ environment. It was tested with Apache 2.0, mod_jk 1.2.37, Tomcat 6.0.32 and ActiveMQ 5.2 (the specific version numbers shouldn’t really matter for this discussion however). The reader is  free to extend clustering to the other tiers if the resources required are available.

All the required artifacts for the following discussion can be downloaded from here.

Steps

1. Install Apache.
   
2. Install mod_jk. Please see attachment for an example workers.properties configuration for a three node setup with sticky sessions enabled.
3. Set up your Tomcat instances. Please use version 6.0.32 or lower. There are a few issues using higher versions with Stardust.
4. Use a server.xml similar to the one provided (see attachment). Configure the AuditTrail.DataSource for your environment. Ensure that the jvmRoute tag in the Engine element of server.xml is different in each server.xml and has a corresponding one-one match with an entry in workers.properties.
5. Add the db driver jar to the Tomcat lib directory.
6. Edit your carnot-spring-context.xml file and configure the “carnotXaAuditTrailDataSource” bean to suit your DB configuration. See attachment for an example MySQL configuration. Do not try clustering with Derby. Modify your carnot.properties file to suit your DB configuration (see attachment). There are additional changes to carnot.properties as well. So this can be done in one go later if desired.
7. Set up JMS so that Stardust can use that for forking rather than its inbuilt threading mechanism. This is a safer option for failover and internal engine state is more predictable. See attachments for files required for configuring ActiveMQ with Stardust. These can go into the same directory location as your carnot-spring-context.xml file. In addition, please add amq-embedded-broker-context.xml (see attachment) to your classpath (like WEB-INF/classes for example). Use the tcp transport as shown rather than the vm transport since the latter will boot an in-vm broker. Add the activemq-core and activemq-ra jars to WEB-INF/lib for the version of ActiveMQ you are using.
8. Set up Jackrabbit as a cluster. This requires the use of a DB filestore to save the documents. Each node will maintain its own index on the file system. When a document is created/updated/deleted from one node, the other nodes update their indexes a few seconds (5 second default) later. You can configure the delay. Refer to the attached repository.xml for a MYSQL configuration. Note that the Cluster id tag has to be different for each TC node.
9. There is a repository startup servlet defined in web.xml. This has an init-param called repository-home. Ensure that this location is unique to each TC node. Else the indexes will be overwritten and behavior will be unpredicatable or you will encounter exceptions.
10. Install Hazelcast for distributed model caching. Refer to the documentation here. Basically this requires you to add the hazelcast jar in your WEB-INF/lib directory, add a property to carnot.properties and add the ModelWatcher class (see attachment for compiled class) to your WEB-INF/classes. Note that the documentation states: "Please note: The automatic model cache synchronization does only work for deployments of process model version. It does not work for overwrite and model delete operation. Both, however, are not meant to be applied in productive environment outside a maintenance window."
11. The Stardust daemons need to be started on only one node. If you want daemon failover as well you will need to set up the daemon watchdog. Refer to the documentation here.