Stardust/Knowledge Base/Performance Tuning/Optimization

In-Memory Cache

It's possible to turn on an in-memory cache for some entities that are not modified very often such as User, User Relam, Grants for Users, Departments, Partitions and User Groups. This in memory cache uses per default Hazelcast, but the implementation can be exchanged. Please refer to the documentation to read more about the usage of Hazelcast-based In-Memory cache for deployments on application servers. If you want to use this In-Memory cache on Tomcat, you need to add some code and apply following configuration changes to your IPP project. In general the In-Memory cache is turned on with property below in server-side carnot.properties.

Infinity.Engine.Caching = true

SpringHazelcastCacheFactory

Hazelcast connections in a non-managed environment are retrieved via Spring-based local JCA connection factory. The managed connection factory is derived from Hazalcast resource adapter. Connection manager and pooling support is taken from Jencks, which is used by IPP in a Spring-managed mode as well to support connection and transaction management.

	<bean lazy-init="true" id="xaHazelcastConnectionFactory"
		class="org.springframework.jca.support.LocalConnectionFactoryBean">
 
		<property name="managedConnectionFactory">
			<bean class="com.hazelcast.jca.ManagedConnectionFactoryImpl" />
		</property>
 
		<property name="connectionManager">
			<bean class="org.jencks.factory.ConnectionManagerFactoryBean">
				<property name="transactionManager" ref="xaTransactionManager" />
				<property name="transaction" value="local" />
				<property name="poolingSupport" ref="poolingSupport" />
			</bean>
		</property>
	</bean>
 
	<bean id="poolingSupport" class="org.jencks.factory.PoolingSupportFactoryBean">
		<property name="poolMaxSize" value="100" />
		<property name="poolMinSize" value="75" />
		<property name="connectionMaxWaitMilliseconds" value="5000" />
		<property name="connectionMaxIdleMinutes" value="1" />
	</bean>

The glue code between Hazelcast connection factory and IPP is shown below. It’s not part of IPP’s code base yet.

package com.infinity.bpm.rt.integration.cache.hazelcast;
 
import java.util.Arrays;
import java.util.Map;
 
import javax.resource.cci.ConnectionFactory;
 
import net.sf.jsr107cache.Cache;
import net.sf.jsr107cache.CacheException;
 
import org.springframework.context.ApplicationContext;
 
import ag.carnot.base.log.LogManager;
import ag.carnot.base.log.Logger;
import ag.carnot.config.Parameters;
import ag.carnot.workflow.runtime.spring.SpringUtils;
 
public class SpringHazelcastCacheFactory extends HazelcastCacheFactory {
 
	private static final Logger trace = LogManager
			.getLogger(SpringHazelcastCacheFactory.class);
 
	@SuppressWarnings("unchecked")
	public Cache createCache(Map env) throws CacheException {
 
		Parameters params = Parameters.instance();
 
		try {
			String txMode = params.getString(PRP_HAZELCAST_TX_MODE, "rw");
 
			if (!Arrays.asList("none", "w", "rw").contains(txMode)) {
 
				trace.warn("Unsupported Hazelcast TX mode \"" + txMode
						+ "\". Using \"rw\" instead.");
 
				txMode = "rw";
			}
 
			ConnectionFactory hzCf = null;
			if (!"none".equals(txMode)) {
 
				ApplicationContext context = SpringUtils
						.getApplicationContext();
 
				hzCf = (ConnectionFactory) context
						.getBean("xaHazelcastConnectionFactory");
			}
			return new HazelcastCacheAdapter(txMode,
					new TxAwareConnectionFactory(hzCf), env);
		} catch (Exception e) {
			trace.warn("Failed initializing Hazelcast cache.", e);
		}
 
		return null;
	}
 
}

To enable this Spring-based version of the Hazelcast Cache Factory, below property needs to be added to server-side carnot.properties file.

Infinity.Engine.Caching.CacheFactory = com.infinity.bpm.rt.integration.cache.hazelcast.SpringHazelcastCacheFactory

TxAwareConnectionFactory

Since neither Hazelcast’s managed connection nor Spirng’S local connection factory do fully support XA, following code is required to handle a Hazelcast connection like an XA resource, which would be commited or rolled back along with any commit / rollback decisions made by the general XA transaction coordinator. The TxAwareConnectionFactory is implicitly used by the SpringHazelcastCacheFactory.

package com.infinity.bpm.rt.integration.cache.hazelcast;
 
import javax.naming.NamingException;
import javax.naming.Reference;
import javax.resource.ResourceException;
import javax.resource.cci.Connection;
import javax.resource.cci.ConnectionFactory;
import javax.resource.cci.ConnectionSpec;
import javax.resource.cci.RecordFactory;
import javax.resource.cci.ResourceAdapterMetaData;
 
import org.springframework.transaction.support.TransactionSynchronizationAdapter;
import org.springframework.transaction.support.TransactionSynchronizationManager;
 
public class TxAwareConnectionFactory implements ConnectionFactory {
 
	private static final long serialVersionUID = 1L;
 
	private final ConnectionFactory connectionFactory;
 
	public TxAwareConnectionFactory(ConnectionFactory connectionFactory) {
		super();
		this.connectionFactory = connectionFactory;
	}
 
	public Connection getConnection() throws ResourceException {
 
		final Connection connection = connectionFactory.getConnection();
 
		TransactionSynchronizationManager
				.registerSynchronization(new TransactionSynchronizationAdapter() {
 
					@Override
					public void beforeCompletion() {
						try {
							connection.close();
						} catch (ResourceException e) {
							e.printStackTrace();
						}
 
					}
				});
 
		return connection;
	}
 
	public Connection getConnection(ConnectionSpec arg0)
			throws ResourceException {
		return connectionFactory.getConnection(arg0);
	}
 
	public ResourceAdapterMetaData getMetaData() throws ResourceException {
		return connectionFactory.getMetaData();
	}
 
	public RecordFactory getRecordFactory() throws ResourceException {
		return connectionFactory.getRecordFactory();
	}
 
	public Reference getReference() throws NamingException {
		return connectionFactory.getReference();
	}
 
	public void setReference(Reference arg0) {
		connectionFactory.setReference(arg0);
	}
 
}

SpringContextPostProcessor

The cache is created and accessed first during engine bootstrap. At that point the Spring context loaded by ContextLoaderListener might not be available yet. Therefore, below Spring context post processor is required to avoid any such race conditions and to make sure the Spring context with all loaded beans are available when the cache is accessed the first time.

package com.infinity.bpm.rt.integration.cache.hazelcast;
 
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.config.BeanFactoryPostProcessor;
import org.springframework.beans.factory.config.ConfigurableListableBeanFactory;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.context.ConfigurableApplicationContext;
 
import ag.carnot.workflow.runtime.spring.SpringUtils;
 
public class SpringContextPostProcessor implements BeanFactoryPostProcessor,
		ApplicationContextAware {
 
	private ApplicationContext context;
 
	@Override
	public void postProcessBeanFactory(
			ConfigurableListableBeanFactory beanFactory) throws BeansException {
 
		SpringUtils
				.setApplicationContext((ConfigurableApplicationContext) this.context);
	}
 
	@Override
	public void setApplicationContext(ApplicationContext context)
			throws BeansException {
		this.context = context;
	}
 
}

The corresponding Spring configuration looks like that:

<bean name="springContextPostProcessor"
		class="com.infinity.bpm.rt.integration.cache.hazelcast.SpringContextPostProcessor" />

Spring Configuration

Find below the complete Spring configuration required to setup Hazelcast In-Memory cache in Spring-managed environments properly. Any references to other beans match bean names of beans pre-configured for an IPP project based on RAD.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:aop="http://www.springframework.org/schema/aop" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
      http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">
 
	<bean name="springContextPostProcessor"
		class="com.infinity.bpm.rt.integration.cache.hazelcast.SpringContextPostProcessor" />
 
	<bean lazy-init="true" id="xaHazelcastConnectionFactory"
		class="org.springframework.jca.support.LocalConnectionFactoryBean">
 
		<property name="managedConnectionFactory">
			<bean class="com.hazelcast.jca.ManagedConnectionFactoryImpl" />
		</property>
 
		<property name="connectionManager">
			<bean class="org.jencks.factory.ConnectionManagerFactoryBean">
				<property name="transactionManager" ref="xaTransactionManager" />
				<property name="transaction" value="local" />
				<property name="poolingSupport" ref="poolingSupport" />
			</bean>
		</property>
	</bean>
 
	<bean id="poolingSupport" class="org.jencks.factory.PoolingSupportFactoryBean">
		<property name="poolMaxSize" value="100" />
		<property name="poolMinSize" value="75" />
		<property name="connectionMaxWaitMilliseconds" value="5000" />
		<property name="connectionMaxIdleMinutes" value="1" />
	</bean>
 
</beans>

Hazelcast Configuration

Please refer to Hazelcast configuration below. It’s basically a default configuration template defining one IPP specific distributed cache object ipp-2nd-level-cache. The configuration has to be somewhere in the runtime’s classpath.

<hazelcast>
    <group>
        <name>dev</name>
        <password>dev-pass</password>
    </group>
    <network>
        <port auto-increment="true">5701</port>
        <join>
            <multicast enabled="false">
                <multicast-group>224.2.2.3</multicast-group>
                <multicast-port>54327</multicast-port>
            </multicast>
            <tcp-ip enabled="true">
                <interface>192.168.1.2</interface>
            </tcp-ip>
        </join>
        <interfaces enabled="false">
            <interface>10.3.17.*</interface>
        </interfaces>
    </network>
    <executor-service>
        <core-pool-size>16</core-pool-size>
        <max-pool-size>64</max-pool-size>
        <keep-alive-seconds>60</keep-alive-seconds>
    </executor-service>
    <queue name="default">
        <!--
            Maximum size of the queue. When a JVM's local queue size reaches the maximum,
            all put/offer operations will get blocked until the queue size
            of the JVM goes down below the maximum.
            Any integer between 0 and Integer.MAX_VALUE. 0 means
            Integer.MAX_VALUE. Default is 0.
        -->
        <max-size-per-jvm>10000</max-size-per-jvm>
        <!--
            Maximum number of seconds for each item to stay in the queue. Items that are
            not consumed in <time-to-live-seconds> will automatically
            get evicted from the queue.
            Any integer between 0 and Integer.MAX_VALUE. 0 means
            infinite. Default is 0.
        -->
        <time-to-live-seconds>0</time-to-live-seconds>
    </queue>
    <map name="default">
        <!--
            Number of backups. If 1 is set as the backup-count for example,
            then all entries of the map will be copied to another JVM for
            fail-safety. Valid numbers are 0 (no backup), 1, 2, 3.
        -->
        <backup-count>1</backup-count>
        <!--
            Valid values are:
            NONE (no eviction),
            LRU (Least Recently Used),
            LFU (Least Frequiently Used).
            NONE is the default.
        -->
        <eviction-policy>NONE</eviction-policy>
        <!--
            Maximum size of the map. When max size is reached,
            map is evicted based on the policy defined.
            Any integer between 0 and Integer.MAX_VALUE. 0 means
            Integer.MAX_VALUE. Default is 0.
        -->
        <max-size>0</max-size>
        <!--
            When max. size is reached, specified percentage of
            the map will be evicted. Any integer between 0 and 100.
            If 25 is set for example, 25% of the entries will
            get evicted.
        -->
        <eviction-percentage>25</eviction-percentage>
    </map>
 
    <map name="ipp-2nd-level-cache">
        <!--
            Number of backups. If 1 is set as the backup-count for example,
            then all entries of the map will be copied to another JVM for
            fail-safety. Valid numbers are 0 (no backup), 1, 2, 3.
        -->
        <backup-count>0</backup-count>
 
        <!--
            Maximum number of seconds for each entry to stay in the map. Entries that are
            older than <time-to-live-seconds> will get automatically evicted from the map.
            Updates on the entry don't change the eviction time.
            Any integer between 0 and Integer.MAX_VALUE. 0 means infinite. Default is 0.
        -->
        <time-to-live-seconds>120</time-to-live-seconds>
 
        <!--
            Maximum number of seconds for each entry to stay idle in the map. Entries that are
            idle(not touched) for more than <max-idle-seconds> will get
            automatically evicted from the map.
            Entry is touched if get, put or containsKey is called.
            Any integer between 0 and Integer.MAX_VALUE.
            0 means infinite. Default is 0.
        -->
        <max-idle-seconds>60</max-idle-seconds>
 
        <!--
            Valid values are:
            NONE (no eviction),
            LRU (Least Recently Used),
            LFU (Least Frequiently Used).
            NONE is the default.
        -->
        <eviction-policy>LFU</eviction-policy>
 
        <!--
            Maximum size of the map. When max size is reached,
            map is evicted based on the policy defined.
            Any integer between 0 and Integer.MAX_VALUE. 0 means
            Integer.MAX_VALUE. Default is 0.
        -->
        <max-size>5000</max-size>
    </map>
 
</hazelcast>

IPP Configuration

Last but not least some additional properties need to be in place of server-side carnot.properties file. The full set is shown below.

Infinity.Engine.Caching = true
Infinity.Engine.Caching.CacheFactory = com.infinity.bpm.rt.integration.cache.hazelcast.SpringHazelcastCacheFactory
Infinity.Engine.Caching.Hazelcast.TxMode = rw
Infinity.Engine.Caching.Hazelcast.GlobalCacheName = ipp-2nd-level-cache

Further Dependencies

Before you start the Tomcat server make sure the libraries hazelcast-all-1.9.2.3.jar, hazelcast-ra-1.9.2.3.jar and jsr107cache-1.1.jar are in your application’s class path (WEB-INF/lib).

A proper configuration of the In-Memory cache can be verified if below log output will be printed to console or log file after the engine has been completely bootstrapped.

16.11.2011 19:31:09 com.hazelcast.impl.Node
INFO: [dev] 
 
 
Members [1] {
	Member [10.244.146.46:5701] this
}