Dear All,
I am having an issue in the Ehcache Clustering using RMI Replication. I am using a Oracle 10g application server with EhCache replication. The versions are Hibernate version: 3.2.6.g and Ehcache version: 2.4.1. I am using a manual peer replication. Please find the ehcache config xml.
Code:
<?xml version="1.0" encoding="UTF-8"?>
<!-- UAT Environment Configuration -->
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd" updateCheck="false">
    <!-- Location of persistent caches on disk -->
    <diskStore path="java.io.tmpdir/CCD-CXIS-EhCache" />
   <transactionManagerLookup
       class="net.sf.ehcache.transaction.manager.DefaultTransactionManagerLookup"
       properties="jndiName=java:comp/UserTransaction" propertySeparator=";"/>
     
    <!-- Automatic Peer Discovery Cache Manager -->
   <cacheManagerPeerProviderFactory 
      class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory" 
      properties="peerDiscovery=manual, rmiUrls=//hostname1:40002/containersCacheUAT1"/>
                             
    <!-- Peer Listener -->
   <cacheManagerPeerListenerFactory
      class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
      properties="hostName=hostname2, port=40002, socketTimeoutMillis=2000"/>
 
    <!-- Default Cache Configuration -->
    <defaultCache
        maxElementsInMemory="10000"
        eternal="false"
        overflowToDisk="false"
        timeToIdleSeconds="0"
        timeToLiveSeconds="86400"
        memoryStoreEvictionPolicy="LRU"
        diskPersistent="false"
        diskExpiryThreadIntervalSeconds="120"/>
    <!-- Containers Cache Configuration -->
   <cache 
      name="containersCacheUAT1" 
      maxElementsInMemory="100"
      eternal="false"
      overflowToDisk="false" 
      diskPersistent="false"
      timeToIdleSeconds="0" 
      timeToLiveSeconds="86400"
      memoryStoreEvictionPolicy="LRU" 
      transactionalMode="xa">
      <cacheEventListenerFactory 
         class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"
          properties="replicatePuts=true, replicateUpdates=true, replicateRemovals=true, replicateUpdatesViaCopy=false, replicateAsynchronously=true, asynchronousReplicationIntervalMillis=500"/>
      <bootstrapCacheLoaderFactory 
         class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory" />
   </cache>
</ehcache>
Now the configuration seems to be working fine for around 20 minutes.  After this the cache synchronization stops working. I am not sure why this is happening. I turned on the debug to get the log statements. One of the log statements says the the Multicast Heartbeat thread is disposed. I am not sure why this should happen 
Code:
[CXIS-SI] [29/06/2011 08:17:37] DEBUG [UnDeployerRunnable Thread for CXIS-SI-EAR-1.0.0.4] MulticastKeepaliveHeartbeatReceiver.dispose(98) | dispose called
[CXIS-SI] [29/06/2011 08:17:37] DEBUG [UnDeployerRunnable Thread for CXIS-SI-EAR-1.0.0.4] RMICacheManagerPeerListener.stopRegistry(346) | rmiregistry unexported.
[CXIS-SI] [29/06/2011 08:17:37] DEBUG [UnDeployerRunnable Thread for CXIS-SI-EAR-1.0.0.4] RMICacheManagerPeerListener.dispose(374) | 1 RMICachePeers unbound from registry in RMI listener
[CXIS-SI] [29/06/2011 08:20:07] ERROR [DeployerRunnable Thread for CXIS-SI-EAR-1.0.0.5] MulticastRMICacheManagerPeerProvider.init(98) | Error starting heartbeat. Error was: Address already in use (errno:226)
java.net.BindException: Address already in use (errno:226)
   at java.net.PlainDatagramSocketImpl.bind0(Native Method)
   at java.net.PlainDatagramSocketImpl.bind(PlainDatagramSocketImpl.java:82)
   at java.net.DatagramSocket.bind(DatagramSocket.java:368)
   at java.net.MulticastSocket.<init>(MulticastSocket.java:147)
   at java.net.MulticastSocket.<init>(MulticastSocket.java:112)
   at net.sf.ehcache.distribution.MulticastKeepaliveHeartbeatReceiver.init(MulticastKeepaliveHeartbeatReceiver.java:84)
   at net.sf.ehcache.distribution.MulticastRMICacheManagerPeerProvider.init(MulticastRMICacheManagerPeerProvider.java:95)
   at net.sf.ehcache.CacheManager.init(CacheManager.java:355)
   at net.sf.ehcache.CacheManager.<init>(CacheManager.java:281)
   at org.springframework.cache.ehcache.EhCacheManagerFactoryBean.afterPropertiesSet(EhCacheManagerFactoryBean.java:112)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
   at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
   at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
   at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
   at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
   at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:563)
   at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
   at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
   at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
   at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
   at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
   at com.evermind.server.http.HttpApplication.initDynamic(HttpApplication.java:1140)
   at com.evermind.server.http.HttpApplication.<init>(HttpApplication.java:741)
   at com.evermind.server.ApplicationStateRunning.getHttpApplication(ApplicationStateRunning.java:431)
   at com.evermind.server.Application.getHttpApplication(Application.java:586)
   at com.evermind.server.http.HttpSite$HttpApplicationRunTimeReference.createHttpApplicationFromReference(HttpSite.java:1987)
   at com.evermind.server.http.HttpSite$HttpApplicationRunTimeReference.<init>(HttpSite.java:1906)
   at com.evermind.server.http.HttpSite.addHttpApplication(HttpSite.java:1603)
   at oracle.oc4j.admin.internal.WebApplicationBinder.bindWebApp(WebApplicationBinder.java:302)
   at oracle.oc4j.admin.internal.WebApplicationBinder.bindWebApp(WebApplicationBinder.java:121)
   at oracle.oc4j.admin.internal.WebApplicationBinder.bindWebApp(WebApplicationBinder.java:99)
   at oracle.oc4j.admin.internal.ApplicationDeployer.bindWebApp(ApplicationDeployer.java:550)
   at oracle.oc4j.admin.internal.ApplicationDeployer.doDeploy(ApplicationDeployer.java:205)
   at oracle.oc4j.admin.internal.DeployerBase.execute(DeployerBase.java:93)
   at oracle.oc4j.admin.jmx.server.mbeans.deploy.OC4JDeployerRunnable.doRun(OC4JDeployerRunnable.java:52)
   at oracle.oc4j.admin.jmx.server.mbeans.deploy.DeployerRunnable.run(DeployerRunnable.java:81)
   at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
   at java.lang.Thread.run(Thread.java:595)
[CXIS-SI] [29/06/2011 08:20:07] ERROR [DeployerRunnable Thread for CXIS-SI-EAR-1.0.0.5] ContextLoader.initWebApplicationContext(220) | Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehCacheManager' defined in class path resource [applicationContext-cache.xml]: Invocation of init method failed; nested exception is net.sf.ehcache.CacheException: Address already in use (errno:226)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
   at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
   at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
   at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
   at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
   at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:563)
   at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
   at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
   at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
   at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
   at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
   at com.evermind.server.http.HttpApplication.initDynamic(HttpApplication.java:1140)
   at com.evermind.server.http.HttpApplication.<init>(HttpApplication.java:741)
   at com.evermind.server.ApplicationStateRunning.getHttpApplication(ApplicationStateRunning.java:431)
   at com.evermind.server.Application.getHttpApplication(Application.java:586)
   at com.evermind.server.http.HttpSite$HttpApplicationRunTimeReference.createHttpApplicationFromReference(HttpSite.java:1987)
   at com.evermind.server.http.HttpSite$HttpApplicationRunTimeReference.<init>(HttpSite.java:1906)
   at com.evermind.server.http.HttpSite.addHttpApplication(HttpSite.java:1603)
   at oracle.oc4j.admin.internal.WebApplicationBinder.bindWebApp(WebApplicationBinder.java:302)
   at oracle.oc4j.admin.internal.WebApplicationBinder.bindWebApp(WebApplicationBinder.java:121)
   at oracle.oc4j.admin.internal.WebApplicationBinder.bindWebApp(WebApplicationBinder.java:99)
   at oracle.oc4j.admin.internal.ApplicationDeployer.bindWebApp(ApplicationDeployer.java:550)
   at oracle.oc4j.admin.internal.ApplicationDeployer.doDeploy(ApplicationDeployer.java:205)
   at oracle.oc4j.admin.internal.DeployerBase.execute(DeployerBase.java:93)
   at oracle.oc4j.admin.jmx.server.mbeans.deploy.OC4JDeployerRunnable.doRun(OC4JDeployerRunnable.java:52)
   at oracle.oc4j.admin.jmx.server.mbeans.deploy.DeployerRunnable.run(DeployerRunnable.java:81)
   at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
   at java.lang.Thread.run(Thread.java:595)
Caused by: net.sf.ehcache.CacheException: Address already in use (errno:226)
   at net.sf.ehcache.distribution.MulticastRMICacheManagerPeerProvider.init(MulticastRMICacheManagerPeerProvider.java:99)
   at net.sf.ehcache.CacheManager.init(CacheManager.java:355)
   at net.sf.ehcache.CacheManager.<init>(CacheManager.java:281)
   at org.springframework.cache.ehcache.EhCacheManagerFactoryBean.afterPropertiesSet(EhCacheManagerFactoryBean.java:112)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477)
   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
   ... 29 more
One thing I can try i,s while using the rmi protocol for looking up the server, instead of hostname i can use the ip since rmi might have issues with host names. Othe than that is there anything I am missing?
Thanks,
Sriram