hibernate + ehcache distributed caching woes

j822 · **Joined:** Tue Apr 24, 2007 11:57 pm **Posts:** 3

We're trying to setup caching for our webapp using Spring + Hibernate + Ehcache. Everything works great and has been working great... that is until we enable the distributed component of ehcache.
When everything is working, our system has cache regions configured as follows:

Code:

 <cache name="com.foo.model.User"
        maxElementsInMemory="10000"
        eternal="false"
        timeToIdleSeconds="3600"
        timeToLiveSeconds="3600"
        overflowToDisk="false">
   <!-- placeholder for distributed caching -->
 </cache>

When we change it to this, we start to see problems:

Code:

 <cache name="com.foo.model.User"
        maxElementsInMemory="10000"
        eternal="false"
        timeToIdleSeconds="3600"
        timeToLiveSeconds="3600"
        overflowToDisk="false">
   <cacheEventListenerFactory class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"/>
 </cache>

The problem happens when Box #1 caches a piece of information and distributes it to box #2. A request hits box #2 requesting this now-cached piece of data. Box #2 goes to pull back this piece of data from the cache. Consistently a TypeMismatchException occurs (see stack trace below). This error does not happen on all requests. Sometimes a load test has to go through the a few requests before the problem occurs.

Code:

[vpcs] ERROR  23 Feb 2007 14:46:23,649 [TP-Processor3] |     org.codehaus.xfire.handler.DefaultFaultHandler:35   .invoke                      | Fault occurred!
org.hibernate.TypeMismatchException: Provided id of the wrong type. Expected: class java.lang.Long, got class java.lang.String
       at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:86)
       at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
       at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846)
       at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:557)
       at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:196)
       at org.hibernate.type.TypeFactory.assemble(TypeFactory.java:400)
       at org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:96)
       at org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:82)
       at org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:553)
       at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:508)
       at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:357)
       at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
       at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
       at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
       at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
       at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846)
       at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:557)
       at org.hibernate.type.EntityType.resolve(EntityType.java:379)
       at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:116)
       at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:842)
       at org.hibernate.loader.Loader.doQuery(Loader.java:717)
       at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
       at org.hibernate.loader.Loader.doList(Loader.java:2211)
       at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2127)
       at org.hibernate.loader.Loader.list(Loader.java:2087)
       at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:375)
       at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
       at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
       at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
       at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
       at com.foo.dao.hibernate.MyObjectDaoHibernate.getObject(MyObjectDaoHibernate.java:73)
       at com.foo.dao.hibernate.MyObjectDaoHibernate.getObject(MyObjectDaoHibernate.java:32)
       at com.foo.service.impl.MyObjectManagerImpl.getObjectByGuid(MyObjectManagerImpl.java:121)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:281)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
       at $Proxy23.getObjectByGuid(Unknown Source)
       at com.foo.service.webservice.MyObjectServiceImpl.getObjectByGuid(MyObjectServiceImpl.java:1038)
       at com.foo.service.webservice.MyObjectServiceImpl.LookupLocalInfo(MyObjectServiceImpl.java:530)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:281)
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:199)
       at $Proxy35.LookupExpertPicks(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.codehaus.xfire.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:54)
       at org.codehaus.xfire.service.binding.ServiceInvocationHandler.sendMessage(ServiceInvocationHandler.java:271)
       at org.codehaus.xfire.service.binding.ServiceInvocationHandler$1.run(ServiceInvocationHandler.java:84)
       at org.codehaus.xfire.service.binding.ServiceInvocationHandler.execute(ServiceInvocationHandler.java:132)
       at org.codehaus.xfire.service.binding.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:107)
       at org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)
       at org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)
       at org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)
       at org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:301)
       at org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:130)
       at org.codehaus.xfire.spring.remoting.XFireServletControllerAdapter.handleRequest(XFireServletControllerAdapter.java:67)
       at org.codehaus.xfire.spring.remoting.XFireExporter.handleRequest(XFireExporter.java:48)
       at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:45)
       at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:820)
       at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:755)
       at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:396)
       at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:360)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:39)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:375)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at com.foo.webapp.filter.WaybackFilter.doFilterInternal(WaybackFilter.java:132)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
       at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at com.foo.webapp.filter.FooSecurityContextFilter.doFilterInternal(FooSecurityContextFilter.java:87)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
       at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)
       at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
       at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:106)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)
       at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
       at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)
       at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:173)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at com.foo.webapp.filter.CrossSiteScriptingFilter.doFilterInternal(CrossSiteScriptingFilter.java:92)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
       at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at com.foo.webapp.filter.RemoteAddrFilter.doFilterInternal(RemoteAddrFilter.java:87)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
       at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
       at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
       at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
       at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
       at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
       at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
       at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
       at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
       at java.lang.Thread.run(Thread.java:613)

We have three different environments where we've tested this:

2 Intel Xeon Linux (Red Hat Enterprise Linux ES release 4 (Nahant Update 4)) boxes
2 Intel Xeon Mac Pro (Tiger) boxes
2 AMD Athlon 64 X2 Dual Core Processor 3800+ Linux(CentOS release 4.4 (Final))boxes

Oddly, we never see this problem in the AMD enivornment, but it happens on the other two environments regularly. I'm not sure if it's worth mentioning or not that the two environments where we see the problem are both Xeon based.

Here's the configuration we're using:

Hibernate 3.2.2 GA
ehcache 1.2.4 (We also tried 1.3.0 beta)
spring 2.0.2
Hibernate Annotations 3.2.1.GA
cglib-2.1.3
Java 1.5 - point release 1.5.0_7 (We also tried 1.5.0_11)
Tomcat 5.5.20
Apache + mod_jk

We've tried all sorts of things, including but not limited to testing older versions of the JDK, Spring, and Hibernate. And we tried the newest beta version of ehcache . Yet the problem remains.

Seems like we are using a fairly standard configuration. So I'm surprised this is happening. I'm also surprised I haven't seen other posts about this problem. As such, I'm thinking it's likely a configuration setting that we've botched. Has anyone seen anything like this? Anything at all would be helpful. We're grasping at straws at the moment.

Big thanks.

-j

christian · **Posted:** Wed Apr 25, 2007 1:45 am

AFAIK nobody on our team has had the time to test the relatively new distribution/cluster features of EHCache.

dharmendra.pandey · **Posted:** Wed Apr 25, 2007 2:14 am

Hi j822,

Could you post your implementation of com.foo.dao.hibernate.MyObjectDaoHibernate,
com.foo.service.impl.MyObjectManagerImpl.getObjectByGuid
and post your HBM file
and which Database you are using .That would help us.

j822 · **Joined:** Tue Apr 24, 2007 11:57 pm **Posts:** 3

We are using MySQL 5.0 as my database.

The code in the Hibernate DAO code is essentially this:

Code:

    Query query = getSessionFactory().getCurrentSession().createQuery( "select objectGuid from ObjectGuid objectGuid where objectGuid.guid = :guid" );
    query.setString( "guid", pGuid );
    results = query.list();

and the MyObjectManagerImpl.getObjectByGuid() is just a simple wrapper to the Hibernate DAO method.

There are no HBM files as we are using Hibernate Annotations instead.

Thanks again.

-j

dharmendra.pandey · **Posted:** Wed Apr 25, 2007 11:51 pm

Hi j822,
As Your Log statement says

Provided id of the wrong type. Expected: class
Expected: class java.lang.Long, got class java.lang.String

in your code You are trying to load By objectGuid.guid

query.setString( "guid", pGuid );

I guess that guid is mapped with long in HBM. That could be the problem.

mooritze · **Joined:** Tue Oct 10, 2006 3:23 am **Posts:** 33

Hi,
May be this will help:
I got ehcache distributed to work fine on my setup. However ehcache uses an RMI based mechanism which I didn't quite like. I replaced that with a JGroup based implementation:
ie you do

Code:

<cacheEventListenerFactory class="my..EHCacheReplicatorFactory"
         properties="replicateAsynchronously=true, replicatePuts=false,
 replicateUpdates=false, replicateUpdatesViaCopy=false,
 replicateRemovals=true"

Then you base the EHCacheReplicator on JGroups:

Code:

public class EHCacheReplicator implements CacheEventListener{
   private static final Log log = LogFactory.getLog(EHCacheReplicator.class);
   
   boolean replicatePut=false;

   private JGroupManager jmanager;
...
   public void notifyElementRemoved(Cache cache, Element element) throws CacheException {
      if (log.isTraceEnabled()) log.trace("Sending out rem el"+element);
      ElNotification e=new ElNotification(cache.getName(),element.getObjectKey());
      jmanager.sendNotification(e);
      
   }

and you have a JGroupManager that does:

Code:

public void handleNotification(Serializable arg0) {
      
      if (arg0 instanceof ElNotification) {

         
         Cache c = cacheManager.getCache(e.getCacheName());
         if (c != null) {
//Test fro type of operation here
   
               c.removeQuiet(e.getKey());
            }
         }
      }

I find this way works very well and can use all the nice features of JGroups, in particular ip_multicast

Pierre

j822 · **Joined:** Tue Apr 24, 2007 11:57 pm **Posts:** 3

Thanks. Unfortunately, I had to give up on this yesterday afternoon - ran out of time. I started using hibernate/ehcache in non-distributed mode, and supplemented it with memcached. When I get some time, I'll give this a try and post my findings.