I have a system that loads a lot of objects from the database through Hibernate with Ehcache as L2 cache. Since the memory for L2 is limited and it is not set up to expand onto disk, the system has to choose objects to evict from L2 cache to make room for new objects read in from the database. When the data set size is moderate, things appear to work fine. But when they are large, the system throws an error inside the L2 cache code. The stack trace is show below.
Code:
2014-10-23 08:56:15,253 ERROR [http-apr-8443-exec-14] [org.hibernate.LazyInitializationException] - could not initialize proxy - the owning Session was closed
org.hibernate.LazyInitializationException: could not initialize proxy - the owning Session was closed
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:170)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:215)
at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:190)
at org.kablink.teaming.domain.Principal_$$_javassist_131.equals(Principal_$$_javassist_131.java)
at org.hibernate.util.EqualsHelper.equals(EqualsHelper.java:33)
at org.hibernate.type.AbstractType.isEqual(AbstractType.java:131)
at org.hibernate.type.ComponentType.isEqual(ComponentType.java:154)
at org.hibernate.cache.CacheKey.equals(CacheKey.java:79)
at net.sf.ehcache.Element.equals(Element.java:318)
at net.sf.ehcache.store.AbstractPolicy.selectedBasedOnPolicy(AbstractPolicy.java:77)
at net.sf.ehcache.store.MemoryStore.findEvictionCandidate(MemoryStore.java:608)
at net.sf.ehcache.store.MemoryStore.removeElementChosenByEvictionPolicy(MemoryStore.java:578)
at net.sf.ehcache.store.MemoryStore.checkCapacity(MemoryStore.java:561)
at net.sf.ehcache.store.MemoryStore.put(MemoryStore.java:261)
at net.sf.ehcache.store.FrontEndCacheTier.put(FrontEndCacheTier.java:267)
at net.sf.ehcache.Cache.putInternal(Cache.java:1455)
at net.sf.ehcache.Cache.put(Cache.java:1383)
at net.sf.ehcache.Cache.put(Cache.java:1348)
at net.sf.ehcache.hibernate.EhCache.put(EhCache.java:141)
at org.hibernate.cache.ReadWriteCache.put(ReadWriteCache.java:184)
at org.hibernate.cache.impl.bridge.EntityAccessStrategyAdapter.putFromLoad(EntityAccessStrategyAdapter.java:68)
at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:195)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:982)
at org.hibernate.loader.Loader.doQuery(Loader.java:857)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1716)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at org.kablink.teaming.dao.impl.CoreDaoImpl$10.doInHibernate(CoreDaoImpl.java:964)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
at org.kablink.teaming.dao.impl.CoreDaoImpl.loadObjects(CoreDaoImpl.java:945)
at org.kablink.teaming.module.ldap.impl.LdapModuleImpl.updateUsers(LdapModuleImpl.java:7539)
at org.kablink.teaming.module.ldap.impl.LdapModuleImpl$UserCoordinator.record(LdapModuleImpl.java:4621)
at org.kablink.teaming.module.ldap.impl.LdapModuleImpl.syncUsers(LdapModuleImpl.java:5193)
at org.kablink.teaming.module.ldap.impl.LdapModuleImpl.syncAll(LdapModuleImpl.java:3103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.kablink.teaming.search.interceptor.IndexSynchronizationManagerInterceptor.invoke(IndexSynchronizationManagerInterceptor.java:73)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.kablink.teaming.module.interceptor.EventListenerManagerInterceptor.invoke(EventListenerManagerInterceptor.java:64)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.kablink.teaming.util.aopalliance.InvocationStatisticsInterceptor.invoke(InvocationStatisticsInterceptor.java:49)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy23.syncAll(Unknown Source)
at org.kablink.teaming.module.ldap.LdapSyncThread.doLdapSync(LdapSyncThread.java:314)
at org.kablink.teaming.gwt.server.util.GwtLdapHelper.startLdapSync(GwtLdapHelper.java:674)
at org.kablink.teaming.gwt.server.GwtRpcServiceImpl.executeCommand(GwtRpcServiceImpl.java:3706)
at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:545)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:507)
at org.kablink.teaming.gwt.server.GwtRpcController.processCall(GwtRpcController.java:174)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:305)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at org.kablink.teaming.gwt.server.GwtRpcController.handleRequest(GwtRpcController.java:82)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:167)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.kablink.teaming.web.servlet.filter.NoEtagFilter.doFilter(NoEtagFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.kablink.teaming.asmodule.servlet.filter.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:127)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.kablink.teaming.asmodule.servlet.filter.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:127)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.kablink.teaming.web.servlet.filter.DefaultCharacterEncodingFilter.doFilter(DefaultCharacterEncodingFilter.java:60)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
at org.kablink.teaming.tomcat.valve.ZoneContextValve.invoke(ZoneContextValve.java:77)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:680)
at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1852)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:795)
In the stack trace above, the loadObjects refers to the method that loads (potentially large # of) objects from the database using Hibernate Criteria query. When the system sees the need to evict some objects from the L2 cache to make room for the newly loaded objects during put operation on ehcache, the ehcache attempts to perform equality test on candidate objects via equals method. This equals method in turn causes Hibernate to try to initialize the proxy object, and that's when the system throws an error because the Hibernate session associated with the previously cached proxy object is no longer valid, and therefore lazy initialization fails.
So, the question becomes -
1) Is it legal (in terms of Hibernate cache specification) for L2 cache implementation to put proxy objects in the L2 cache, or should it have fully resolved/initialized the proxy object before storing it in the cache?
2) If the answer to the first question is yes (meaning it is allowed to put proxy objects in L2), then how is application layer supposed to deal with situation like this? In other word, is this a bug in the Hibernate/Ehcache or in the application? How can I work around this issue? One work around I thought of is to fully initialize every object that I load from the database. However, I'm not sure if that strategy is going to work, because my understanding is that Hibernate layer is putting the object in the L2 cache BEFORE it passes the object to the application layer. But I assume initializing the object even at that point in the application layer would help, since by the time the object is to be evicted from L2, it would have already been fully initialized. But this whole strategy feels like a hack rather than a solution.
Login
FAQ
Search