These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]

Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 5 posts ] 
Author Message
 Post subject: OutofMemory: SessionImpl cached even after close and flush?
PostPosted: Thu Mar 05, 2009 3:23 am 

Joined: Thu Mar 05, 2009 12:39 am
Posts: 2
Location: Singapore
Hi all,

Hibernate version:3.2.6

Database and Version :DB2 9.1

Application Server and version:Websphere 6.1

I had an Web-based application running on Websphere6.1+EJB2.1+Hibernate 3.2.6. The application is running fine for a first days until a OutOfMemory exception was thrown.

The Memory argument was set to -Xms512m -Xmx1024m.

After tracing the JavaCore dump and Heapdump generated by IBM JVM. The memory leak report (using Ecplise MemoryAnalyzer) showed that org.hibernate.impl.SessionFactoryImpl is holding up 80% of the heap memory, with the size of 700mb.

At first, We thought it is normal since Hibernate will cache the SessionImpl object in "first level" cache for "dirty" state check, the cache shall be clear and gc after session.flush and session.close is called. So the initial conclusion for the outofMemory is due to insufficient memory locate to the JVM to handle the number of concurrent transaction. So we simply increase the heapsize to -Xmx1500m.

Unfortunately the bigger heapsize didn't save our day for too long, more OutOfMemory error were encounter in coming days, so what we can do is restart the websphere server everyday. Bigger heapsize will only delay the outofmemory error, but not prevent it.

The new memory leak report showed that, there are few hundred thousands of SessionImpl object was referenced by the SessionFactoryImpl.

More study was conducted to find out the root cause:
1. Second Level Cache
ehcache was turn on for JBoss JBPM optimization, my understanding is that second level cache will only cache the entity which had manually registed with the cacheManager. So if my application mapping.hbm.xml never setup with any cache algorithm then this should not be the cause.

For testing purpose, we actually turn off second level cache but the problem still persist.

2. use_query_cache is set to false
as suggested in a "Unassigned" hibernate jira issue HHH-3195 (http://opensource.atlassian.com/projects/hibernate/browse/HHH-3195). we turned the query cache to false since our application didn't actually utilize query cache.

3. session.setCacheMode(CacheMode.IGNORE)
to avoid caching we change the sessionFactory to return a session with cachemode ignore. session.setCacheMode(CacheMode.IGNORE). but it doesn't solve the problem as well.

4. First Level Cache
Since Hibernate's first level cache is invalidated only when session is closed, this means that first-level cache can be used through several transaction.
in our case, we are using EJB and CMT transaction managed by Websphere application server, so our application code doesn't need to call session.close and session.flush ourself. To play safe and make sure session was closed, I recompiled the Hibernate3.jar and add in some print.out in session.close and session.flush. So i am sure that the CMT transaction had invoked close and flush had been called after every transaction commited. So the session object should be invalidate and GC!

We implemented very similar project with Spring and Hibernate and doesn't have this problem. So i suspect the problem can be due to the different transaction handling approach between Spring and EJB2.1. but i cannot figured out where.

my question is, since the EJB CMT Transaction had invoke session.flush and session.close, and why is that the SessionImpl are still cached by SessionFactory?

Following is my Hibernate config setup.

<!-- DB2 DataSource properties (begin) -->
<property name="hibernate.connection.driver_class">com.ibm.db2.jcc.DB2Driver</property>
<property name="hibernate.connection.datasource">PoolName</property>
<property name="hibernate.connection.pool_size">50</property>
<property name="dialect">org.hibernate.dialect.DB2Dialect</property>
<!-- DB2 DataSource properties (end) -->

<!-- logging properties (begin) -->
<property name="show_sql">false</property>
<property name="hibernate.format_sql">false</property>
<property name="hibernate.use_sql_comments">false</property>
<!-- logging properties (end) -->

<!-- Batch processing properties (begin) -->
<property name="default_batch_fetch_size">20</property>
<property name="hibernate.jdbc.batch_size">20</property>
<property name="cache.use_second_level_cache">true</property>
<!-- Batch processing properties (end) -->

<!-- Enable Hibernate's automatic session context management -->
<property name="current_session_context_class">thread</property>

<!-- Cache provider (begin) -->
<property name="hibernate.cache.provider_class">org.hibernate.cache.EhCacheProvider</property>
<property name="hibernate.cache.provider_configuration_file_resource_path">/ehcache.xml</property>
<property name="hibernate.cache.use_query_cache">false</property>
<!-- Cache provider (end) -->

<!-- CMT transaction properties (begin) -->
<property name="hibernate.transaction.factory_class">org.hibernate.transaction.CMTTransactionFactory</property>
<property name="hibernate.transaction.manager_lookup_class">org.hibernate.transaction.WebSphereExtendedJTATransactionLookup</property>
<!-- CMT transaction properties (end) -->

Other JVM Arguement:
-noclassgc -Xgcpolicy:gencon -Xms512m -Xmx1500m -Xmns256m -Xmnx1024m -Xdisableexplicitgc -Xmxcl30000 -Xlp64k

Your help is much appreciated.

 Post subject: Remedy!
PostPosted: Fri Mar 06, 2009 12:19 pm 

Joined: Thu Mar 05, 2009 12:39 am
Posts: 2
Location: Singapore
finally i spotted the root caused of the problem. It is an integration problem between the earlier release of Hibernate with Websphere.

Managed to digged out some related Hibernate Jira issue.

According to one of the Jira
Method JTASessionContext.CleanupSynch.afterCompletion() is called as expected but code "context.currentSessionMap.remove( txn );" does not remove session from Map because of key's hashcode has changed. This is due to fact that com.ibm.websphere.jtaextensions.ExtendedJTATransaction.hashCode is actually ID of underlaying transaction. But if it comes to the afterCompletion method in CleanupSynch the underlaying transaction is already closed. Closed transaction has ID 0 (default value) and it is different from ID under which the Hibernate session was previously inserted into Map.

Related Jira

Thanks and hope this might help other victim. :)


 Post subject: Re: OutofMemory: SessionImpl cached even after close and flush?
PostPosted: Tue Jun 29, 2010 3:29 am 

Joined: Tue Jun 29, 2010 3:24 am
Posts: 1
Hello All,
I'm facing the same problem with JBoss AS 4.2.3 GA.

JBoss is using following libraries of Hibernate

hibernate3.jar : 3.2.4SP1
hibernate-entitymanager.jar : 3.2.1 GA
hibernate-annotations.jar : 3.2.1GA

Is the problem specified in the JTA library of Websphere too there in JBoss ?

 Post subject: Re: OutofMemory: SessionImpl cached even after close and flush?
PostPosted: Mon Jul 19, 2010 12:38 pm 

Joined: Mon Jul 19, 2010 12:29 pm
Posts: 1
We're having an uncannily similar situation, including 80% of memory! Using MAT to analyze a dump that was generated after a forced full GC, we're seeing 535M of 663M total (80.6%) allocated to org.hibernate.impl.SessionFactoryImpl.

Following the links here, it seems like there was a fix in hibernate 3.3.0 and perhaps something was done on the WebSphere side? We're using Hibernate 3.3.1 GA in Opentaps 1.4-preview (which uses an embedded Tomcat 5.5 container). Is it possible there needs to be a fix within Opentaps' hibernate integration to resolve this issue?

In terms of transaction identification, it does seem like the relevant method (getTransactionIdentifier) is ever called.

The leak rate is about 50kb per request! Any help (even the slightest tip or thought) would be hugely appreciated.

- Lon

 Post subject: Re: OutofMemory: SessionImpl cached even after close and flush?
PostPosted: Wed Aug 04, 2010 9:22 am 

Joined: Wed Aug 04, 2010 9:08 am
Posts: 1
I've experienced a very similar problem, having 3.8GB over 4GB of available RAM allocated for a lot of SessionFactoryImpl.
My enviroment is the following:
- JBoss 4.2.2
- Hibernate 3.2.7
- JBPM 3.1.4
After a long inspection with MAT, i found that the problem was related to a JSF bean containing an instance of JbpmConfiguration used to manage JBPM workflows; the problem resides in the fact that there was no call to the method close() in the code, and for some reason this implies that the SessionFactoryImpl associated to it keeps living. I made on override of the method finalize of my bean, in which I called explicitly the close method on the instance of JbpmConfiguration.
Hope this will be useful, and sorry for my bad english.
Marco A.

Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 5 posts ] 

All times are UTC - 5 hours [ DST ]

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.