-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 37 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: Anmsoft : hibernate 3.1+DB2 7.2 - Memory leak after quries.
PostPosted: Tue Jan 31, 2006 3:59 am 
Newbie

Joined: Mon Jan 30, 2006 11:28 am
Posts: 13
Location: Mumbai India
Hi,

Problem : After performing query, closing the session, closing the session factory, memory is not released.
I am using lazy collections (3.1.2 has this as a default one for collections).

After performing query -> With lazy collections.
1) (Map) Session.persistenceContext -> Contains the reference of the returned results.
2) On session.close(), it cllears all the maps in the cleanup() method which is in the final clause of close method. So hibernate clearing all the session data on closing. Still memory is not reclaimed. The proxy objects have reference to session and session->persisitence(Map)->proxy. Is this a case of circular referenec. I dont think so?.

After Performing query -> Without lazy collections.
This time i changed the .hbm.xml configuration file so that lazy="false".
1) Perfomed query Memory increased by 5MB. Results have actual pojo object not proxy. This is a simple case of filling a List with some objects and after the list is out of scope it should be gced.
2) Closed the session. Memory did not come down. I added System.gc() too after closing the session to requeset the gc.
3) Closed the session factory. Memory still did not come down.


Is there any other place, some listener etc, where there are still the references of the pojos returned in the query.

I believe in solving problems myself, but i failed this time. I may be wrong in my testing.



Code:
public static void testHibernateMemory() {
        Runtime runtime = Runtime.getRuntime();
        System.err.println("Before Session factory");
        System.err.println("MAX [" + runtime.maxMemory() + "] Total ["
                + runtime.totalMemory() + "] Free [" + runtime.freeMemory()
                + "]");
        SessionFactory sessionFactory = buildSessionFactory();
       
        Session session = sessionFactory.openSession();
       
        Transaction transaction = session.beginTransaction();
        System.err.println("After Session factory");
        System.err.println("MAX [" + runtime.maxMemory() + "] Total ["
                + runtime.totalMemory() + "] Free [" + runtime.freeMemory()
                + "]");
       
        System.err.println("Thread :" + Thread.currentThread().getName());
        try {
            System.err.println("-- Before Query  ---------------");
            List results = findByExampleWithRange(
                    new Ismmdm(), 0, 5000, new Integer[1],session);
            // List results = findByExampleWithRange(new Ismpot(), 0, 50, null);
            System.err.println("Total records [" + results.size() + "]");
            System.err.println("-- After Query  ---------------");
           
        } catch (DAOException e) {
            e.printStackTrace();
        }

        SessionImpl impl = null;
       
        transaction.commit();
       
        System.err.println("-- After Transaction  commit  ------");
       
        session.close();
        session = null;
       
        System.err.println("-- After Session Close  ------");
        sessionFactory.close();
        sessionFactory = null;
        System.gc();
        System.err.println("-- After Session Factory Close  ---------------");
        System.err.println("MAX [" + runtime.maxMemory() + "] Total ["
                + runtime.totalMemory() + "] Free [" + runtime.freeMemory()
                + "]");
       
    }


Details about my environment is below.




I am very old user of hibernate. We are successfully using hibernate 2.x version in one of our applications.

Now we are using hibernate 3.x (3.1.2).
------- hibernate.cfg.xml -
<property name="dialect">
org.hibernate.dialect.DB2Dialect
</property>

<property name="hibernate.connection.release_mode">
auto
</property>
<property name="hibernate.current_session_context_class"> thread
</property>

1) Using session per request pattern with ThreadLocalSessionContext. Everything works well functionality wise.
2) Session factory is building code is taken from HibernateUtils.java from CaveatEmptor sample application.
3) Session is created like a caveat emptor application with same web filter.

4) Transaction is committed and session is closed from web filter. Which automatically unbinds the session from ThreadLocal.

5) SessionFactory instance is closed and set to null in Application ContextListener.

I have gone through all the postings related the memory leaks and up to date with all the progress. Aware of all the release notes, eg upgrade cglib to fix memory leak etc.

Even now, i m facing memory leak problem. It grows and end up with out of memory on WAS 5.1. Same is the case on jboss 4.0.2. I have removed the hibernate sar from deploy directoy so that the classes are always loaded from my web application.

Please some one help me.

Help me. Help me.
I think i have the similar problem quite close like you. I have the huge ecomm application where thousands of query perfomed. And after each request, memory grows continously.
Typically, i would expect the memory to increase and them come down once request is processed.
I configured the web application HTTP Thread pool with 10 min and 25 max threads. Pool thread never expires. This helped me to make the application run for longer time but not forever.

Which made me to conclude that processor thread are not releasing memory. And memory is leaked when the thread is dead. (Thread instances not garbase collected).

I am using following jars in my WEB-INF/lib folder :
commons-anm-1.0.0.jar
ism-dist.jar
jstl.jar
lucene-1.4.3.jar
myfaces-extensions.jar
myfaces-impl.jar
myfaces-jsf-api.jar
standard.jar
struts.jar
commons-beanutils.jar
commons-codec-1.2.jar
commons-dbcp-1.1.jar
commons-digester.jar
commons-fileupload.jar
commons-httpclient.jar
commons-lang-1.0.1.jar
commons-pool-1.1.jar
commons-validator.jar
antlr-2.7.6rc1.jar
asm-attrs.jar
asm.jar
cglib-2.1.3.jar
commons-collections-2.1.1.jar
commons-logging-1.0.4.jar
concurrent-1.3.2.jar
dom4j-1.6.1.jar
ehcache-1.1.jar
oscache-2.1.jar
hibernate3.jar (hibernate 3.1.2)

I m ready to produce any other data if needed.

Regards,
-Chandra
_________________
-Chandra Singh (Anmsoft)

Post rating: This posting has not been rated so far.


Need help with Hibernate? Read this first:
http://www.hibernate.org/ForumMailingli ... AskForHelp

Hibernate version:

Mapping documents:

Code between sessionFactory.openSession() and session.close():

Full stack trace of any exception that occurs:

Name and version of the database you are using:

The generated SQL (show_sql=true):

Debug level Hibernate log excerpt:

Code:

_________________
-Chandra Singh (Anmsoft)


Last edited by chndu6ue on Thu Feb 02, 2006 2:35 am, edited 1 time in total.

Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 4:24 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
have you chekked that *your* code doesn't hold on to the sessions/sessionfactory ?

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 4:32 am 
Newbie

Joined: Thu Jan 26, 2006 2:13 pm
Posts: 9
I have quite a similar problem with hibernate queries that won't free themselves.

(For nobody answered in my post, I'll give it a try here.)

I profiled and memory debugged my case and it turned out, that there are too many antlr.CommonAST and org.hibernate.hql.ast.SqlNode, .LiteralNode, .IdentNode objects. They increase a hundert percent everytime I do my queries.

Does anyone have the same problem? No one ever had a memory leak in relation with hibernate or some logger?


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 4:39 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
no - all memory issues have always been traced back to:

a) not closing session/sessionfactory
b) closing session/sessionfactory but still keep reference to them (or their objects)
c) putting to many objects into the session


your issue seem to be that you are not using parameters for your hql, but instead are building a new hql query string every time...inefficient, but not a memory leak.

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 4:52 am 
Newbie

Joined: Thu Jan 26, 2006 2:13 pm
Posts: 9
Sure it isn't the best way to do it... but why are the queries stored?

They are big, of course, but that should not be a problem. The parameters are not possible, because there are 1000 ids to fill in. (At least, I see no way to do that)

But what do you recommend in my case? Which way is efficient enough and gets by without those huge hql-queries?

Thanks for an answer!


By the way: Why are Danish, when you live in Neuchâtel?


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 4:56 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
the queries are stored so if you execute the same query twice it will reuse the AST tree - avoiding overhead.

it sounds like you have something like "x in ( 234,242,45,2,3,43..)" in your queries which in it self is pretty bad...isn't a join feasible ?

in anycase, the queries will be released after some time since the cache is not unlimited. I think it is something like the last 128 queries we hold on to.

p.s. and i'm a Dane that has moved to Neuchatel to get a better view of the alps.

/max

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject: Re: Memory not released after quries (session closed).
PostPosted: Tue Jan 31, 2006 4:58 am 
CGLIB Developer
CGLIB Developer

Joined: Thu Aug 28, 2003 1:44 pm
Posts: 1217
Location: Vilnius, Lithuania
chndu6ue wrote:

2) On session.close(), it cllears all the maps in the cleanup() method which is in the final clause of close method. So hibernate clearing all the session data on closing. Still memory is not reclaimed. The proxy objects have reference to session and session->persisitence(Map)->proxy. Is this a case of circular referenec. I dont think so?.


Circular references is not a problem for GC. Try "results = null" before to call GC.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 5:37 am 
Newbie

Joined: Thu Jan 26, 2006 2:13 pm
Posts: 9
That's probably the answer!
The last 128 queries are stored, so that means, if these queries are HUGE, the OutOfMemoryError is near...

My problem is the following:

Let's say I have a table employees (the hello world example in databases =)


employees
id, name, salary, orders


orders are the orders that this employee has worked on


orders
order_id, products


products are the products contained in this order


products
id, price, weight


I hope it is clear how these tables are connected.
Relations are as follows:

employees -> orders 1 to n
orders -> products n to n



Now I would like to give the user the ability to search with some criterias and then display a list with search results. This list of employees can be quite big (>10k entries), but it should also display the total amount of price per employee and weight per employee.

I search with criteria and fetches and everything works fine because I have the lazy collections function turned on. I'm getting back the employee objects I want. (You know, this employee stuff is just a sample)
But when I start to display these entries, all the collections have to be associated and for every order and every product within, objects have to be created. Just for calculating the total price and weight?
This takes way too long and is not even possible when searching for more than 1000 entries in my app.

So I changed the search system to my current one:


1. Search with criterias and fetches on the tables. Works fine and quick. Gets me the objects I desire, without sub-objects.
2. I extract all the IDs of the employees and then go over my view and retrieve the data from there. The view is quite fast and gets me exactly the fields I wish (like totals of price and weight). BUT I have to do queries that are pretty ugly (like you mentioned) with every ID in them.


What way do you recommend me? How would you handle such a situation? I am very eager to hear your ideas or even solutions.

Kind regards

ps: ok, that's nice ;-) I live in Zurich, for that matter


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 5:40 am 
Newbie

Joined: Mon Jan 30, 2006 11:28 am
Posts: 13
Location: Mumbai India
Hi,
Firstly, no reference of session or session factory is held anywhere in my code. Session factory is built, session is opened, query is performed, session is closed, session factory is closed.
Setting results = null; Did not work either.

thanx for the replies and hope that soon there will be a solution or hint if i m doing any wrong.
Regards,
-Chandra Singh

_________________
-Chandra Singh (Anmsoft)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 5:45 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
look into batch and subselect fetching.

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 7:36 am 
Newbie

Joined: Thu Jan 26, 2006 2:13 pm
Posts: 9
Hi Max

Thank you for your answer.
I looked into these two topics in the Hibernate Reference Documentation. It's like getting nearer, but still not reaching the target.

How do I use batch-fetching? I tried it several times, but it makes no sense to me. How will Hibernate know, if I'm going to select multiple or just one of these collections? And doesn't that end up in fetching every collection on the database into objects in my application?

Is it not possible to simply execute a query and display the retrieved data? Do I have to use Named Queries and create a SearchObject? I don't need Objects and Sub-Collections... How can I select, which data is important for me?

How would you do a massive search on the database?

As my brain cells are getting warmer, it seems to me, that the best solution would be querying the database with a search query and store the entries right in a specially created object.

What do you think of that?

regards,
Rolf


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 7:49 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
you are asking so many trivial questions that i don't have time to answer them all...i can recommend reading Hibernate In Action about how to retreive objects efficiently (remember it as chapter 4)

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject: Max -> Please can you give me some hints to solve the iss
PostPosted: Tue Jan 31, 2006 8:46 am 
Newbie

Joined: Mon Jan 30, 2006 11:28 am
Posts: 13
Location: Mumbai India
Max,
Thanx for the replies so far regarding the memory.

I want to know from the hibernate team, are there any other references some where, in the list returned from the query.
One place is the persistenceContext inside session. I vaguely suspect that when the results are made using hibernate advanced fundaes from the results set, there might be reference of the same objects being set in persistence context.

Difficult to understand as code is too much nested in so many tiny methods.
Loader.java
private List doQuery(final SessionImplementor session, final QueryParameters queryParameters,final boolean returnProxies) throws SQLException, HibernateException {
Code which prepare the results and fills the results list.
}

_________________
-Chandra Singh (Anmsoft)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 9:07 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
again, we do not cache the *objects*, the only thing that is strongly held is the last 128 query instances (meaning the AST tree etc. NOT the persistent objects)

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 31, 2006 12:07 pm 
Pro
Pro

Joined: Mon Jan 24, 2005 5:39 am
Posts: 216
Location: Germany
Hi,

we are dealing with mass data too.
We could avoid a lot of problems by using the StatelessSession.
Of course, this way you have to do a lot more manually.

And we too have performance problems because
of the maps in StatelessSession.persistenceContext.
After a large query this map contains a large list
of null entries, where each is set to null again
after each query or scroll operation.
We solved this be implementing our own
StatelessSession class. Not a nice solution
but it works.

_________________
dont forget to rate !


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 37 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.