-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 9 posts ] 
Author Message
 Post subject: Bizzare hanging threads problem in Hibernate
PostPosted: Thu Dec 04, 2008 4:18 pm 
Beginner
Beginner

Joined: Mon Jul 07, 2008 8:07 pm
Posts: 42
OS: Windows Vista
Java: 1.6.0
Hibernate: 3.2

I'm experiencing some really frustrating and bizzare problems in Hibernate. It seems that I am getting hanging threads when trying to repeatedly access data. I am able to make read requests around 5-10 times, but after that, all of my calls get lost in hanging threads that never end or get fulfilled. I'm using a J2EE server with Spring to handle all of the Hibernate background configuration for this. The only solution I have come up with so far is to restart the server.

I have reason to believe that the problem is occurring in the second-level cache, but I can't prove it directly. When I manually shut down the server in the middle of a call (that never completes), I see this exception is thrown:

Code:
java.lang.NullPointerException
   at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:788)
   at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
   at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
   at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
   at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
   at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
   at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
   at org.hibernate.loader.Loader.doQuery(Loader.java:673)
   at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
   at org.hibernate.loader.Loader.doList(Loader.java:2213)
   at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
   at org.hibernate.loader.Loader.list(Loader.java:2099)
   at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94)
   at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1569)
   at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283)
   at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:305)
   at com.tkassembled.database.UserDAOImpl.getUserById(UserDAOImpl.java:67)
   at com.tkassembled.service.HibernateServiceImpl.getUserById(HibernateServiceImpl.java:30)
   at com.tkassembled.RemotingHandler.getUserById(RemotingHandler.java:58)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:217)
   at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:123)
   at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:157)
   at org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java:409)
   at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:143)
   at org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived(RTMPMinaIoHandler.java:119)
   at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:570)
   at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
   at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
   at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)
   at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:220)
   at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:264)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)


Other than that, it's like shooting in the dark.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Dec 05, 2008 4:18 pm 
Beginner
Beginner

Joined: Thu Oct 02, 2008 10:15 am
Posts: 21
If you take thread dumps on your server you can find out on what exactly your threads are waiting for. And then use thread analyzer to look at these dumps.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Dec 05, 2008 4:35 pm 
Beginner
Beginner

Joined: Mon Jul 07, 2008 8:07 pm
Posts: 42
You'll have to forgive me, I'm new to thread dumps. How should I do this and what exactly is it? Can I monitor this in Eclipse through the debug viewer, or no?


Top
 Profile  
 
 Post subject:
PostPosted: Fri Dec 05, 2008 8:01 pm 
Beginner
Beginner

Joined: Mon Jul 07, 2008 8:07 pm
Posts: 42
Also, I just found a really weird error happening deep in the inner workings of the server in a DaemonThread:
Exception in thread "Thread-42" java.lang.NullPointerException
at com.mysql.jdbc.Statement$1.run(Statement.java:102)

This hasn't happened before. I'll look up thread dumps.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Dec 08, 2008 3:26 pm 
Beginner
Beginner

Joined: Thu Oct 02, 2008 10:15 am
Posts: 21
Thread dumps can be taken on a J2EE server using the utilities provider by the vendor .e.g we took thread dumps which provides the Thread view and heap dumps which provide the Memory[heap] view of the runtime enviroment. You need to look for something similar so that you can watch your server real-time. It seems you are using red5 with which I have never worked with but if you look at the documentation you should be able to find something.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Dec 12, 2008 6:14 pm 
Beginner
Beginner

Joined: Mon Jul 07, 2008 8:07 pm
Posts: 42
As far as I know, my server isn't yet capable of thread dumps :(
However, I have another theory as to why threads keep getting stuck. If I have a thread of execution which opens a Session object by SessionFactory.openSession(), does its transaction/criteria lookup, then explicitly closes the session with Session.close(), and then the same thread looks to do another operation, could that cause it?

EG: Let's say I have a validation service that validates which user is asking for what from the server. When the user's call is issued, Hibernate does a check to see if the user is valid against the DB. This is achieved by the following code:
Code:
Session session = this.getSessionFactory().openSession();
Criteria criteria = session.createCriteria(MyUser.class);
criteria.add(Restrictions.eq("username", user.getUsername());
criteria.add(Restrictions.eq("password", user.getPassword());
criteria.uniqueResult();

session.close();


If the user is validated, the thread continues on, otherwise it returns. When it continues, it does another similar select from the database, perhaps for a different type of object, but still. At the end of THAT transaction, the session is again closed with Session.close().

Would this scenario freeze up the connection pool and disallow sessions from being opened? I'm going to try and investigate this, in the meantime.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Dec 15, 2008 3:26 pm 
Beginner
Beginner

Joined: Thu Oct 02, 2008 10:15 am
Posts: 21
What you have just mentioned should not cause any problems. You must have verified this, but see if there are any bottlenecks, like connection pool or other database resources, on which the threads could be sitting/waiting and not doing anything becuase that resource is already in use.
Another way to verify this is to look at the application logs.If not already there write timings at each layer.[Like starttime and endtime]. You must be having some UUID for each request.If not add one, this would help to see each request uniqely in the logs. This way you will know which layer is taking most time and at what step. Thereafter you can further drill it down to the finest possible steps.
I hope this helps.Let me know.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 30, 2008 6:16 pm 
Beginner
Beginner

Joined: Mon Jul 07, 2008 8:07 pm
Posts: 42
Should I be using C3P0?

http://www.hibernate.org/214.html


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 30, 2008 6:21 pm 
Newbie

Joined: Fri Nov 21, 2008 3:35 pm
Posts: 12
Try C3P0 and see if that helps at all.


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 9 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.