Hello everyone. I am having an issue which I haven't yet been able to come up with a reason for.

Hibernate Version: 3.0.1
MySQL Version: 4.1.12
OS: Windows 2003 Server Standard Edition

What is happening is my system will basically attempt to start a transaction, and never return from it. It doesn't throw an error, it doesn't return null, it just hangs indefinitely. I am using the standard HibernateUtil which was supplied with CaveatEmptor written by Christian@hibernate.org... I would assume most are familiar with this.

I haven't been able to figure out if this is a MySQL, of Java issue. I can temporarily correct the problem by restarting my Java application. It seems to be OK for a while, then it happens again randomly.

I don't believe this is an issue with my application not releasing sessions, because this same code has been in place since August 1st, and has done 40,000+ transactions... nothing has changed. The problem has occured after as little as 150 transactions (since a restart)... so it wouldn't really make sense that it was a coding/resource release issue.

From my best guess, it appears that Hibernate is unable to obtain a connection from the c3po connection pool, or the BEGIN; statement which starts the transaction at the MySQL server is failing and never returning.

Has anyone ever seen this before? Any help, or ideas would be MUCH MUCH appreciated.


Thank you in advance.
Geoff

Anyone?

Do you _HAVE_ to use 3.0.1 ?

There are known issues with early 3.0 releases, I have no clue what they were but would recommend you upgrade to at least 3.0.5 and test your application again and report back.

Thanks for the response. I upgraded the production system to 3.0.5 and the problem continues. This time I was prepared with slightly more information. I launched the app with remote JMX enabled and was prepared with JConsole to check out what all my Threads were stuck doing. The production system died again tonight, I tapped into the JVM and this is what I saw:

Name: pool-2-thread-275
State: WAITING on com.mchange.v2.resourcepool.BasicResourcePool@943dc4
Total blocked: 0 Total waited: 1

Stack trace:
java.lang.Object.wait(Native Method)
com.mchange.v2.resourcepool.BasicResourcePool.awaitAcquire(BasicResourcePool.java:966)
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:208)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:232)
com.mchange.v2.c3p0.PoolBackedDataSource.getConnection(PoolBackedDataSource.java:94)
org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:35)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:298)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:110)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:137)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:49)
org.hibernate.transaction.JDBCTransactionFactory.beginTransaction(JDBCTransactionFactory.java:24)
org.hibernate.jdbc.JDBCContext.beginTransaction(JDBCContext.java:271)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1079)
com.connectfirst.intelliqueue.utils.HibernateUtil.beginTransaction(HibernateUtil.java:162)
com.connectfirst.intelliqueue.dao.GateDnisDAO.<init>(GateDnisDAO.java:21)
com.connectfirst.intelliqueue.telephony.inbound.TelephonyEventEnqueue.processRequest(TelephonyEventEnqueue.java:72)
com.connectfirst.intelliqueue.telephony.TelephonyMessageHandler.run(TelephonyMessageHandler.java:50)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)


There were approx. 30-40 threads which were all stuck in the same exact state. The sytem accepts XML packets through TCP connections, when each packet is recieved a worker thread spawns to process the request. The first thing every thread does is obtain a Session and being a Transaction. So... pretty much this shuts everything down.

The connection is being made to a MySQL server on the local network. Restarting my Java application temporarily fixes the problem... so that leads me to believe it isn't the MySQL server (but I could be wrong).

If anyone has any ideas, I would REALLY REALLY REALLY appreciate it.

Thanks,
Geoff

sounds like there is potentially a problem releasing connections back to the connection pool.

first thing is to make *certain* that you always close the Hibernate sessions you open (which will release the connection it uses back to the connection pool).

I have double checked my code, and haven't been able to find anywhere I have not closed the session... my app has about 100 classes that access the database, so I suppose I could have missed a session.close() somewhere. Not that this is the right way to look at it, but wouldnt the finalize() on the ConnectionManager catch those during GC?

I am wondering if there is possibly an underlying OS/network issue here. I have had the same version of my code in production since August 10. There were absolutely no issues like this until August 28. The traffic between the 10th and the 28th was at least 10x higher than between the 28th and now, yet this error didn't occur at all between 10-28, but it has happened 5 times since the 28th.

I have seen an increase of "Got an error reading communication packets" warning messages on the MySQL server. I was assuming they were connections which were aborted due to inactivity.

Here is a theory... what if the MySQL server is aborting connections due to inactivity (which wouldn't happen as often when the system was under very heavy load). Could it be possible that something is going on when the MySQL server closes connection that is causing the connection pool to eventually become broken?

Hi Geoff,

Were you able to solve this issue?
Even I am facing similar issue which is very strange.
session.beginTransaction() is not returning at all if I am calling from one method, although its returning when I am calling from other method. I am using Sybase ASE as a database.
Hibernate version is 3.2.3
We never faced this issue in last 1.5 years (even in out production environment) but not its getting reproduced every now and then.

Any help on this would be of great help.

Thanks
Vishal