We were able to solve our application's APPARENT DEADLOCK problems by analyzing stack traces. A good idea is to set up a script to trigger stack trace dumping when APPARENT DEADLOCK appears in the log file.
There's a nasty mechanism where a single long-running Oracle query can block other c3p0-managed connections from preparing statements.
A single long-running query will hold the synchronization lock on the OracleConnection it is using.
If OracleStatements have to be evicted from the c3p0 statement cache a c3p0 background thread will try to call OracleStatement.close. OracleStatement holds a reference to the original statement it was created with and will try to obtain lock on it. This will block, if the long running query is using the same connection.
If there's a lot of churn on the statement cache this will happen more often and once in a while all the c3p0 background threads may be blocked by this mechanism.
C3P0 uses background threads to prepare statements (see http://www.java2s.com/Open-Source/Java- ... e.java.htm
When all the background threads are stuck waiting for access to the single connection no statements will be prepared and all Hibernate operatations will be blocked like so:
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab3b7c4d8> (a com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache)
- locked <0x00002aaab3b7c4d8> (a com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache)
- locked <0x00002aab175582d0> (a com.mchange.v2.c3p0.impl.NewPooledConnection)
- locked <0x00002aaab1b3a1b8> (a com.mchange.v2.c3p0.impl.NewProxyConnection)