Hibernate version: 3.0.5
c3p0 version:: 0.9.1
Postgres version: 8.1.4

Below is the end of my Tomcat's log file (catalina.out). As you can see my app is apparently deadlocking during c3p0 initialization. This is on Linux. I have to kill Tomcat with "kill -9" to get out of this.

I am using the identical configuration on Windows and it is working fine: same hibernate, same c3p0, same postgres.

I was getting the exact same behavior on c3p0 version 8.3.2, except I was not seeing the APPARENT DEADLOCK message.

What am I doing wrong? Thanks.

-sbq

End of Tomcat's log file
...
- MLog clients using log4j logging.
- Initializing c3p0-0.9.1-pre9 [built 23-September-2006 01:47:24; debug? true; trace: 10]
- Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@9b024057 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@9db53404 [ acquireIncrement -> 1, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> c3p0_connection_test, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 801059, idleConnectionTestPeriod -> 0, initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 120, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@9c6de8bc [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 7f8062, jdbcUrl -> jdbc:postgresql://deadbeef:5432/reldbx, properties -> {user=******, password=******} ], preferredTestQuery -> null, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 8a2023, numHelperThreads -> 3 ]
- com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1e536d6 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

sbq, is that really the end of your log file? an APPARENT DEADLOCK should be followed by a long status string with information about the deadlock. I can't see any obvious way the APPARENT DEADLOCK message could get logged, but not the rest. if there is a full status message, please send that along.

in any case, something strange is going on, as APPARENT DEADLOCKS usually recover, they don't usually freeze up the VM. (alternatively, it could be that something else is freezing first, provoking the APPARENT DEADLOCK as thread pool tasks are unable to complete.)

if there is nothing else logged, please force your JVM to dump its stacktraces (kill -QUIT before you kill -9), so we can try to see on exactly what things are hanging.

thanks!

swaldman, thank you very much for the "kill -QUIT" suggestion. That did indeed print out a stack dump for all threads. This revealed to us that all user threads were blocked trying to authenticate something about sockets. We think this gives us exactly the information we need to work around the problem.

We have an unusual authentication model implemented in this code (which we have subsequently removed) that uses the database. Obviously, during c3p0 initialization the database is not yet open, so answering an authentication question is a problem. Below there is a relevant portion of the stack trace of one thread.

An interesting question is why does this happen on Linux, but not on Windows. We speculate that the Java security model on the two machines must be different, possibly in the area of socket creation.

Here is the stack trace. CayusePolicy.java is our module:

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon prio=1 tid=0x7bcf73a0 nid=0x2a4a waiting for monitor entry [0x788aa000..0x788aaf30]
at com.cayuse.ui.struts.auth.CayusePolicy.implies(CayusePolicy.java:222)
- waiting to lock <0xb0035c68> (a com.cayuse.ui.struts.auth.CayusePolicy)
at java.security.ProtectionDomain.implies(ProtectionDomain.java:195)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:249)
at java.security.AccessController.checkPermission(AccessController.java:427)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at java.lang.SecurityManager.checkConnect(SecurityManager.java:1031)
at java.net.InetAddress.getAllByName0(InetAddress.java:1117)
at java.net.InetAddress.getAllByName0(InetAddress.java:1098)
at java.net.InetAddress.getAllByName(InetAddress.java:1061)
at java.net.InetAddress.getByName(InetAddress.java:958)
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:124)
at java.net.Socket.<init>(Socket.java:179)
at org.postgresql.core.PGStream.<init>(PGStream.java:60)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:77)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:65)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:116)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3.Jdbc3Connection.<init>(Jdbc3Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:369)
at org.postgresql.Driver.connect(Driver.java:245)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)