Configuration:
C3P0 version: 0.9.1.2
Settings:
<property name="hibernate.c3p0.max_size">300</property>
<property name="hibernate.c3p0.min_size">0</property>
<property name="hibernate.c3p0.timeout">5000</property>
<property name="hibernate.c3p0.max_statements">300</property>
<property name="hibernate.c3p0.idle_test_period">300</property>
<property name="hibernate.c3p0.acquire_increment">2</property>
Problem:
Under heavy traffic on our production environment, we noticed a loss of connectivity to our DB from one of the components of our system. When looking at the logs, we saw that c3p0 was reporting deadlocks which we assume is what caused the loss of connectivity.
Has anybody seen this type of problem before?
WARN [27 Jan 2009 17:38:54,052] [Timer-2] com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@32b40a9a -- APPA
RENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
WARN [27 Jan 2009 17:41:06,347] [Timer-2] com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@32b40a9a -- APPA
RENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5b57c3b5 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5ef1ceec (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@554818a6 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@704a6607
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@59c6df6f
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@f771e75
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@55709b41
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@53e0f439
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@55c1f534
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@30c954e8
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@71e5645d
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@b205016
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7669da0e
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,Cluster]
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:169)
oracle.jdbc.driver.PhysicalConnection.safelyGetClassForName(PhysicalConnection.java:4822)
oracle.jdbc.driver.PhysicalConnection.addClassMapEntry(PhysicalConnection.java:2851)
oracle.jdbc.driver.PhysicalConnection.addDefaultClassMapEntriesTo(PhysicalConnection.java:2840)
oracle.jdbc.driver.PhysicalConnection.initializeClassMap(PhysicalConnection.java:2544)
oracle.jdbc.driver.PhysicalConnection.ensureClassMapExists(PhysicalConnection.java:2537)
oracle.jdbc.driver.PhysicalConnection.getTypeMap(PhysicalConnection.java:2824)
com.mchange.v2.c3p0.impl.NewPooledConnection.carefulCheckTypeMap(NewPooledConnection.java:173)
com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:111)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:198)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,Cluster]
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,Cluster]
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:169)
oracle.jdbc.driver.DatabaseError.findMessage(DatabaseError.java:547)
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:143)
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,Cluster]
java.util.zip.InflaterInputStream.<init>(InflaterInputStream.java:71)
java.util.zip.ZipFile$1.<init>(ZipFile.java:212)
java.util.zip.ZipFile.getInputStream(ZipFile.java:212)
java.util.zip.ZipFile.getInputStream(ZipFile.java:180)
java.util.jar.JarFile.getInputStream(JarFile.java:383)
sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:689)
sun.misc.Resource.cachedInputStream(Resource.java:59)
sun.misc.Resource.getByteBuffer(Resource.java:154)
java.net.URLClassLoader.defineClass(URLClassLoader.java:249)
java.net.URLClassLoader.access$000(URLClassLoader.java:56)
java.net.URLClassLoader$1.run(URLClassLoader.java:195)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:188)
java.lang.ClassLoader.loadClass(ClassLoader.java:306)
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
java.lang.ClassLoader.loadClass(ClassLoader.java:251)
java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:169)
oracle.jdbc.driver.DatabaseError.findMessage(DatabaseError.java:547)
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:143)
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
WARN [27 Jan 2009 17:41:52,291] [Timer-1] com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7768de52 -- APPA
RENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
WARN [27 Jan 2009 17:43:35,960] [Timer-1] com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7768de52 -- APPA
RENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 1
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3458d54c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@341b8b6d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@7c5ebbd4
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2f7d9b16
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5f50d9b5
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@35052af6
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@664797ab
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@66369002
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@828a7bb
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@251bea2c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4bab01bc
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@47ff0473
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@377d3ef2
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@7faf9e53
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@62c39ca2
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@18608256
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@680ce89a
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@277b0bd4
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7f459f8b
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,Cluster]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,]
java.util.Arrays.copyOf(Arrays.java:2882)
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
java.lang.StringBuilder.append(StringBuilder.java:119)
java.lang.StackTraceElement.toString(StackTraceElement.java:157)
java.lang.String.valueOf(String.java:2827)
java.lang.StringBuilder.append(StringBuilder.java:115)
java.lang.Throwable.printStackTrace(Throwable.java:464)
java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:974)
java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:965)
java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:965)
java.lang.Thread.dispatchUncaughtException(Thread.java:1821)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,]
WARN [27 Jan 2009 17:59:59,505] [DistributedCache-mayhem-dWorker:1] com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool@21aac775 -- an
attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have either interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:56)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
at org.hibernate.loader.Loader.doQuery(Loader.java:673)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
at org.hibernate.loader.Loader.doList(Loader.java:2220)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
at org.hibernate.loader.Loader.list(Loader.java:2099)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1569)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283)
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:305)
at
...
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.load(ReadWriteBackingMap.java:3912)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.loadInternal(ReadWriteBackingMap.java:3630)
at com.tangosol.net.cache.ReadWriteBackingMap.get(ReadWriteBackingMap.java:640)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache.onGetRequest(DistributedCache.CDB:25)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache$GetRequest.run(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:24)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:49)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35)
at java.lang.Thread.run(Thread.java:619)
Login
FAQ
Search