14 - com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2
12 - com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0



Found one Java-level deadlock:
=============================
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2":
  waiting to lock monitor 0x00007fac8d64cdb0 (object 0x00007fac99d41db8, a com.mysql.jdbc.Connection),
  which is held by "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0"
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0":
  waiting to lock monitor 0x0000000042328ed0 (object 0x00007fac99e70828, a com.mysql.jdbc.ResultSet),
  which is held by "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2"

Java stack information for the threads listed above:
===================================================
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2":
   at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3187)
   - waiting to lock <0x00007fac99d41db8> (a com.mysql.jdbc.Connection)
   at com.mysql.jdbc.Field.getStringFromBytes(Field.java:592)
   at com.mysql.jdbc.Field.getName(Field.java:496)
   at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593)
   at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:930)
   - locked <0x00007fac99e70828> (a com.mysql.jdbc.ResultSet)
   at com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4486)
   at com.mysql.jdbc.DatabaseMetaData$IterateBlock.doForAll(DatabaseMetaData.java:79)
   at com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4440)
   at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
   at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
   at com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
   at com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0":
   at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
   - waiting to lock <0x00007fac99e70828> (a com.mysql.jdbc.ResultSet)
   at com.mysql.jdbc.Statement.realClose(Statement.java:1754)
   - locked <0x00007fac99d41db8> (a com.mysql.jdbc.Connection)
   - locked <0x00007fac99e706f0> (a com.mysql.jdbc.Statement)
   at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2128)
   at com.mysql.jdbc.Connection.realClose(Connection.java:4468)
   at com.mysql.jdbc.Connection.close(Connection.java:2100)
   - locked <0x00007fac99d41db8> (a com.mysql.jdbc.Connection)
   at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:474)
   at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:191)
   - locked <0x00007fac99d6a1c0> (a com.mchange.v2.c3p0.impl.NewPooledConnection)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
   at com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Found 1 deadlock.

Heap
PSYoungGen      total 108224K, used 59424K [0x00007facada50000, 0x00007facb79d0000, 0x00007facb7ef0000)
  eden space 58240K, 78% used [0x00007facada50000,0x00007facb06d3f68,0x00007facb1330000)
  from space 49984K, 27% used [0x00007facb4900000,0x00007facb5684108,0x00007facb79d0000)
  to   space 52544K, 0% used [0x00007facb1330000,0x00007facb1330000,0x00007facb4680000)
PSOldGen        total 170496K, used 111397K [0x00007fac990f0000, 0x00007faca3770000, 0x00007facada50000)
  object space 170496K, 65% used [0x00007fac990f0000,0x00007fac9fdb95c8,0x00007faca3770000)
PSPermGen       total 43392K, used 38185K [0x00007fac93cf0000, 0x00007fac96750000, 0x00007fac990f0000)
  object space 43392K, 88% used [0x00007fac93cf0000,0x00007fac9623a4d8,0x00007fac96750000)




**********




2010-12-16 05:30:39
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):

"TP-Processor20" daemon prio=10 tid=0x00007fac8d644800 nid=0x800 in Object.wait() [0x00007fac90d8d000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a55fa8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a55fa8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor19" daemon prio=10 tid=0x00007fac8df4a000 nid=0x7ff runnable [0x00007fac90e8e000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007facaedef550> (a java.io.BufferedInputStream)
   at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
   at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:559)
   at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
   at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor18" daemon prio=10 tid=0x00007fac8c529800 nid=0x7fe in Object.wait() [0x00007fac90f8f000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a49380> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a49380> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor17" daemon prio=10 tid=0x00007fac8daa4800 nid=0x7fd in Object.wait() [0x00007fac91090000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a49718> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a49718> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor16" daemon prio=10 tid=0x00007fac8df82000 nid=0x7fc in Object.wait() [0x00007fac91191000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a49ab0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a49ab0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor15" daemon prio=10 tid=0x00007fac8d640800 nid=0x7fa runnable [0x00007fac91292000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007facaee23830> (a java.io.BufferedInputStream)
   at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
   at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:559)
   at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
   at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor14" daemon prio=10 tid=0x00007fac8c3c7800 nid=0x7f9 in Object.wait() [0x00007fac91393000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a39b38> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a39b38> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor13" daemon prio=10 tid=0x00007fac8c3c6000 nid=0x7f8 in Object.wait() [0x00007fac91494000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a56718> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a56718> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor12" daemon prio=10 tid=0x00007fac8d98c800 nid=0x7f7 in Object.wait() [0x00007fac91595000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a3abe0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a3abe0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor11" daemon prio=10 tid=0x00007fac8d98c000 nid=0x7f6 in Object.wait() [0x00007fac91696000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99a57890> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac99a57890> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor10" daemon prio=10 tid=0x00007fac8d48d000 nid=0x740 in Object.wait() [0x00007fac91797000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997ccd88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997ccd88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor9" daemon prio=10 tid=0x0000000042aa9800 nid=0x73f in Object.wait() [0x00007fac91898000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997ebd08> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997ebd08> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor8" daemon prio=10 tid=0x0000000042a60000 nid=0x73e runnable [0x00007fac91999000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007facaed8bc28> (a java.io.BufferedInputStream)
   at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
   at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:559)
   at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
   at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor7" daemon prio=10 tid=0x0000000042a5d000 nid=0x73d in Object.wait() [0x00007fac91a9a000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997d9178> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997d9178> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor6" daemon prio=10 tid=0x0000000042a5c800 nid=0x73c in Object.wait() [0x00007fac91b9b000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997e15a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997e15a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Monitor" daemon prio=10 tid=0x00007fac8df18800 nid=0x733 in Object.wait() [0x00007fac91c9c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997e1938> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
   at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)
   - locked <0x00007fac997e1938> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor5" daemon prio=10 tid=0x00007fac8df17800 nid=0x732 runnable [0x00007fac91d9d000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007fac997cf2f8> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:307)
   at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:661)
   at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:872)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor4" daemon prio=10 tid=0x0000000042375800 nid=0x731 in Object.wait() [0x00007fac91e9e000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997cd2e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997cd2e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor3" daemon prio=10 tid=0x0000000042373800 nid=0x730 in Object.wait() [0x00007fac91f9f000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997d7970> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997d7970> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor2" daemon prio=10 tid=0x0000000042372800 nid=0x72f in Object.wait() [0x00007fac920a0000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997eb508> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997eb508> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"TP-Processor1" daemon prio=10 tid=0x00000000428a1800 nid=0x72e in Object.wait() [0x00007fac921a1000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac997e22a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
   - locked <0x00007fac997e22a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
   at java.lang.Thread.run(Thread.java:662)

"http-2110-Acceptor-0" daemon prio=10 tid=0x000000004289b000 nid=0x72d runnable [0x00007fac922a2000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007fac9935bcb0> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352)
   at java.lang.Thread.run(Thread.java:662)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007fac8c538000 nid=0x72c waiting on condition [0x00007fac923a3000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
   at java.lang.Thread.run(Thread.java:662)

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon prio=10 tid=0x00000000429c3000 nid=0x72b waiting for monitor entry [0x00007fac924a4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
   at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3187)
   - waiting to lock <0x00007fac99b37a18> (a com.mysql.jdbc.Connection)
   at com.mysql.jdbc.Field.getStringFromBytes(Field.java:592)
   at com.mysql.jdbc.Field.getName(Field.java:496)
   at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593)
   at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:930)
   - locked <0x00007fac99b6b150> (a com.mysql.jdbc.ResultSet)
   at com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4486)
   at com.mysql.jdbc.DatabaseMetaData$IterateBlock.doForAll(DatabaseMetaData.java:79)
   at com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4440)
   at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
   at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
   at com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
   at com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon prio=10 tid=0x00000000423f0000 nid=0x72a in Object.wait() [0x00007fac925a5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac993b9d90> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
   - locked <0x00007fac993b9d90> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon prio=10 tid=0x0000000042cde800 nid=0x729 waiting for monitor entry [0x00007fac926a6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
   at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
   - waiting to lock <0x00007fac99b6b150> (a com.mysql.jdbc.ResultSet)
   at com.mysql.jdbc.Statement.realClose(Statement.java:1754)
   - locked <0x00007fac99b37a18> (a com.mysql.jdbc.Connection)
   - locked <0x00007fac99b6b018> (a com.mysql.jdbc.Statement)
   at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2128)
   at com.mysql.jdbc.Connection.realClose(Connection.java:4468)
   at com.mysql.jdbc.Connection.close(Connection.java:2100)
   - locked <0x00007fac99b37a18> (a com.mysql.jdbc.Connection)
   at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:474)
   at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:191)
   - locked <0x00007fac99b5fe20> (a com.mchange.v2.c3p0.impl.NewPooledConnection)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
   at com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

"Timer-0" daemon prio=10 tid=0x0000000042282800 nid=0x728 in Object.wait() [0x00007fac927a7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac993bb360> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00007fac993bb360> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"GC Daemon" daemon prio=10 tid=0x00007fac8c2a2000 nid=0x727 in Object.wait() [0x00007fac929c7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99245ed0> (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:100)
   - locked <0x00007fac99245ed0> (a sun.misc.GC$LatencyLock)

"Low Memory Detector" daemon prio=10 tid=0x00007fac8c001800 nid=0x725 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00000000420f7800 nid=0x724 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00000000420f4800 nid=0x723 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00000000420f2000 nid=0x722 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00000000420d2000 nid=0x721 in Object.wait() [0x00007fac93203000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99222c08> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00007fac99222c08> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00000000420ca800 nid=0x720 in Object.wait() [0x00007fac93304000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007fac99222bc0> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:485)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   - locked <0x00007fac99222bc0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x000000004206d000 nid=0x71c runnable [0x00007facbbe92000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007fac9987ac98> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
   at org.apache.catalina.startup.Catalina.await(Catalina.java:662)
   at org.apache.catalina.startup.Catalina.start(Catalina.java:614)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

"VM Thread" prio=10 tid=0x00000000420c6800 nid=0x71f runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000042080000 nid=0x71d runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000042082000 nid=0x71e runnable

"VM Periodic Task Thread" prio=10 tid=0x00007fac8c00c800 nid=0x726 waiting on condition

JNI global references: 1368


Found one Java-level deadlock:
=============================
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2":
  waiting to lock monitor 0x00007fac8d64cdb0 (object 0x00007fac99b37a18, a com.mysql.jdbc.Connection),
  which is held by "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0"
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0":
  waiting to lock monitor 0x0000000042328ed0 (object 0x00007fac99b6b150, a com.mysql.jdbc.ResultSet),
  which is held by "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2"

Java stack information for the threads listed above:
===================================================
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2":
   at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3187)
   - waiting to lock <0x00007fac99b37a18> (a com.mysql.jdbc.Connection)
   at com.mysql.jdbc.Field.getStringFromBytes(Field.java:592)
   at com.mysql.jdbc.Field.getName(Field.java:496)
   at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593)
   at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:930)
   - locked <0x00007fac99b6b150> (a com.mysql.jdbc.ResultSet)
   at com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4486)
   at com.mysql.jdbc.DatabaseMetaData$IterateBlock.doForAll(DatabaseMetaData.java:79)
   at com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4440)
   at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
   at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
   at com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
   at com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0":
   at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
   - waiting to lock <0x00007fac99b6b150> (a com.mysql.jdbc.ResultSet)
   at com.mysql.jdbc.Statement.realClose(Statement.java:1754)
   - locked <0x00007fac99b37a18> (a com.mysql.jdbc.Connection)
   - locked <0x00007fac99b6b018> (a com.mysql.jdbc.Statement)
   at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2128)
   at com.mysql.jdbc.Connection.realClose(Connection.java:4468)
   at com.mysql.jdbc.Connection.close(Connection.java:2100)
   - locked <0x00007fac99b37a18> (a com.mysql.jdbc.Connection)
   at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:474)
   at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:191)
   - locked <0x00007fac99b5fe20> (a com.mchange.v2.c3p0.impl.NewPooledConnection)
   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
   at com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
   at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Found 1 deadlock.

Heap
PSYoungGen      total 57024K, used 43662K [0x00007facada50000, 0x00007facb3690000, 0x00007facb7ef0000)
  eden space 22656K, 97% used [0x00007facada50000,0x00007facaefef658,0x00007facaf070000)
  from space 34368K, 62% used [0x00007facb1500000,0x00007facb2a043c0,0x00007facb3690000)
  to   space 35904K, 0% used [0x00007facaf070000,0x00007facaf070000,0x00007facb1380000)
PSOldGen        total 230400K, used 217111K [0x00007fac990f0000, 0x00007faca71f0000, 0x00007facada50000)
  object space 230400K, 94% used [0x00007fac990f0000,0x00007faca64f5d98,0x00007faca71f0000)
PSPermGen       total 40640K, used 39254K [0x00007fac93cf0000, 0x00007fac964a0000, 0x00007fac990f0000)
  object space 40640K, 96% used [0x00007fac93cf0000,0x00007fac96345bd8,0x00007fac964a0000)




*
*
*
** occasional error that appear in tomcat log.


19:51:30,153  WARN ThreadPoolAsynchronousRunner:608 - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@22409774 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
19:51:30,155  WARN ThreadPoolAsynchronousRunner:624 - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@22409774 -- APPARENT DEADLOCK!!! Complete Status:
   Managed Threads: 3
   Active Threads: 3
   Active Tasks:
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@33c50522 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4bd415bb (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@50decee1 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
   Pending Tasks:
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@53e03f41
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2758ef6c
Pool thread stack traces:
   Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
      java.net.PlainSocketImpl.socketConnect(Native Method)
      java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      java.net.Socket.connect(Socket.java:529)
      java.net.Socket.connect(Socket.java:478)
      java.net.Socket.<init>(Socket.java:375)
      java.net.Socket.<init>(Socket.java:218)
      com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:147)
      com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:284)
      com.mysql.jdbc.Connection.createNewIO(Connection.java:2565)
      com.mysql.jdbc.Connection.<init>(Connection.java:1485)
      com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
      com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
      com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
      com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
      com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
      com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
   Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
      java.net.PlainSocketImpl.socketConnect(Native Method)
      java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      java.net.Socket.connect(Socket.java:529)
      java.net.Socket.connect(Socket.java:478)
      java.net.Socket.<init>(Socket.java:375)
      java.net.Socket.<init>(Socket.java:218)
      com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:147)
      com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:284)
      com.mysql.jdbc.Connection.createNewIO(Connection.java:2565)
      com.mysql.jdbc.Connection.<init>(Connection.java:1485)
      com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
      com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
      com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
      com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
      com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
      com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
   Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
      java.net.PlainSocketImpl.socketConnect(Native Method)
      java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      java.net.Socket.connect(Socket.java:529)
      java.net.Socket.connect(Socket.java:478)
      java.net.Socket.<init>(Socket.java:375)
      java.net.Socket.<init>(Socket.java:218)
      com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:147)
      com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:284)
      com.mysql.jdbc.Connection.createNewIO(Connection.java:2565)
      com.mysql.jdbc.Connection.<init>(Connection.java:1485)
      com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
      com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
      com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
      com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
      com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
      com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
      com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
      com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)