There is a point where my application no longer responds and gives this error:
All threads (4,000) are currently busy, waiting. Increase maxThreads (4,000) or check the servlet status

I have discovered that what actually triggers this is that at some point, for some reason, any line of code that tries to use the database gets stuck there. I know this because my application has several schedulers. The ones that use the database start and never finish, while the others start and finish regularly. When the application reaches the state where the database no longer responds, if I go to the server that has the database, I see that all the database transactions that the application uses are idle. As if the application thinks that those transactions are being used when they actually are idle and ready to be used.

Every connection with the database is done by using Spring's JDBCTemplate or Hibernate (using the same Datasource I use with JDBCTemplate). The database is PostgreSQL 8.3.

Do you have any ideas? Anything will do. I don't know where to start looking. What to try... I need some hint. Any help will be appreciated.

Edit: I now re-checked, and in the database server the connections when the application reaches this point are "idle in transaction", nos just "idle". And also, I used the following implementation for my DataSource: com.mchange.v2.c3p0.ComboPooledDataSource

First you have to find out if really c3p0 is responsible for such many threads.
Please execute
jstack <pid>
on your application and report us the output.

regards
Guenther

Ok. The stack is kinda big. I got this one yesterday when the application was about to die (I could see the "idle in transaction" number increasing rapidly), as well as the number of threads. I won't post the whole thing, as there is too much repetition. I'll try to post one of each different entry:

I have this one from 7 to 1, excluding 2 that's different
"http-8011-7" daemon prio=10 tid=0x00007fa1f2022800 nid=0x1920 in Object.wait() [0x00000000769ff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa2fe24c3d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
- locked <0x00007fa2fe24c3d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
at java.lang.Thread.run(Thread.java:619)

Here it is, number 2
"http-8011-2" daemon prio=10 tid=0x000000004049f000 nid=0x191b runnable [0x00007fa1ea50d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:740)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:764)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:573)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:325)
at org.apache.tomcat.util.buf.IntermediateOutputStream.write(C2BConverter.java:242)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
- locked <0x00007fa2fe2b2c70> (a org.apache.tomcat.util.buf.WriteConvertor)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at org.apache.tomcat.util.buf.WriteConvertor.flush(C2BConverter.java:191)
at org.apache.tomcat.util.buf.C2BConverter.flushBuffer(C2BConverter.java:134)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:439)
at org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:143)
at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:326)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:342)
at com.opensymphony.module.sitemesh.parser.FastPage.writeBody(FastPage.java:48)
at com.opensymphony.module.sitemesh.taglib.decorator.BodyTag.doEndTag(BodyTag.java:26)
at org.apache.jsp.WEB_002dINF.jsp.decorators.probe_jsp._jspx_meth_decorator_005fbody_005f1(probe_jsp.java:1466)
at org.apache.jsp.WEB_002dINF.jsp.decorators.probe_jsp._jspx_meth_c_005fotherwise_005f0(probe_jsp.java:1439)
at org.apache.jsp.WEB_002dINF.jsp.decorators.probe_jsp._jspx_meth_c_005fchoose_005f0(probe_jsp.java:1320)
at org.apache.jsp.WEB_002dINF.jsp.decorators.probe_jsp._jspService(probe_jsp.java:285)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:105)
at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:965)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:744)
at org.jstripe.tomcat.probe.ProbeServlet.doDispatch(ProbeServlet.java:49)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:663)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:394)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:348)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at com.opensymphony.module.sitemesh.filter.PageFilter.applyDecorator(PageFilter.java:158)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)

This one only once...
"Keep-Alive-Timer" daemon prio=10 tid=0x00007fa1e47c7800 nid=0x1908 waiting on condition [0x000000004af47000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:149)
at java.lang.Thread.run(Thread.java:619)

I have this one 1333 times
"TP-Processor1600" daemon prio=10 tid=0x00007fa1e59a7000 nid=0x18ec runnable [0x00007fa1bd59a000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00007fa23a468f88> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:559)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)

I have this one 99 times
"TP-Processor1576" daemon prio=10 tid=0x00007fa1e5978000 nid=0x18d4 in Object.wait() [0x00007fa1bedb2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa23a617cf8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
- locked <0x00007fa23a617cf8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:619)

I have this one 88 times, which would actually explain 88 idle in transactions... I'll close the transaction before blocking...
"TP-Processor1513" daemon prio=10 tid=0x00007fa1e4197800 nid=0x188a waiting for monitor entry [0x00007fa1c2cef000]
java.lang.Thread.State: BLOCKED (on object monitor)
at ar.com.contenta.mvc.actions.suscripciones.CrearSuscriptorAction.execute(CrearSuscriptorAction.java:45)
- waiting to lock <0x00007fa22b5f9a68> (a java.lang.Object)
at sun.reflect.GeneratedMethodAccessor394.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.opensymphony.xwork.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:358)
at com.opensymphony.xwork.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:218)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:192)
at com.opensymphony.xwork.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:175)
at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:115)
at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.webwork.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:174)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.webwork.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:169)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:151)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:186)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at ar.com.contenta.interceptors.NoCacheInterceptor.intercept(NoCacheInterceptor.java:25)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.DefaultActionProxy.execute(DefaultActionProxy.java:116)
at com.opensymphony.webwork.dispatcher.DispatcherUtils.serviceAction(DispatcherUtils.java:225)
at com.opensymphony.webwork.dispatcher.ServletDispatcher.service(ServletDispatcher.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at ar.com.contenta.mvc.FiltroLiberarSessionHibernate.doFilter(FiltroLiberarSessionHibernate.java:39)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)

I have this one 3 times
"TP-Processor1354" daemon prio=10 tid=0x00007fa1e5152800 nid=0x17eb runnable [0x00007fa1cd08e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00007fa22d7c1ce0> (a java.io.BufferedInputStream)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:254)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188)
- locked <0x00007fa22d7c1f38> (a org.postgresql.core.v3.QueryExecutorImpl)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:430)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:346)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:250)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)
at org.hibernate.loader.Loader.doQuery(Loader.java:674)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
at org.hibernate.loader.Loader.doList(Loader.java:2220)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
at org.hibernate.loader.Loader.list(Loader.java:2099)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1569)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283)
at ar.com.contenta.db.impl.SuscripcionesDAOHibernateImpl.getSuscriptorVigenteByMsisdn(SuscripcionesDAOHibernateImpl.java:55)
at ar.com.contenta.mvc.actions.suscripciones.CrearSuscriptorAction.execute(CrearSuscriptorAction.java:45)
- locked <0x00007fa22b5f9a68> (a java.lang.Object)
at sun.reflect.GeneratedMethodAccessor394.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.opensymphony.xwork.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:358)
at com.opensymphony.xwork.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:218)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:192)
at com.opensymphony.xwork.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:175)
at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:115)
at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.webwork.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:174)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.webwork.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:169)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:151)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:186)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at ar.com.contenta.interceptors.NoCacheInterceptor.intercept(NoCacheInterceptor.java:25)
at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190)
at com.opensymphony.xwork.DefaultActionProxy.execute(DefaultActionProxy.java:116)
at com.opensymphony.webwork.dispatcher.DispatcherUtils.serviceAction(DispatcherUtils.java:225)
at com.opensymphony.webwork.dispatcher.ServletDispatcher.service(ServletDispatcher.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at ar.com.contenta.mvc.FiltroLiberarSessionHibernate.doFilter(FiltroLiberarSessionHibernate.java:39)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)

I have this one 1 time
"DefaultQuartzScheduler_QuartzSchedulerThread" prio=10 tid=0x00007fa1e43e2800 nid=0x1286 sleeping[0x0000000047c14000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)

I have this one from 9 to 0
"DefaultQuartzScheduler_Worker-9" prio=10 tid=0x00007fa1e43e0800 nid=0x1285 in Object.wait() [0x0000000047b13000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa221d18dd0> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
- locked <0x00007fa221d18dd0> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

I have this one from 2 to 0, two times each, so I have a total of 6
"com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon prio=10 tid=0x00007fa1f1abb800 nid=0x127a in Object.wait() [0x0000000047008000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa228168888> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
- locked <0x00007fa228168888> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

I have this one from 1 to 0
"Timer-1" daemon prio=10 tid=0x00007fa1f0337800 nid=0x1277 in Object.wait() [0x0000000046d05000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa228169d10> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00007fa228169d10> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

I have this one once
"http-8011-Acceptor-0" daemon prio=10 tid=0x00007fa1e4240800 nid=0x123e runnable [0x00000000438d1000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x00007fa22292afe0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:317)
at java.lang.Thread.run(Thread.java:619)

I have this one once
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007fa1e4240000 nid=0x123d waiting on condition [0x0000000040bb3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
at java.lang.Thread.run(Thread.java:619)

I have this one once
"Java2D Disposer" daemon prio=10 tid=0x00007fa1e40d0800 nid=0x123c in Object.wait() [0x0000000040ab2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa22206aa38> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00007fa22206aa38> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.java2d.Disposer.run(Disposer.java:125)
at java.lang.Thread.run(Thread.java:619)

I have 17 like this one. Each is a different scheduler
"timerFactoryMcert" prio=10 tid=0x00007fa1e408d800 nid=0x123b in Object.wait() [0x00000000437d0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa228ccdae8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00007fa228ccdae8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

I have 4 like this one once. Each is a different scheduler
"timerFactorySuscripciones-18" prio=10 tid=0x00007fa1e4075000 nid=0x122e runnable [0x0000000041bf6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00007fa228515b40> (a java.io.BufferedInputStream)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:254)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188)
- locked <0x00007fa228515d98> (a org.postgresql.core.v3.QueryExecutorImpl)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:430)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:346)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:250)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:595)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:538)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:588)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:617)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:625)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:668)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:679)
at org.springframework.jdbc.core.JdbcTemplate.queryForInt(JdbcTemplate.java:706)
at ar.com.contenta.db.impl.ServicesDaoHibernateImpl.getCantidadMensajesSalientesPendientes(ServicesDaoHibernateImpl.java:361)
at ar.com.contenta.mw.suscripciones.EnviarSuscripciones.noHayProblemas(EnviarSuscripciones.java:522)
at ar.com.contenta.mw.suscripciones.EnviarSuscripciones.enviarSuscripciones(EnviarSuscripciones.java:166)
at ar.com.contenta.mw.suscripciones.EnviarSuscripciones.run(EnviarSuscripciones.java:136)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

I have this one once
"Store org.hibernate.cache.StandardQueryCache Expiry Thread" daemon prio=10 tid=0x00007fa1e4019800 nid=0x1226 waiting on condition [0x0000000042cc5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.expiryThreadMain(DiskStore.java:831)
at net.sf.ehcache.store.DiskStore.access$700(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$ExpiryThread.run(DiskStore.java:1008)

I have everything else only once...

"Low Memory Detector" daemon prio=10 tid=0x00000000401a4000 nid=0x121a runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00000000401a1000 nid=0x1219 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x000000004019d000 nid=0x1218 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x000000004019b000 nid=0x1217 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000040177800 nid=0x1216 in Object.wait() [0x00000000424bd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa22b535ab0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00007fa22b535ab0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x0000000040176000 nid=0x1215 in Object.wait() [0x0000000041f72000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fa22b535a68> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00007fa22b535a68> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0000000040113000 nid=0x120f runnable [0x0000000040781000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x00007fa223a95d58> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:647)
at org.apache.catalina.startup.Catalina.start(Catalina.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

"VM Thread" prio=10 tid=0x000000004016f000 nid=0x1214 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004011d800 nid=0x1210 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000004011f000 nid=0x1211 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040121000 nid=0x1212 runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040122800 nid=0x1213 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007fa1ec049800 nid=0x121c waiting on condition

JNI global references: 991

Heap
PSYoungGen total 1126272K, used 413964K [0x00007fa2c9ef0000, 0x00007fa31f440000, 0x00007fa31f440000)
eden space 854464K, 43% used [0x00007fa2c9ef0000,0x00007fa2e067abe8,0x00007fa2fe160000)
from space 271808K, 16% used [0x00007fa2fe160000,0x00007fa300e18530,0x00007fa30ead0000)
to space 268544K, 0% used [0x00007fa30ee00000,0x00007fa30ee00000,0x00007fa31f440000)
PSOldGen total 2796224K, used 484136K [0x00007fa21f440000, 0x00007fa2c9ef0000, 0x00007fa2c9ef0000)
object space 2796224K, 17% used [0x00007fa21f440000,0x00007fa23cd0a0d0,0x00007fa2c9ef0000)
PSPermGen total 62528K, used 62369K [0x00007fa1f3840000, 0x00007fa1f7550000, 0x00007fa21f440000)
object space 62528K, 99% used [0x00007fa1f3840000,0x00007fa1f7528418,0x00007fa1f7550000)

OK, there problem are definitely the 1333 TP-Processor threads.
As next step I would suggest to try find out,
with which actions the number of TP-Processor threads increases so much.
Is the number already so high, when your tomcat is started or is it growing gradually
as http-clients are interacting with tomcat?

Anyway i guess that the problem has not to do with hibernate.

Ok... For now, I'm deploying a new version in which I close the transaction before locking, and then re-open it after I get the lock. That should get rid of the "idle in transaction" I believe. If the application keeps dying, I'll come back and post.

Ok... Now that I've been constantly checking the stack, I found out that this is a pretty common problem:




What could cause this locking? I have many different threads running at the same time. I use a SessionFactory with a ComboPooledDataSource. Shouldn't every thread be using a different connection to the DataBase? How could the be locking like this?

This thread is executing a query on database and waiting answer from the database.
I can immagine 2 scenarios:
-either the query builds up a large result set and therefore it take long time for answer
-or the query iexecution is blocked on db due locks deriving from other concurrent sessions

As first step I suggest to find out which query is executed at ServicesDaoHibernateImpl.java:361
ar.com.contenta.db.impl.ServicesDaoHibernateImpl.getCantidadMensajesSalientesPendientes(ServicesDaoHibernateImpl.java:361)

Then try to find out with which isolation level you access the postgres database...

Thanks for the quick response. The isolation level is Read Commited. And that query in particular, is just a count(*) on a not so big table. With not so big I mean that it's kinda small, like 5,000 entries. But it's not always that line the one with the problem. I sometimes get the same error on other queries.

I suggest to trace your queries wit P6Spy (its free).
P6Spy logs queries by logging also the execution time.
Furthermore P6Spy can detect 'outages' which are queries which never return (or take very long time)

It is still not clear if In your case the query is only taking long for answering of if
it not answering at all (= blocked on db-server).

I know Postgres not very well,
but I know there are some database which
with ReadCommited isolation level simply do wait for all other transactions
(keeping locks on the interested table) being closed before executing the actual query.
Such scenario could easily lead to query-outages.

Ok, I'll look into it and post the results.

I asked the guy who handles the servers to set up P6Spy. In the meantime, I wanted to ask you something that I thought. The communication with database within the application is not always done by Hibernate. We also use JdbcTemplate. I use the same DataSource instance for both the SessionFactory I use with Hibernate and to instanciate my JdbcTemplate objects. I have Hibernate's current_session_context_class property set to "thread". The threads that get locked, begin a transaction, do stuff with Hibernate and also stuff with JdbcTemplate, and then commit the transaction. Could this have anything to do with my problem? Maybe not, but just to be sure.

I don't have any experience with 'JdbcTemplate',
anyway I assume (and I really hope for you) that JdbcTemplate and Hiberante are not sharing the same connection/transaction when interacting with the database, but using separate connections.
So in theory the fact that you use JdbcTemplate contemporaneosly with Hiberante on the same datasource
should be no problem.

Sorry that I didn't post anything for such a long time, we had some other issues. Now I'm back at this. We haven't set up what you told me to. At least not yet. However, I've been thinking, and want to at least try to lower the isolation level and see what happens. I want to set it to READ UNCOMITTED for a day or so. But before I do, I'd like to understand how Hibernate behaves in that case.
Suppose a transaction is modifying a certain table and I get, via Hibernate, an entry on that table. Now, the transaction that was modifying the table, commits, having modified a value within my recently acquired Hibernate object. Now, the value in my Hibernate object is old. But I don't modify it, I just read it. Now, I save my Hibernate object. Will the old value return to the table? Or will Hibernate realize I never touched that column and do nothing about it?

Edit: Also, I said that it was set to READ COMMITTED, because that's how my database is set. However, I don't have the hibernate.connection.isolation property set. What value does it default to?

Edit 2: I just found out that PostgreSQL doesn't support READ UNCOMMITTED... Oh well, there goes my idea.

Did you use some database tool to verify there are no deadlocks at database level? What are these locks you mentioned?
Do all these threads finish eventually if you wait enough time?

Hibernate realizes that you never touched that entity will not produce any DML statement on flush/commit.