I have some really annoying freezes on my production website. Hope someone can help :)
Server specs:
-OS: Linux CentOS
-Webserver: Tomcat 6
-MySQL 5 with default settings
-Frameworks: Wicket 1.4-rc2 (webframework), Spring 2.5.6, Hibernate 3.3.1.GA, C3P0 db pooling 0.9.1.2, URLRewriteFilter 3.1.0, Spring OpenSessionInViewFilter
Problem:
At some point (about 5 to 24 hours) after a boot, Tomcat seems to stop serving requests, although it does create new threads for new incoming connections. But since it does not serve those connections, the new connections will show and keep showing 0kb transfer. Shortly after that, max-thread is reached, so then I'm unable to access any webapp (the main website + tomcat manager) running on that Tomcat server. I'm not sure if it is able to serve static resources in the short time window where it still has some threads free, because the time window is too short to notice when it happens.
Observations:
-No exceptions or errors in the catalina logs. So no memory problems, since no error occurs in the logs
-Java Thread dump using command: kill -QUIT (see output below), shows the text "locked" and "WAITING".
-Adding c3p0 db pooling idle checks, tests and timeout settings (see below) did not help.
-Using Mysql Administrator GUI shows that after a freeze, there will be 15 threads, all sleeping. Normally this would return to a minPoolSize of 5, if I'm correct, but thats not the case.
-The freeze continues for hours, and does not recover. I have to restart Tomcat.
I'm suspecting that it has something to do with DB pooling. I'm not sure if "locked" and "WAITING" are normal behaviour. But if there is something wrong, why doesn't c3p0 recover from it?
Can somebody shed some light on this? :)
Configuration:
Code:
=== DB pooling in Spring applicationContext.xml ====
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass"><value>com.mysql.jdbc.Driver</value></property>
<property name="jdbcUrl" value="jdbc:mysql://localhost/xxxxxx" />
<property name="user"><value>xxxxxxxx</value></property>
<property name="password"><value>xxxxxxxxx</value></property>
<property name="maxConnectionAge"><value>600</value></property>
<property name="idleConnectionTestPeriod"><value>180</value></property>
<property name="testConnectionOnCheckin"><value>true</value></property>
<property name="acquireIncrement"><value>5</value></property>
<property name="maxIdleTime"><value>180</value></property>
<property name="maxPoolSize"><value>15</value></property>
<property name="maxStatements"><value>100</value></property>
<property name="minPoolSize"><value>5</value></property>
</bean>
===========================
Code:
===== Thread dump part ===========
"http-8080-28" daemon prio=10 tid=0xb4fbb400 nid=0x4ae2 in Object.wait() [0xb4316000..0xb4318130]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x72c74b10> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:510)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy37.findById(Unknown Source)
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.wicket.proxy.LazyInitProxyFactory$JdkHandler.invoke(LazyInitProxyFactory.java:416)
at org.apache.wicket.proxy.$Proxy2.findById(Unknown Source)
at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
at org.apache.wicket.model.LoadableDetachableModel.getObject(LoadableDetachableModel.java:119)
at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:151)
at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:89)
at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.newPage(BookmarkablePageRequestTarget.java:306)
at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.getPage(BookmarkablePageRequestTarget.java:321)
at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.processEvents(BookmarkablePageRequestTarget.java:234)
at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)
at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1192)
at org.apache.wicket.RequestCycle.step(RequestCycle.java:1271)
at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1370)
at org.apache.wicket.RequestCycle.request(RequestCycle.java:501)
at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:455)
at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:288)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:164)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:90)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:417)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
=======================================
Login
FAQ
Search
