Hello,
first i have to say that i am not involved with any Java programming. I have to maintain a debian root-Server running Apache2/mod_jk-1.2.19/Tomcat5.5/PostgreSQL 8.0.4 with only one Java Webapp deployed.
The App uses hibernate2 and struts/strutsCX, the company that has made the app is not existing any more, so there is also no support.
I already searched the www and this (and other) forum extensively, made a lot of tests, read any existing logs and did suggested changes to config files, environment variables of tomcat, tried different JVM versions and vendors (sun, bea) but the problems persist. it has been about six months now...
Problems:
under heavy load, many database connections get stuck in "idle in transaction", so do all available tomcat threads. After restart, the App gets connected to postgres but the max_connection limit gets exhausted. The c3p0 connection pool is configured to 60 connections maximum. Sometimes, at the time the webapp is available again after a restart, the JVM has more than 200 connections in idle mode. then the amount of "idle" and "idle in transaction" connections starts to grow again until the maximum number of postgres server connections is reached (400 at the time). Switching the connection-pool provider does not change anything in this behaviour but when using the built in hibernate connection pool provider, the count of total connections minus "idle in transaction" connections is always two until the configured maximum settings of the pool is reached, then the App is choking on requests again.
I managed to write some perl scripts to monitor the number of "idle in transaction" connections and total connections, as well as the number of apache2 processes. if any of these three parameters exeeds its maximum (50 idle_it,400 dbconn,380 ap2), the webapp gets restarted (killed and started) automatically, which can happen in up to 3 minute intervals on busy evenings sometimes.
Since this doesnt turn out to be a JVM or config issue, i grabbed the classfiles and decompiled them to look at some code. As far as i read all the postings in this forum on the "idle in transaction" problem, the problem is in the code most of the time. I also read about the autoCommit Property and started to change the code and recompiled the class to read a flag from a configuration file, to turn autoCommit on and off for testing purposes (see below). When autoCommit is enabled, there are no more "idle in transaction" connections but instead many "idle" connections which never get reused, so this really wasnt a fix to the problem.
So i tried to post to this forum finally...
Please remember, i am NOT a Java programmer. I can code Perl and PHP on a pretty high level and do this on a daily basis, so coding is anything but new to me but i am still not much into java.
I think i can provide any source and config information on request, since the server is in my hands :)
I can edit the decompiled source and so i can implement any suggested changes, please try to be a little specific, remember my java-lacking background.
Below is the source of the hibernate session creation class.
This one gets the setting of "autocommit" (which is "false" at the moment):
configmanager.getFirstElement("/config/DatabaseConfig/AutoCommit");
I didnt see any session.transaction.begin() nor .commit() or .rollback() method calls in any of the decompiled classes of the application. The error handling of the code posted below seems like a bad joke to me too.
Thanks a LOT for any help!
Hibernate version: hibernate2.jar
Code:
Code:
package com.company.core.database;
import com.company.core.config.ConfigManager;
import java.sql.Connection;
import java.util.*;
import net.sf.hibernate.Session;
import net.sf.hibernate.SessionFactory;
import net.sf.hibernate.cfg.Configuration;
import org.apache.log4j.Logger;
public class SessionProvider2
{
public SessionProvider2()
{
}
public static Session getSession()
throws Exception
{
if(sfFactory == null)
{
ConfigManager configmanager = ConfigManager.getInstance();
String s = configmanager.getFirstElement("/config/DatabaseConfig/AutoCommit");
if(s.equals("true"))
autocommit = true;
Configuration configuration = new Configuration();
Object obj = null;
Vector vector = null;
vector = configmanager.getElements("/config/DatabaseConfig/StoreFile");
String s2 = configmanager.getFirstElement("/config/DatabaseConfig/StoreFilePath");
String s3 = System.getProperty("file.separator");
if(vector != null)
{
for(Iterator iterator = vector.iterator(); iterator.hasNext();)
{
String s1 = (String)iterator.next();
lLog.info((new StringBuilder()).append("SessionProvider - before storeFile ").append(s2).append(s3).append(s1).toString());
configuration = configuration.addFile((new StringBuilder()).append(s2).append(s3).append(s1).toString());
}
}
try
{
sfFactory = configuration.buildSessionFactory();
}
catch(Exception exception)
{
lLog.fatal("Error building SessionFactory", exception);
throw exception;
}
configuration = null;
}
Session session = sfFactory.openSession();
if(isAutoCommit())
session.connection().setAutoCommit(true);
if(lLog.isDebugEnabled())
{
iOpenSessions++;
Throwable throwable = new Throwable();
StackTraceElement astacktraceelement[] = throwable.getStackTrace();
if(astacktraceelement.length >= 2)
lLog.debug((new StringBuilder()).append("Sessions open (getSession):\t\t").append(iOpenSessions).append(" -- Method: ").append(astacktraceelement[1].getClassName()).append(" ").append(astacktraceelement[1].getMethodName()).append(" ").append(astacktraceelement[1].getLineNumber()).toString());
else
lLog.debug((new StringBuilder()).append("Sessions open (getSession):\t\t").append(iOpenSessions).toString());
if(hmTimes == null)
hmTimes = new HashMap(5);
hmTimes.put(session, new Long(System.currentTimeMillis()));
}
return session;
}
public static boolean closeSession(Session session)
{
boolean flag;
if(session != null)
break MISSING_BLOCK_LABEL_30;
lLog.error("Session is null - should not happen");
flag = false;
if(hmTimes != null)
hmTimes.remove(session);
return flag;
ConfigManager configmanager = ConfigManager.getInstance();
String s = configmanager.getFirstElement("/config/DatabaseConfig/AutoCommit");
if(isAutoCommit())
session.connection().setAutoCommit(false);
session.close();
if(lLog.isDebugEnabled())
{
if(hmTimes == null)
hmTimes = new HashMap(5);
long l = System.currentTimeMillis() - ((Long)hmTimes.get(session)).longValue();
iOpenSessions--;
Throwable throwable = new Throwable();
StackTraceElement astacktraceelement[] = throwable.getStackTrace();
if(astacktraceelement.length >= 2)
lLog.debug((new StringBuilder()).append("Sessions open (closeSession ").append(l).append(" ms):\t").append(iOpenSessions).append(" -- Method: ").append(astacktraceelement[1].getClassName()).append(" ").append(astacktraceelement[1].getMethodName()).append(" ").append(astacktraceelement[1].getLineNumber()).toString());
else
lLog.debug((new StringBuilder()).append("Sessions open (closeSession ").append(l).append(" ms):\t").append(iOpenSessions).toString());
}
if(hmTimes != null)
hmTimes.remove(session);
break MISSING_BLOCK_LABEL_325;
Exception exception;
exception;
boolean flag1;
lLog.error("Error closing session", exception);
flag1 = false;
if(hmTimes != null)
hmTimes.remove(session);
return flag1;
Exception exception1;
exception1;
if(hmTimes != null)
hmTimes.remove(session);
throw exception1;
return true;
}
public static boolean isAutoCommit()
{
return autocommit;
}
private static SessionFactory sfFactory = null;
protected static Logger lLog = Logger.getLogger(com/company/core/database/SessionProvider2);
protected static int iOpenSessions = 0;
protected static HashMap hmTimes = null;
protected static boolean autocommit = false;
}
Name and version of the database you are using:
PostgreSQL 8.0.4 on Debian Sarge, Kernel 2.6.3
Login
FAQ
Search