Hi,
I am using hibernate 3.2 on top of CGLib 2.1.3. I have noticed some objects not released even after the session factory and session are closed. After some profiling, the following is the trace from hibernate sessionimpl to gc root


org.hibernate.impl.SessionImpl
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer
<class>[]
java.lang.ThreadLocal$ThreadLocalMap$Entry
<class>[]
java.lang.ThreadLocal$ThreadLocalMap
org.apache.tomcat.util.threads.ThreadWithAttributes
<class>[]
java.lang.ThreadGroup
.....

There was a similar problem reported a while back
http://forum.hibernate.org/viewtopic.ph ... sc&start=0
it was fixed in CGLIB 2.1.3 (hibernate 3.1) but I was able to reproduced it.

One more thing I noticed is this memory leak only showed up under concurrent access (with more than 10 threads running the same test). If I performed the test sequenitally, the memory is GCed correctly.

1. download hibernate 3.2 source code bundle
2. open eg/org/hibernate/auction/Main.java
add the following line at the end of main function


mainItem = null;
mainBidder = null;
mainSeller = null;
test.factory = null;

Thread.currentThread().sleep(10000000);

this will remove all the reference from the Main class to hibernate and hanging the process for profiling.

3. run the profiler, trigger the gc a couple times. If you check the class
org.hibernate.impl.SessionImpl, there is always an instance hanging there with reference from CGLibLazyInitializer. CGLibLazyInitializer is referenced by ThreadLocal

The problem I encountered is I am using tomcat and Tomcat reuse the threads in its thread pool. So I have many dangling SessionImpl even after the requests are served.

Send me heap dump "java -Xrunhprof:format=b"

Hello, I also have a strange behavior when reloading my tomcat application, it uses hibernate 3.1.3, I upgraded cglib jar to version 2.1.3, trying to find out where the problem could be or if there's some cleanUp() method I forgot to call I included a small code to inspect threadLocals in all threads in the JVM, this is the result: (sorry, I don't know how to format an html table but you can copy this text an open it in a browser)

<table align="center" border="1"><tr><td>threadName</td><td>threadClass</td><td>classLoader</td><td>threadLocals</td></tr>
<tr><td>main</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
<tr><td>java.lang.Integer</td><td>10</td></tr>
<tr><td>[C</td><td>[C@1779885</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@be76c7 (pointing to: null)</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@682406 (pointing to: null)</td></tr>
</table></td></tr>
<tr><td>StandardManager[/admin]<br>(org.apache.catalina.session.StandardManager)</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.WebappClassLoader@1ad6b4b</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>StandardManager[/manager]<br>(org.apache.catalina.session.StandardManager)</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.WebappClassLoader@2b249</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>HostConfig[localhost]<br>(org.apache.catalina.startup.HostConfig)</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>http-8080-Processor1<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>http-8080-Processor2<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>http-8080-Processor3<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.WebappClassLoader@e222eb</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@74c144</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@13c765e</td></tr>
<tr><td>[C</td><td>[C@1e3bfb6</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@7f2036</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@1bd427 (pointing to: java.lang.StringCoding$CharsetSD instance)</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1054f93</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1ab4292</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@39f16f (pointing to: java.lang.StringCoding$CharsetSE instance)</td></tr>
</table></td></tr>
<tr><td>http-8080-Processor4<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@9fac35</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@153e0c0 (pointing to: null)</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@e2e276</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@181c4eb (pointing to: null)</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@1e514a9 (pointing to: null)</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@a75e9f</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@b91602 (pointing to: null)</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@3534c1</td></tr>
<tr><td>[C</td><td>[C@1c88f2b</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@11a7e7f</td></tr>
</table></td></tr>
<tr><td>http-8080-Processor5<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@af8b32</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1c5809e</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@66fcd5 (pointing to: java.lang.StringCoding$CharsetSD instance)</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@baa573</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@6e8504</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@2110fc (pointing to: java.lang.StringCoding$CharsetSE instance)</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1c44a6d</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1f2412a</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@d9205</td></tr>
<tr><td>[C</td><td>[C@edbca8</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@194566d</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1b64b70</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1fcca7b</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@112bc7b</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@11456c5</td></tr>
<tr><td>java.lang.ref.SoftReference</td><td>java.lang.ref.SoftReference@b4faeb (pointing to: null)</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@4ca42b</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1ae0e7d</td></tr>
<tr><td>[Lnet.sf.cglib.proxy.Callback;</td><td>[Lnet.sf.cglib.proxy.Callback;@1517e5e</td></tr>
</table></td></tr>
<tr><td>http-8080-Monitor<br>(org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>TP-Processor1<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>TP-Processor2<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>TP-Processor3<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>TP-Processor4<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>TP-Processor5<br>(org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)</td><td>org.apache.tomcat.util.threads.ThreadWithAttributes</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>TP-Monitor<br>(org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.StandardClassLoader@e86da0</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>Java2D Disposer<br>(sun.java2d.Disposer)</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.WebappClassLoader@419d05</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>WebappLoader[/estructura]<br>(org.apache.catalina.loader.WebappLoader)</td><td>java.lang.Thread</td><td>null</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>estructura gTimer started at 2/01/07 09:52 AM</td><td>java.util.TimerThread</td><td>org.apache.catalina.loader.WebappClassLoader@e222eb</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
<tr><td>StandardManager[/estructura]<br>(org.apache.catalina.session.StandardManager)</td><td>java.lang.Thread</td><td>org.apache.catalina.loader.WebappClassLoader@e222eb</td><td><table border="1"><tr><td>class</td><td>object</td></tr>
</table></td></tr>
</table>

This code is executed after a couple of reloads in my application executing some jsp code that uses hibernate between reloads. The jsp that prints this thread table is executed just after a reload and before calling any code that initializes hibernate. I implemented a ServletContextListener class that executes (among other things) this:



I also reproduced the problem including in my contextInitialized method this code:



But this code kind of confused me because I don't know which thread is executing it. Just loading an instance of com.msd.database.sfs.Territory generates the problem, this class has references to other classes, this is the .hbm.xml definition:



Any help would be greatly appreciated I don't know what else to do...

I configured my jvm to generate a heap dump automatically and I have many dumps to share, but a .zip dump is over 9MB, If interested just tell me how can I share it.

This may be related to my recent experince, using Tomcat under JBoss.

My problem was due to using ThreadLocal variables, without ever calling


Inside code handling a web serevice request, we attach an object to a thread as ThreadLocal<MyObject>. This object has references to some other objects, including Hibernate Session. But don't blame Hibernate.

Since Tomcat is recycling/pooling Threads, I (naively) thought that whenever the thread gets recycled, the thread local variables will be rre-set and eleased. But that is NOT the case.

Hence, one would think that I end up with that many instances of my object tree as there are threads in Tomcat pool - some reasonable number such as a few hundred. But that is NOT the case again. Under the profiler, the number of my thread local variables goes up and up - despite the total thread count staying low (i.e. 50).

Tomcat uses a ThreadWithAttributes wrapper around the (pooled) java Thread. And whenever you set a thread local variable, it somehow (I don't have Tomcat source, so this is just speculation) attaches the thread local variable to the Java Thread ThreadLocalMap hash map, with a key (IMHO) being that of the wrapper - which is almost never the same.

As a result, my thread local variables just keep piling up inside this map - and live for ever.
And when handling a web service with a request rate an order of 100 requests/sec, it takes just a few hours to eat all available server memory.

How is it related to class reloading?
Well, I suspect that your 'old' classes remain attached to the thread local (hash map), causing havoc.

I just had the same problem so I created an issue on jira

http://opensource.atlassian.com/project ... e/HHH-2456

I needed a fast solution while it gets fixed because my app is in production now, so I've written this code as a workaround with success after two months, inside my web application I included a <listener> tag pointing to a class supporting the ServletContextListener interface, in the contextDestroyed() event I call:



this is the method:



The method clears every ThreadLocal holding cglib classes only if it belongs to the same classLoader, I didn't call ThreadLocal.remove() just in case there's some code that still tries to access this threadlocal but It may work, after all we are inside a contextDestroyed event, hope this helps as a temporary solution...

Regards
[/code]

If I understand HHH-2481 correctly, this should be fixed in v.3.2.5, though I continue to see the CBLIBLazyInitializer leak in v3.2.5. Is the fix shown here still necessary to avoid this leak?