We have a web application (Appfuse, Struts2, Spring) that allows importing records. We can import 2 million records easily with very little memory usage (we do a session flush and clear every time we hit a preset batch limit).

We also wanted to use Hibernate Search to index parts of these records. Once we annotated the models, we ran into a memory problem. What happens is that memory will climb to to a certain point and get collected. That's good, but the amount of memory being committed doesn't stop climbing. Eventually 2GB gets committed and the JRE crashes. Without HSearch memory stays around 160MB.

Has anybody seen anything like this before?

We are using:

Jave 1.6.0_02-b06
Hibernate 3.2.5GA
HSearch 3.0.0.CR1

HSearch config looks like

<property name="hibernate.search.default.directory_provider">org.hibernate.search.store.FSDirectoryProvider</property>
<property name="hibernate.search.default.indexBase">target/indexes2</property>
<property name="hibernate.search.worker.batch_size">200</property>
<property name="hibernate.search.default.transaction.merge_factor">200</property>
<property name="hibernate.search.default.transaction.max_merge_docs">20000</property>
<property name="hibernate.search.default.transaction.max_buffered_docs">200</property>

Thanks in advance.

Hum that is weird. It looks like either: the GC somehow is screwed or the batch_size is not used.
Are you sure there is no typo? Do you flush() and clear() every 200 elements?

Can you try and use a profiler to see which objects are kept around?
JBoss Profiler is not too GUI friendly but it does the job.

I've checked for typos and nothing is jumping out at me. We are doing a flush and clear after every 200 records.



I've tried YourKit to catch some snapshots. The last time that I did it I had 1.6GB committed and 650MB being used. It didn't show me where the 950MB was being used. A minute leter it blew up with 2GB committed and 630MB being used.

I have no problem tryinh JBoss Profiler to try to figure this out.

What do you mean by committed? Available for GC but not actually freed?

RAM committed from the OS, but not being released. It can be seen using PerfMon on Windows. Once it hits the 2GB Windows limit the JRE crashes. It never gets released back to the OS, even when a GC happens (which does release private memory used).

So what we see in our webapp without Hibernate Search is around 165MB committed and a private memory usage of 160MB-164MB (it will climb to around 164MB and then collect and go back down to about 160MB).

Once we enable Hibernate Search we see something like private memory between 600MB-700MB (climbs to 700MB, collects, and then drops to 600MB), but committed memory just keeps climbing until it hits the 2GB wall and the JRE crashes.

I hope I explained that well enough.

strange, so you don't even have OutOfMemoryExceptions then, correct?

What is the exact VM version you are using. Sounds a lot like a VM bug that HSearch triggers indirectly.

No. We haven't had an OutOfMemoryException since we started doing the flush() and clear() during import.

The VM version is 1.6.0_02-b06.

I would really love to see what object get stuck in memory though.
It is expected for the OS memory not to be released by the VM. What is not is that the GC doesn't kick off. the profiler should help to find the culprit memory leak (if any)

can you also post a somewhat details code (both entity and the indexing loop) so that I can try and reproduce?

BTW can you play with the -client -server modes of the VM? and potentially some other garbage collector implementations

If the profiler don't show enough, you can try some of those tricks
http://wiki.jboss.org/wiki/Wiki.jsp?page=StackTrace

Actually, I think it's something to do with Eclipse. I downloaded Jetty, ran the WAR outside of Eclipse, and memory stays steady in the process. I have no idea why spawning the process from Eclipse is causing the spawned process to commit memory like it does.

It could be bad if we ever had to debug an import with a large record set, but at least we know we can get around it when we go to production.