Memory use with transactional batch load

raman · **Joined:** Mon Jan 26, 2004 12:22 pm **Posts:** 4

I am attempting to batch load about a hundred thousand rows via hibernate. I have followed the advice in section 13 and am flushing and clearing the session every hundred rows. I have also disabled the second level cache for the Session.

Unfortunately, memory usage still increases rapidly as rows are inserted. Using jhat, I traced the memory use to references held to my persistent objects by the hibernate SessionImpl ActionQueue, as follows:

Code:

   ...skipped...
   --> org.springframework.orm.hibernate3.SessionHolder@0x2aaab6113080[new] (54 bytes) (field transaction:)
   --> org.hibernate.transaction.JDBCTransaction@0x2aaab6112bc8[new] (50 bytes) (field transactionContext:)
   --> org.hibernate.impl.SessionImpl@0x2aaab60da3a8[new] (144 bytes) (field actionQueue:)
   --> org.hibernate.engine.ActionQueue@0x2aaab60da440[new] (80 bytes) (field executions:)
   --> java.util.ArrayList@0x2aaab60db888[new] (32 bytes) (field elementData:)
   --> [Ljava.lang.Object;@0x2aaab8870240[new] (643856 bytes) (Element 70255 of [Ljava.lang.Object;@0x2aaab8870240:)
   --> org.hibernate.action.EntityInsertAction@0x2aaab96273c0[new] (80 bytes) (field instance:)
   --> my.persistent.Object@0x2aaab95fd7b8[new] (152 bytes)

The ActionQueue javadoc states that DML operations are queued until a flush. Since I am flushing the session periodically, I am uncertain why it is keeping these references:

Code:

[2007-11-11 02:12:34,624][DEBUG][def.AbstractFlushingEventListener] executing flush
[2007-11-11 02:12:34,624][DEBUG][jdbc.ConnectionManager] registering flush begin
...
[2007-11-11 02:12:34,626][DEBUG][jdbc.ConnectionManager] registering flush end
[2007-11-11 02:12:34,626][DEBUG][def.AbstractFlushingEventListener] post flush

Committing and restarting the jdbc transaction every hundred rows solves the memory issue -- after the commit, the persistent objects that have been flushed and cleared are garbage collected. However, obviously I'd like to be able to complete the batch load in a single transaction. Is it possible to do this with Hibernate, without memory use increasing without bounds as the number of rows increases?

Cheers,
Raman Gupta

VampBoy · **Posted:** Wed Nov 14, 2007 1:13 pm

If you're using query caching the action queue will remain full even after a flush, clear, etc. It will only get cleared after the transaction is committed.

raman · **Joined:** Mon Jan 26, 2004 12:22 pm **Posts:** 4

VampBoy wrote:

If you're using query caching the action queue will remain full even after a flush, clear, etc. It will only get cleared after the transaction is committed.

Bingo! Turning off the query cache solved the memory problem. Now, the follow up question -- is there any way to turn off the query cache for the individual session that is executing the batch load?

The only other option I see (other than a periodic commit) is to execute the batch load in a separate JVM, with a different Hibernate config that has disabled query caching.

VampBoy · **Posted:** Thu Nov 15, 2007 11:31 am

raman wrote:

VampBoy wrote:

If you're using query caching the action queue will remain full even after a flush, clear, etc. It will only get cleared after the transaction is committed.

Bingo! Turning off the query cache solved the memory problem. Now, the follow up question -- is there any way to turn off the query cache for the individual session that is executing the batch load?

The only other option I see (other than a periodic commit) is to execute the batch load in a separate JVM, with a different Hibernate config that has disabled query caching.

Not that I know of, there was some chatter around the memory usage on the dev mailing list in the past week but I'm not sure if anything is going to come out of it.