I am looking to switch over from Castor to Hibernate so that I can enjoy Hibernate's richer feature set. However, I'm getting a java.lang.OutOfMemory error during my record update processing.

Since there are no memory leaks in Hibernate, I must be doing something wrong! I've only been using hibernate for about a week now so that could very well be the case. Be that as it may, I've gone through the faqs and forums and documentation (which I must say all of which is very impressive). Yet, for the life of me, I can't figure out what I'm doing wrong.

The code is pretty basic and so is my mapping. My updates are taking place in a Message Driven Bean running on Oracle 9ias 9.0.2 running on Windows 2000. The database is Oracle8i Enterprise Edition Release 8.1.6.0.0 running on HP-UXUB.11.22. I'm using Oracle's JDBC driver 9.2.0.1.0 as a pooling data source. I've tried DBCP and Oracle's driver as a direct (non-pooling) source and am still seeing the problem.

I'm using Hibernate 2.1.2 downloaded last week on 3/9/2004.

Each message to the MDB is a record update. I have just over 90,000 updates to perform. The 9ias app server is at around 73M before I start the record update with Hibernate. I can get around 70,000 records updated before I start seeing the OutOfMemory exception. The process size is now around 86M. I've increased the max java heap size, but that only seems to delay the error.

Here's my relavant code in the MDB:
=================================


Here's my Alert mapping:
=================================


Here's my Hibernate config:
=================================


Thanks in advance for your help (and for making it to the bottom of this post)!

Chrisjan

The Session is also the first-level cache, it can't be turned of (essential for circular reference detection and automatic dirty checking). You can however evict() objects or clear() the whole Session.

Of course if you do all this in a single session, you will be likely to see OOME. It is best to create a fresh session for every batch of objects. At the very least you should call Transaction.commit() / session.clear() for every batch.

Christian, I saw the evict() and clear() methods in the api doc, but I thought they'd be unnecessary since I was calling close(). Well, I inserted the clear() like you suggested and put it just before my close() in the finally block. No change. I still get OutOfMemory after about 70,000 records.

Gavin, I believe I am already doing what you suggested. My method call persists a single object. Each method call does an open/begin/query/update/commit/close for a single object. Since I have 90,000 records to update. I'll be creating 90,000 sessions over the course of the entire batch.

Both you guys mention calling clear(). Is that necessary if I call close()? Does a close() also clear the cache?

What else can I try?

Chrisjan

what about using only one session (only one close at the end of your batch instead of 70000) and calling .commit() & .clear() on each iteration (better per 50 iterations for example cause flush is expensive)?
It depends how you want to manage commi/rollback.

This may be what gavin and christian wanted to say.

do the test again with proxool, there are problem with dbcp

delpouve, thanks for the clarification. I tried batching the updates in groups of 50. So, for every 50 updates, I have one session open/close with one begin/commit & clear. I did see a slight performance improvement, but the OutOfMemory still occurred at the same spot.

I am using Oracle's pooling driver. This is the driver I want to use. I may try proxool as you suggest, but I'm hesitant about using 3rd party pooling drivers because 9ias doesn't seem to play well with them.

I'm anxious to solve this problem. I appreciate all the feedback people have given. Keep 'em coming! Thanks!

i think the problem may not be hibernate.
If your batch is not too complex, take an hour and write it in JDBC, i'm afraid you'll get the same error.

If you can show more of your code.
I might be a detail...

Try some debug tool, it must find this problem.

Well, the problem *is* the orm (not Hibernate specifically - others will suffer from the same kind of problem during large updates).

As described above in the thread, Hibernate has to keep an image of your objects in the current Session - which double the space required in memory.

It has to do so because it is its only way to detect changes you may have made to your persistent entities. When the session is flushed, Hibernate will compare the internal images (snapshots) with the current state of your entities. Changes will trigger updates/inserts/deletes.

That's actually why Gvan (or Christian) told you to clear() the session at some points in your batch - freeing some memory.

I think you could take the following approach during your batch:
1/ create new entity
2/ tell Hibernate to persist it (save()/saveOrUpdate())
3/ repeat until a certain amount of objects are created
4/ when you hit your treshold, flush() the session and then clear() it. Then restart at point 1/ for the remaining entities.

This should work just fine (I believe). Just be carreful if you keep reference after the clear() to entities that were loaded by the session: they will not be associated with it, so changes made to them will not be detected anymore. Unless you explicitly re-associate them with the cleared session by calling saveOrUpdate().


Another hint:
By default, Hibernate flushes the session automatically, when it believes it is required. It does it also before executing (some) queries: it is required to persist all changes still in memory so the query (executed against the database) will return accurate results.

This behavior is fine and is actually one of the features that make Hibernate transparent and so powerful...

Unfortunately, the flush process may consume a fairly large amount of time. Remember that during the flush, Hibernate has to compare its snapshots with the current state of your entities associated with the session - the time required for this process is directly proportinal to the total number of properties to compare.

Think about the following scenario:
- you have 10.000 entities in the session;
- each entity has 10 properties;
- this gives a total of 100.000 properties to compare

Here is your hypothetical process:
1. create a new entity and add it to the session - nothing special happens;
2. before creating the next one, you have to lookup some references in the database. For this, you have to issue 3 HQL queries:
2.1 first query - before execution, Hibernate needs to flush the session - 1st flush - the changes made in 1/ will be transfered to the db;
2.2 second query - second flush - for 'nothing';
2.3 third query - third flush - for 'nothing'

As you can see, the two latest flushes were not required, because you haven't made any changes since 1. Unfortunately, Hibernate is not away of this - so it has no other alternatives than doing these flushes...

When doing large updates as you are doing, these extra flushes may have a great impact on performance - I'm pretty sure you observe this same behavior in your batch: the creation rate drops down as the number of objects increases.

Hopefully, Hibernate provides a solution - as usual...
If the queries you make during the creation of an object do not depend on changes made during the creation of previous ones, then you can turn-off the auto-flush (session.setFlushMode(FlushMode.NEVER)). This way, Hibernate will not flush the session anymore until explicilty instructed to do so. But remember that queries against the DB will not take your latest changes into account (they are not flushed!).
At the end of the process, flush() the session - everything will be sent to the database - and eventually re-enable the auto-flush.

This strategy give me 2000% performance boost (20 times faster!) with some large batches...

brenuart,
all i wanted cmatser to test is a JDBC approach to see if the problem was really specific to ORM.
But you seem to control the best approach for this kind of use (batch)
what about copy/pasting this post into the wiki? It might help a lot of users.

Anthony

I understood your point... and I'm pretty sure a plain JDBC approach will be faster because you will then drop everything that's part of an ORM a keep only the needed SQL (you will take full control)...

But I still believe you can achieve good-enough performance with an orm when you know how it behaves.

I'm intrested to have feedback from cmaster if he applies the recommendations ;)



Well, that's what I experienced in my own case - not sure it applies to everybody :(

Before posting this explanation to the wiki, I'd like to see first if Gavin won't sat I'm wrong ;) He may even give some more usefull advices...

plain jdbc will be faster... surely but out of memory can occure for other reason: remember dbcp ;) (thanks again for proxool advice on other posts)

ok let hibernate team react on this post to see if a wiki is welcome

I appreciate all the comments. I have a greater understanding of how Hibernate works. Unfortunately, I still have the OutOfMemory error.

delpouve, I can get my updates to work with Castor. Castor is the ORM tool we are currently using on this project. I can get all 90,000 records to update and memory holds steady at 76M. The code is basically the same (only substitute Castor classes instead of Hibernate). For other parts of this project, we have more complicated mappings which Castor doesn't handle very easily. I think Hibernate does a much better job with that and also with locking. Hibernate has other features which I think make it better than Castor and that is why we are looking to switch. But I can't recommend Hibernate to my team if I can't solve this memory problem.

I think I have already posted everthing of importance. The Alert object is a simple bean. It has 5 int, 2 java.lang.String, and 1 java.util.Date. The Alert table is in Oracle and has 5 INTEGER, 1 VARCHAR2(150), 1 CHAR(1), and 1 DATE. Is my mapping ok? I tried unmapping everything but the 5 integers. I still ran out of memory.

brenuart, I added flush() and clear() to my code. If you refer to my code in my 1st post, I added flush() followed by clear() right before the close() in my finally block. I still get OutOfMemory at around 70,000 records.

brenuart, since I am already doing a transaction commit(), this was already doing a flush() according to the api documentation. Also, since I open/close a new session for each record, shouldn't this already clear any associated memory? Hibernate debugging log seems to confirm that my code was already flushing.

brenuart, I appreciate the performance comments, but they are difficult to implement with my architecture. My code is running in a Message Driven Bean. Each message is a single record to update. So, each record is a fully contained transaction. If I try to group the updates together, I'll need to add triggers to flush the updates should the app server decide to pull my MDB out of main memory. I would also need to devise a specific flush message for when the batch is done or for one-off updates, but there is no easy way to send a message to a specific MDB. The app server starts 10 MDBs simultaneously so that processing can run in parallel. How do you tell a specific MDB to flush it's data?

The good news is that these updates are not time critical, and I haven't had the need to look for perfomance enhancing yet. The batch will be performed at night. So as long as I can be done within 8 hours, there's no problem. Running out of memory is a problem - it's a show stopper.

As baliukas suggested, perhaps it's time to use a profiler. Can anyone recommend a good memory profiler tool? Ideally, this tool will run in an app server(Oracle 9ias) and is FREE!

Based on others' posts, it seems obviously true, but can anyone confirm that they have been able to update a large quantity of records (more that 100,000) successfully? Did you have to do anything special? Did your code look like mine?

Thanks,
Chrisjan

We usually recommend to use stored procedures for mass updates/deletes, as it rarely makes sense to get that amount of data into memory and back for trivial tasks.