I have a problem, which can probably be resolved easily, but I'm not sure how to do it. So if anyone has any tips to try, I'm all ears.

The problem is this. I have what is basically a read only database. But after I load data into it, I have certain 'derived' data that I generate and have to populate the database with.

So I Load data (no problems).
Then I run a loader program that basically goes out with Hibernate(I've tried 2.0 and 2.1b3) to the mySQL database (4.04) and grabs some data. I then parse it and need to repopulate another table. This part works with no problems.

However, when I try to populate is when I run into problems. It'll work for a while, but eventually, runs out of memory (after about 2000 entries). The code is rather simple:




I have a session that is open in the function that calls this function. I've tried multiple things, including closing the session every 250 entries or whatever. I still get the problem (in fact, it happens quicker). I've tried using the evict statement below the connection().commit() statement above. No luck.

When I run memory profilers, I can see that java.lang.reflection has A LOT of bytes that it is allocating, but only a few that it is using. The calling class is a getting the hibernate. So basically, each set call, or creation of a session, allocates a lot of bytes, stops using them, but never releases control to be garbage collected.

I don't know if using JTA would help or not, it's about the only thing I haven't tried I think. But I think there is a leak.

Again, if anyone has had this problem, or has any ideas, please let me know.

Thank You,
Jeff

It's tough to be sure there's not something else wrong without more surrounding code, but one thing you could try is to tell hibernate not to optimize reflection via the config setting hibernate.cglib.use_reflection_optimizer = false

Then at least we'll eliminate the unlikely event of a memory leak in CGLIB.

Matt

Something I learnt the very first time I started doing database connectivity, is:

Create your connection to the database,
Get/save what you want,
Close your connection.

What I suggest, is that for your code, is to create a Configuration and a SessionFactory for each database when your application starts. Then, whenever you need to do a get/save, create a session, and give that session to the method that actually does the getting/saving. When it is done, kill the session, i.e. session.close().

I had the same initial problem with Hibernate, and once I started applying this principle to it, it ran faster and never ran out of memory. Also, you may want to tell your session to evict all objects you save, if you're not going to return them or manipulate them further, this also increases speed and helps with memory.

Do like this in your initialisation area:

Configuration cfg = new Configuration().configure(new File("hibernate.cfg.xml"));
SessionFactory sessionFactory = cfg.buildSessionFactory();
Session session;
...
//Then each time you need to save/get, do this...

session = sessionFactory..openSession();

//Then pass this session to your class...like you did... :)

try {
addAuthors(citation_id, struct_id, authors, session);
if(session != null) {
session.close();
}
}
catch (...) {}

//And last, but not least, remember to have a finally somewhere that includes this...

if(sessionFactory != null) {
sessionFactory.close();
}


-------------------------------------------------------------------------------------
Hope this helps. :)

-G

Have you tried to use simpe SQL statement like "INSERT ... FROM SELECT ..." ?