-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 1 post ] 
Author Message
 Post subject: Still getting outofmemory errors with batch processing
PostPosted: Thu Jan 13, 2005 1:28 pm 
Beginner
Beginner

Joined: Mon Dec 08, 2003 12:15 am
Posts: 47
Hibernate version: 2.16

Mapping documents:N/A

int rows = 0;
try{
Session as400Session = getHibernateTemplate().getSessionFactory().openSession();

//set up the ora session
HibernateDaoSupport dao = (HibernateDaoSupport) getOraDao();
oraSession = dao.getSessionFactory().openSession();
oraTx = oraSession.beginTransaction();

//delete the vins
int rowsDeleted = getOraDao().deleteVins(oraSession);
log.info("There were " + rowsDeleted + " vins deleted");

//set up the as400 transaction
as400Tx = as400Session.beginTransaction();

rows = ( (Integer) as400Session.iterate("select count(*) from As400Vin").next() ).intValue();

Query q = as400Session.createQuery(VINS);
q.setMaxResults(50000);
ScrollableResults result = q.scroll();
int count = 0;
while ( result.next() ) {
As400Vin as400Vin = (As400Vin) result.get(0);
Vin oraVin = new Vin();
BeanUtils.copyProperties(oraVin, as400Vin);

//save the oracle object
oraSession.save(oraVin);

if ( ++count % 20 == 0 || count == rows) {
log.info("Flushing/Clearing sessions " + count);
as400Session.flush();
as400Session.clear();
oraSession.flush();
oraSession.clear();
}
}

//close Oracle and AS400 sessions
oraTx.commit();
oraSession.close();
as400Tx.commit();
as400Session.close();
}
catch(HibernateException ex){
BaseMigration.handleExceptions(ex, oraTx);
throw new RuntimeException("HibernateException", ex);
}
catch(Exception ex){
BaseMigration.handleExceptions(ex, oraTx);
throw new RuntimeException("Exception", ex);
}
return new Integer(rows);


Full stack trace of any exception that occurs: N/A

Name and version of the database you are using: DB2 / AS400 and Oracle 10g

The generated SQL (show_sql=true): N/A

Debug level Hibernate log excerpt: N/A

I know that using ORM for retrieving and saving bulk data is not the suggested way of accomplishing such task, but I am in a position where I don't have much say for the time being. So having gotten that out of the way, I am following the approach suggest by Gavin at: http://blog.hibernate.org/cgi-bin/blosx ... batch.html

My problem is that I am still getting out of memory errors even though I am flushing my sessions (Please look at code above).

I have about 97,000 rows of data that I am retrieving via a ScrollableResults. Whenever I set q.setMaxResults... to 50,000 all 50,000 records get inserted. If I set q.setMaxResults... to 60,000 I get an out of memory error on the 21360th row. I am closing all open hibernate sessions every 20th trip through the loop, so I don't know what else is causing by out of memory errors. My situation is slightly different than the example used at: http://blog.hibernate.org/cgi-bin/blosx ... batch.html in the sense that I need a Hibernate session for DB2 and a different session for Oracle open at the same time.

Any help is greatly appreciated.

Thanks


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 1 post ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.