-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 
Author Message
 Post subject: OutOfMemoryException using scroll() and Oracle10g
PostPosted: Wed May 27, 2009 9:29 am 
Newbie

Joined: Tue May 26, 2009 2:17 am
Posts: 3
The forums already contain a large number of posts on the topic of scrolling through a large data set.
Unfortunately I have not been able to find an answer. Our problem is that we experience
OutOfMemoryException when we try to iterate through a large data set. The problem seems to be in
the Oracle JDBC driver. We are using version (10.2.0.4)
The same unit test run with Hypersonic db runs just fine.

Using JVisualVM, we see that not all Oracle JDBC driver objects get cleared. We see:
oracle.jdbc.driver.T4CPreparedStatement (100007 instances after 10000 iterations)
oracle.jdbc.driver.OracleResultSetImpl (10001 instances after 10000 iterations)
objects floating around.

By reading documentation, we already did all of the following:
- Set up Manual flushing and clearing.
- Disable the cache.
- Setting the cursor into FORWARD_ONLY mode.

We are using:
Oracle 10g (JDBC drivers 10.2.0.4)
Hibernate 3.2.5.GA
Spring 2.5.6

My questions?
- Has anynody encountered this problem before?
- Is there some sort of solution?
- Is the problem indeed in the Oracle JDBC driver?


The code of the unit test:

Code:
package com.hi.dao;
import java.sql.SQLException;
import javax.annotation.Resource;
import org.hibernate.CacheMode;
import org.hibernate.FlushMode;
import org.hibernate.Query;
import org.hibernate.ScrollMode;
import org.hibernate.ScrollableResults;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.junit.Ignore;
import org.junit.Test;
import static org.junit.Assert.*;
import org.junit.runner.RunWith;
import org.springframework.orm.hibernate3.SessionFactoryUtils;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.hi.dataimprover.core.BdsContact;
import com.hi.dataimprover.core.BdsException;
import com.hi.dataimprover.installer.DatabaseCreator;

/**
* This test generates a table with 1 million rows and tries to
* iterate through it using a scrollable Hibernate query.
*
*/
@RunWith(value=SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/configuration-test.xml","/dao.xml","/jndi-test.xml","/dao-installer.xml"})
public class TestLargeDataSet {

   // fixtures
   @Resource
   DatabaseCreator databaseCreator;

   @Resource
   private Dao dao;

   @Test
   public void testDummy() {
      assertTrue("This dummy test should always succeed.",true);
   }
   
   @Test
   public void testIterateThroughLargeDataset() throws SQLException, BdsException {
      databaseCreator.create();
      generateManyRecords();
      iterateThroughManyRecords();
   }

   private void generateManyRecords() throws BdsException {
      for(int i=0 ; i<10000 ; i++) {
         BdsContact c = new BdsContact();
         dao.saveOrUpdate(c);
         if(i % 10000==0) {
            System.out.println("Inserted "+i+" items.");
         }
      }
   }
   
   private void iterateThroughManyRecords() {
      String query = "from BdsContact";
      SessionFactory sessionFactory = dao.getSessionFactory();
      Session session = SessionFactoryUtils.getSession(sessionFactory,true);
      session.setFlushMode(FlushMode.MANUAL);
      session.setCacheMode(CacheMode.IGNORE);
      Query q = session.createQuery(query);
      ScrollableResults scrollable = q.scroll(ScrollMode.FORWARD_ONLY);
      long cursorCounter = 0;
      while(scrollable.next()) {
         BdsContact c = (BdsContact) scrollable.get(0);
         cursorCounter++;
         if(cursorCounter % 1000==0) {
            session.flush();
            session.clear();
         }
         if(cursorCounter % 10000==0) {
            System.out.println("Retrieved "+cursorCounter+" items.");
         }
      }
      session.close();
      System.out.println("Finished. Retrieved "+cursorCounter+" items");
   }
}


Top
 Profile  
 
 Post subject: Re: OutOfMemoryException using scroll() and Oracle10g
PostPosted: Wed Jun 03, 2009 9:36 am 
Newbie

Joined: Tue May 26, 2009 2:17 am
Posts: 3
I have experimented by using the Oracle11 drivers in combination with an Oracle10g database. The result is that there no longer is a leak in the Oracle driver classes (oracle.jdbc.*). Unfortunately, now two hibernate classes have started to accumulate:

org.hibernate.collection.PersistentSet
org.hibernate.action.EntityUpdateAction

These classes still cause an OutOfMemoryException.

Any thoughts please?


Top
 Profile  
 
 Post subject: Re: OutOfMemoryException using scroll() and Oracle10g
PostPosted: Fri Jun 05, 2009 10:22 am 
Newbie

Joined: Tue May 26, 2009 2:17 am
Posts: 3
Solved. Removing the flush() also removed the leak.

We noticed that different HibernateDialects do not give the same results. Also, using different JDBC drivers yield different outcomes.
In particular, when using a scrollable result set as shown in the example code (with the flush()):

Oracle10gDialect in combination with JDBC drivers version 10.2.0.4 - Oracle JDBC drivers starts leaking the classes oracle.jdbc.driver.T4CPreparedStatement and oracle.jdbc.driver.OracleResultSetImpl

Oracle10gDialect in combination with JDBC drivers 11.1.0.7.0 - Hibernate starts leaking the classes org.hibernate.collection.PersistentSet and org.hibernate.action.EntityUpdateAction

SQLServerDialect in combination with the jtds drivers 1.2.2 - No leak.

These are all the combinations I tried. I think your suggestion that the flush() fools hibernate into thinking that the scroll is not read only, but not with all HibernateDialects and/or drivers.


Top
 Profile  
 
 Post subject: Re: OutOfMemoryException using scroll() and Oracle10g
PostPosted: Sun May 01, 2011 6:50 am 
Newbie

Joined: Sun Aug 30, 2009 10:07 am
Posts: 3
Dear Hans
In this sample code you can remove flush ()! Because this code just iterate the result set but if you want to manipulate or modify records of result set or save new objects you can’t call clear() without flush() and if you remove flush all new object and modified object was Ignored by hibernate.


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.