The forums already contain a large number of posts on the topic of scrolling through a large data set.
Unfortunately I have not been able to find an answer. Our problem is that we experience
OutOfMemoryException when we try to iterate through a large data set. The problem seems to be in
the Oracle JDBC driver. We are using version (10.2.0.4)
The same unit test run with Hypersonic db runs just fine.
Using JVisualVM, we see that not all Oracle JDBC driver objects get cleared. We see:
oracle.jdbc.driver.T4CPreparedStatement (100007 instances after 10000 iterations)
oracle.jdbc.driver.OracleResultSetImpl (10001 instances after 10000 iterations)
objects floating around.
By reading documentation, we already did all of the following:
- Set up Manual flushing and clearing.
- Disable the cache.
- Setting the cursor into FORWARD_ONLY mode.
We are using:
Oracle 10g (JDBC drivers 10.2.0.4)
Hibernate 3.2.5.GA
Spring 2.5.6
My questions?
- Has anynody encountered this problem before?
- Is there some sort of solution?
- Is the problem indeed in the Oracle JDBC driver?
The code of the unit test:
Code:
package com.hi.dao;
import java.sql.SQLException;
import javax.annotation.Resource;
import org.hibernate.CacheMode;
import org.hibernate.FlushMode;
import org.hibernate.Query;
import org.hibernate.ScrollMode;
import org.hibernate.ScrollableResults;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.junit.Ignore;
import org.junit.Test;
import static org.junit.Assert.*;
import org.junit.runner.RunWith;
import org.springframework.orm.hibernate3.SessionFactoryUtils;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import com.hi.dataimprover.core.BdsContact;
import com.hi.dataimprover.core.BdsException;
import com.hi.dataimprover.installer.DatabaseCreator;
/**
* This test generates a table with 1 million rows and tries to
* iterate through it using a scrollable Hibernate query.
*
*/
@RunWith(value=SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/configuration-test.xml","/dao.xml","/jndi-test.xml","/dao-installer.xml"})
public class TestLargeDataSet {
// fixtures
@Resource
DatabaseCreator databaseCreator;
@Resource
private Dao dao;
@Test
public void testDummy() {
assertTrue("This dummy test should always succeed.",true);
}
@Test
public void testIterateThroughLargeDataset() throws SQLException, BdsException {
databaseCreator.create();
generateManyRecords();
iterateThroughManyRecords();
}
private void generateManyRecords() throws BdsException {
for(int i=0 ; i<10000 ; i++) {
BdsContact c = new BdsContact();
dao.saveOrUpdate(c);
if(i % 10000==0) {
System.out.println("Inserted "+i+" items.");
}
}
}
private void iterateThroughManyRecords() {
String query = "from BdsContact";
SessionFactory sessionFactory = dao.getSessionFactory();
Session session = SessionFactoryUtils.getSession(sessionFactory,true);
session.setFlushMode(FlushMode.MANUAL);
session.setCacheMode(CacheMode.IGNORE);
Query q = session.createQuery(query);
ScrollableResults scrollable = q.scroll(ScrollMode.FORWARD_ONLY);
long cursorCounter = 0;
while(scrollable.next()) {
BdsContact c = (BdsContact) scrollable.get(0);
cursorCounter++;
if(cursorCounter % 1000==0) {
session.flush();
session.clear();
}
if(cursorCounter % 10000==0) {
System.out.println("Retrieved "+cursorCounter+" items.");
}
}
session.close();
System.out.println("Finished. Retrieved "+cursorCounter+" items");
}
}