Out of memory error for mass tests.

ph_w · **Posted:** Sat Mar 06, 2004 8:06 am

Hello,

I have trouble trying to have charge tests using hibernate.
The problem is that when I insert lots of data (20.000 records), I get "out of memory error" from the jvm.

Obvious code for doing this is:

Code:

Configuration cfg = new Configuration();
cfg.addClass(Blsmatst.class);
SessionFactory sf = cfg.buildSessionFactory();
session = sf.openSession();
int maxRecord = 100000;
Blsmatst record;
int i;
for (i = 0; i < maxRecord; i++){
   record = createNewBlsmatst(i);
   session.save(record);
}
session.flush();
session.connection().commit();

with createNewBlsmatst(...) filling each field of a newly created record.

Next, I thought: "too much data in the session makes it blow".
Sounds reasonable. then, I inserted a memory cleaner once in a while:

Code:

Configuration cfg = new Configuration();
cfg.addClass(Blsmatst.class);
SessionFactory sf = cfg.buildSessionFactory();
session = sf.openSession();
int maxRecord = 100000;
Blsmatst record;
int i;
for (i = 0; i < maxRecord; i++){
   record = createNewBlsmatst(i);
   session.save(record);
   if (i % 10000 == 0){
      session.flush();
      session.clear();
   }
}
session.flush();
session.connection().commit();

And it worked better, but same problem occurred only later (around 100.000 record)

Calling garbage collector once in a while did not change anything ...
But closing connection and reopening it once in a while made me allow to test up to 10.000.000 inserts with no problems.

Here is final code:

Code:

Configuration cfg = new Configuration();
cfg.addClass(Blsmatst.class);
SessionFactory sf = cfg.buildSessionFactory();
session = sf.openSession();
int maxRecord = 100000;
Blsmatst record;
int i;
for (i = 0; i < maxRecord; i++){
   record = createNewBlsmatst(i);
   session.save(record);
   if (i % 10000 == 0){
      session.flush();
      session.connection().commit();
      session.clear();
      session.close();
      session = sf.openSession();
      System.gc();
   }
}
session.flush();
session.connection().commit();

The most annoying is that I had same error and had to apply the same tactic for being able to real as much data: close and reopen the session!

My questions are:
Why does the session.clear() not clear it completely?
If so: is there no other way to do it than to destory the session? Because, of course, I would like to be able to work with this very same session for query purposes.

Thanks

FYI:
- I use hibernate 2.1.2
- I use Sun's JVM on Win2000 Pro (jdk 1.4.2_01)
- I use Oracle 9 with oci driver (from ojdbc14.jar)
- the object I use here (Blsmatst) is very simple: one table only.
- Here follows my .hbm.xml file:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 2.0//EN" "http://hibernate.sourceforge.net/hibernate-mapping-2.0.dtd">
<hibernate-mapping>
<class name="test.netbnk.Blsmatst" table="PWA_BLSMATST">
<id name="hdruid" column="HDRUID" type="string">
<generator class="assigned"/>
</id>
<property name="hdrstt" column="HDRSTT" type="string" length="1" not-null="true"/>
<property name="hdrsyn" column="HDRSYN" type="string" length="20" not-null="true"/>
<property name="accidn" column="ACCIDN" type="string" length="36" not-null="true"/>
<property name="baluid" column="BALUID" type="string" length="9" not-null="true"/>
<property name="detuid" column="DETUID" type="string" length="9"/>
<property name="sttdup" column="STTDUP" type="string" length="1"/>
<property name="en1seqnbr" column="EN1SEQNBR" type="string" length="4"/>
<property name="en1detnbr" column="EN1DETNBR" type="string" length="4"/>
<property name="en1reffin" column="EN1REFFIN" type="string" length="13"/>
<property name="en1refext" column="EN1REFEXT" type="string" length="8"/>
<property name="en1trxsgn" column="EN1TRXSGN" type="string" length="1"/>
<property name="en1trxamt" column="EN1TRXAMT" type="string" length="15"/>
<property name="en1trxdat" column="EN1TRXDAT" type="string" length="8"/>
<property name="en1trxcod" column="EN1TRXCOD" type="string" length="8"/>
<property name="en1msgtyp" column="EN1MSGTYP" type="string" length="1"/>
<property name="en1msgzon" column="EN1MSGZON" type="string" length="53"/>
<property name="en1trxent" column="EN1TRXENT" type="string" length="8"/>
<property name="en1seqpap" column="EN1SEQPAP" type="string" length="3"/>
<property name="en1glocod" column="EN1GLOCOD" type="string" length="1"/>
<property name="en1codnxt" column="EN1CODNXT" type="string" length="1"/>
<property name="en1filcod" column="EN1FILCOD" type="string" length="1"/>
<property name="en1codlnk" column="EN1CODLNK" type="string" length="1"/>
<property name="en2seqnbr" column="EN2SEQNBR" type="string" length="4"/>
<property name="en2detnbr" column="EN2DETNBR" type="string" length="4"/>
<property name="en2msgzon" column="EN2MSGZON" type="string" length="53"/>
<property name="en2cliref" column="EN2CLIREF" type="string" length="26"/>
<property name="en2oricur" column="EN2ORICUR" type="string" length="3"/>
<property name="en2oriamt" column="EN2ORIAMT" type="string" length="15"/>
<property name="en2codnxt" column="EN2CODNXT" type="string" length="1"/>
<property name="en2codlnk" column="EN2CODLNK" type="string" length="1"/>
<property name="en3seqnbr" column="EN3SEQNBR" type="string" length="4"/>
<property name="en3detnbr" column="EN3DETNBR" type="string" length="4"/>
<property name="en3benacc" column="EN3BENACC" type="string" length="12"/>
<property name="en3bencod" column="EN3BENCOD" type="string" length="10"/>
<property name="en3benacx" column="EN3BENACX" type="string" length="15"/>
<property name="en3bencrd" column="EN3BENCRD" type="string" length="78"/>
<property name="en3codlnk" column="EN3CODLNK" type="string" length="1"/>
<property name="trxcur" column="TRXCUR" type="string" length="3" not-null="true"/>
</class>
</hibernate-mapping>

jlawmi · **Joined:** Mon Nov 24, 2003 6:36 pm **Posts:** 105

you might want to try in each iteration of your loop calling
session.evict(object) instead. I have a unit test that creates about 10k objects (with some children) this way and I think it is working okay...

James

ph_w · **Posted:** Sun Mar 07, 2004 6:26 am

Thank you James.

in my investigations I did think about that.
But session.clear() is said in the reference to stand for evict(all objects). So I used clear().

What's more, your strategy forces us to flush the session after every save, which is (to my tests) heavily time consuming.

FYI, tests I dod on my rather poor testong infrastructure are:
1 flush out of 1.000 insert : 210.000 inserts in 6 minutes
1 flush in each insert : 10.000 (i) inserts in 34 minutes
sample code for this is:

Code:

Configuration cfg = new Configuration();
cfg.addClass(Blsmatst.class);
SessionFactory sf = cfg.buildSessionFactory();
session = sf.openSession();
int maxRecord = 100000; 
Blsmatst record;
int i;
for (i = 0; i < maxRecord; i++){
   record = createNewBlsmatst(new Integer(i));
   session.save(record);
   if (i % 1 == 0){
      session.flush();
      session.evict(record);
      session.connection().commit();
      System.gc();
   }
}
session.flush();
session.connection().commit();

One flush on each insert is just not an option for me having this performances.

Thanks for responding anyway.

Any other lead?

gavin · **Posted:** Sun Mar 07, 2004 7:01 am

Quote:

What's more, your strategy forces us to flush the session after every save, which is (to my tests) heavily time consuming.

I find this incredibly difficult to believe. flush() with no objects loaded is pretty much a no-op!

I notice, however, that in your test code you are committing the transaction for every object, which is certainly NOT a no-op. So looks like you are mistaken, and it is the 10 000 commits which makes the difference.

Quote:

Why does the session.clear() not clear it completely?

Cos there is certain book-keeping to be done in afterTransactionCompletion() or, since you have chose not to use the Transaction API, at close() time. You might get better performance by using Hibernate Transaction API.

Quote:

Because, of course, I would like to be able to work with this very same session for query purposes.

Why "of course"? I can't think of any good reason why you need to be always in the same session in this usecase.

jlawmi · **Joined:** Mon Nov 24, 2003 6:36 pm **Posts:** 105

Possibly this will help? Here's my code- I can create about 1000 objects in about 1 minute, AND i'm committing after each object, which i'm sure doesn't help performance at all. (I'm using HIbernate transaction)

Code:

Transaction tx = null;
      Session sess = null;
      try {
         
         sess = Manager.getSession();
         tx = sess.beginTransaction();
         allFeatures = LookupManager.findPropertyFeatureList();
         for (int i = 0; i < NUMBER_OF_PROPS; i++) {
            Property prop = createTestPropertyForDirectSave(profile);
            
            randomizeFeatures(prop);
            setRandomZipCode(prop, i);
            props[i] = prop;
            sess.save(prop);
            tx.commit();
            sess.evict(prop);
            
         }
      } catch (Exception e) {
         e.printStackTrace();
         throw e;
      } finally {
         Manager.closeConnection(sess);
      }

ph_w · **Posted:** Mon Mar 08, 2004 4:39 am

Thank your for responding Gavin & James.

Quote:

believe. flush() with no objects loaded is pretty much a no-op

Having read this from you, I double checked, and it appeared that the most time consuming was the Garbage Collector (System.gc()) I put for testing purposes. (FYI: ~3 milisec for flush; ~4 milisec for commit; ~60 milisec for gc).

Quote:

session.clear()

Thanks for the explanation (I had not seen this subtility it in the doc): it was the missing link: Transaction.
When I use transcation API, session.clear() works as I wish it to un to 1000.000 records.
sample of my working code is:
(btw, thank you James for sample code. I did use it.)

Code:

Configuration cfg = new Configuration(); 
cfg.addClass(Blsmatst.class); 
SessionFactory sf = cfg.buildSessionFactory(); 
session = sf.openSession(); 
Transaction tx = session.beginTransaction();
int maxRecord = 1000000; 
Blsmatst record; 
int i; 
for (i = 0; i < maxRecord; i++){ 
   record = createNewBlsmatst(new Integer(i)); 
   session.save(record); 
   if (i % 10000 == 0){ 
      tx.commit();
      session.clear()
   } 
} 

This procedure also works with my select queries.

Quote:

Why "of course"? ... always in the same session in this usecase.

Simply because (as mentionned initially) after inserting records, I have to read and manipulate part or all of them. So I use queries which belong to the session. If I close the session, I also must recreate queries, which does not feel acceptable to me.

Thank you guys, this was the thing.

michael · **Posted:** Mon Mar 08, 2004 4:59 am

What do you mean by recreate queries? Querys dont 'belong' to a session, they are cached on SessionFactory level.

ph_w · **Posted:** Tue Mar 09, 2004 6:01 am

Hello Michael,

I'm not sure its the same issue, but here are the details of why I felt like queries belong to session:

Sample code is:

Code:

Configuration cfg = new Configuration();
cfg.addClass(Blsmatst.class);
SessionFactory sf = cfg.buildSessionFactory();
session = sf.openSession();
Transaction tx = session.beginTransaction();
   
List countList = session.find("select count(*) from Blsmatst");
int count = ((Integer)(countList.get(0))).intValue();
Query q = session.createQuery("from Blsmatst");      
for (i = 0; i < count; i = i + fecthNumber){
   q.setFirstResult(i);
   q.setMaxResults(fecthNumber);
   List Blsmatsts = q.list();
   session.clear();
   session.close();
   session = sf.openSession();
   System.gc();         
}

And when accessing the second time "List Blsmatsts = q.list();" I get this exception:

Code:

net.sf.hibernate.HibernateException: Session is closed
   at net.sf.hibernate.impl.SessionImpl.connection(SessionImpl.java:3250)
   at net.sf.hibernate.impl.BatcherImpl.prepareQueryStatement(BatcherImpl.java:65)
   at net.sf.hibernate.loader.Loader.prepareQueryStatement(Loader.java:704)
   at net.sf.hibernate.loader.Loader.doQuery(Loader.java:185)
   at net.sf.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:133)
   at net.sf.hibernate.loader.Loader.doList(Loader.java:950)
   at net.sf.hibernate.loader.Loader.list(Loader.java:941)
   at net.sf.hibernate.hql.QueryTranslator.list(QueryTranslator.java:834)
   at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1512)
   at net.sf.hibernate.impl.QueryImpl.list(QueryImpl.java:39)
   at test.Main.main(Main.java:64)

Should you need any more info, just ask.
If I did anything wring, please tell me.
Note that I worked this around using the transaction API and not closing the session, so now, I'm fine :-)

gavin · **Posted:** Tue Mar 09, 2004 6:06 am

Y'know, one of the nice things about open source software is that you can always look at the code.

Its quite easy to verify that what Michael says is correct. The object that implements Query is quite different to the object that actually does the querying and caches the generated SQL.

michael · **Posted:** Tue Mar 09, 2004 9:07 am

You are correct in your impression that you can't reuse the Query object. But you can easily recreate the query in a new session, which does not bring a performance hit (compiled queries sql is cached at SessionFactory level)