-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 7 posts ] 
Author Message
 Post subject: Problem with loading huge amount of rows from database
PostPosted: Thu Jul 17, 2008 9:19 am 
Newbie

Joined: Wed Jun 18, 2008 5:46 am
Posts: 7
Hibernate version: 3.2.6

Postgres: 8.3-603


Hi,

I need help with loading huge amount of rows from database. Correctly I need load 2 million rows from db and save it to the collection (for example map or list ...). Problem is with hibernate memory restriction. I tried advices from hibernate manual:

Code:
//define session and transaction
Session session = getSessionFactory().openSession();
Transaction transaction = session.beginTransaction();

//DB query for load all rows
        StringBuffer hql = new StringBuffer();
hql.append("SELECT entity ");
hql.append(" FROM " + Entity.class.getName() + " entity ");
hql.append(" WHERE 1=1 ");
Query query = session.createQuery(hql.toString());

//counter
Integer a = 1;

//scroll
ScrollableResults allItems = query.setCacheMode(CacheMode.IGNORE).scroll(ScrollMode.FORWARD_ONLY);
Map<Long, Entity> itemsId = new HashMap<String, Entity>();

//iterace
while (allItems.next()) {
   
    //work with line
    Entity itemTmp = (Entity)allItems.get(0);
    itemsId.put(itemTmp.getId(),itemTmp);
    session.evict(itemTmp);
   
    //save and clean
    if (a % 200 == 0){
        session.flush();
        session.clear();
    }
    a++;
}


but it crash somewere after 15 200. row :


Code:
org.springframework.orm.hibernate3.HibernateSystemException: Could not instantiate entity: com.project.Entity; nested exception is org.hibernate.InstantiationException: Could not instantiate entity: com.project.Entity
   at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:661)
   at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:412)
   at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:117)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:113)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:113)
   at org.springframework.aop.interceptor.DebugInterceptor.invoke(DebugInterceptor.java:61)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)


From debug information I found out its outOfMemoryError. I dont undertan why it behives like that. I'm using batch proccesing with imports (1-2 million rows) and it's
work fine. Can anybody help me ?

regards


Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 17, 2008 9:59 am 
Newbie

Joined: Thu Jul 17, 2008 7:49 am
Posts: 5
I don't think its a good idea to load such a huge amount of objects into your JVM, if hibernate doesnt stop you, the memory restrictions of the VM will (sooner or later). Why do you need that huge amount of objects? Maybe theres a better solution for this.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 17, 2008 10:42 am 
Newbie

Joined: Fri May 23, 2008 5:43 am
Posts: 13
Location: Switzerland
Do you set the amount of memory that can be used by JVM? Something like -Xmx1048m
(1Gb for Heap)

If you have 2 million row and lets say each row uses 500 byte then JVM will use 1Gb of memory. If this amount is too much for Hibernate I quit programming Java forever.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 17, 2008 10:51 am 
Newbie

Joined: Thu Jul 17, 2008 7:49 am
Posts: 5
if I didnt miscalculate what eh wrote above then (taking into account the default jvm limit of 128MB) his rows use more than 8kb each which means he will probably end up needing 16GB of memory

(very rough calculation)


Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 17, 2008 11:46 am 
Newbie

Joined: Fri May 23, 2008 5:43 am
Posts: 13
Location: Switzerland
How did you get to the number 8K? As far as I know Java uses 16 bytes for each object, Hibernate adds some overhead, let's say another 16 bytes. So in your 8K one can fit about 2000 integer columns. Getting 2M rows of text information at once sounds indeed too much, but small rows for some calculations should be ok


Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 17, 2008 11:56 am 
Newbie

Joined: Fri May 23, 2008 5:43 am
Posts: 13
Location: Switzerland
Sorry, resubmitted the form. How do I delete this post?


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jul 18, 2008 2:50 am 
Newbie

Joined: Thu Jul 17, 2008 7:49 am
Posts: 5
lock51 wrote:
How did you get to the number 8K? As far as I know Java uses 16 bytes for each object, Hibernate adds some overhead, let's say another 16 bytes. So in your 8K one can fit about 2000 integer columns. Getting 2M rows of text information at once sounds indeed too much, but small rows for some calculations should be ok


he stated that he ran out of memory at 15200 rows already, so I did the math ;)

well actually the rows should be less big in size but I think they contain text information. I'm not sure was he wants to accomplish but I'm sure there is a better solution for this.


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 7 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.