Hibernate version: 3.2.6
Postgres: 8.3-603
Hi,
I need help with loading huge amount of rows from database. Correctly I need load 2 million rows from db and save it to the collection (for example map or list ...). Problem is with hibernate memory restriction. I tried advices from hibernate manual:
Code:
//define session and transaction
Session session = getSessionFactory().openSession();
Transaction transaction = session.beginTransaction();
//DB query for load all rows
StringBuffer hql = new StringBuffer();
hql.append("SELECT entity ");
hql.append(" FROM " + Entity.class.getName() + " entity ");
hql.append(" WHERE 1=1 ");
Query query = session.createQuery(hql.toString());
//counter
Integer a = 1;
//scroll
ScrollableResults allItems = query.setCacheMode(CacheMode.IGNORE).scroll(ScrollMode.FORWARD_ONLY);
Map<Long, Entity> itemsId = new HashMap<String, Entity>();
//iterace
while (allItems.next()) {
//work with line
Entity itemTmp = (Entity)allItems.get(0);
itemsId.put(itemTmp.getId(),itemTmp);
session.evict(itemTmp);
//save and clean
if (a % 200 == 0){
session.flush();
session.clear();
}
a++;
}
but it crash somewere after 15 200. row :
Code:
org.springframework.orm.hibernate3.HibernateSystemException: Could not instantiate entity: com.project.Entity; nested exception is org.hibernate.InstantiationException: Could not instantiate entity: com.project.Entity
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:661)
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:412)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:117)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:113)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:113)
at org.springframework.aop.interceptor.DebugInterceptor.invoke(DebugInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
From debug information I found out its outOfMemoryError. I dont undertan why it behives like that. I'm using batch proccesing with imports (1-2 million rows) and it's
work fine. Can anybody help me ?
regards