-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 2 posts ] 
Author Message
 Post subject: what is the best practice to work with large datasets?
PostPosted: Thu Feb 23, 2006 10:01 am 
Beginner
Beginner

Joined: Tue Jan 18, 2005 9:37 am
Posts: 29
Location: The Netherlands
I am using hibernate for a weblogging statistic application.
There is a method that writes a textfile with a selection of records in the database.

The problem is that Hibernate tries to load all records into the heap memory. This will go very slowly and even results in out of heapspace errors.

What is the best way to do this? Maybe I should use some sort of Buffer?

Bas



Hibernate version: 3.0

Code:
   private void saveLogFile() {
        Session s = this.sessionFactory.openSession();
        try{
            BufferedWriter out = new BufferedWriter ( new FileWriter ( 'c:/log.txt' ) );

            Criteria q  = s.createCriteria(LogString.class);
            q.add(Restrictions.isNull("site"));

            // go back one month
            GregorianCalendar cal = new GregorianCalendar();
            cal.add(Calendar.MONTH, -1);
            Date startDate = cal.getTime();
            q.add(Restrictions.gt("timeStamp", startDate));
            q.addOrder(Order.asc("timeStamp"));
            for (Iterator iter = q.list().iterator(); iter.hasNext();) {
                LogString ls = (LogString) iter.next();
                out.write(ls.getLogString()+"\r\n");
            }
            out.close();
        }catch(Exception e){
            log.error("Error saving logfile",e);
        }finally{
            s.close();
        }
    }





Quote:
14:57:30,343 INFO Environment:460 - Hibernate 3.0
14:57:30,359 INFO Environment:478 - loaded properties from resource hibernate.properties: {hibernate.connection.username=sa, hibernate.connection.password=****, hibernate.cglib.use_reflection_optimizer=true, hibernate.dialect=org.hibernate.dialect.SQLServerDialect, hibernate.connection.url=jdbc:jtds:sqlserver://localhost:1433/WebLog, hibernate.connection.driver_class=net.sourceforge.jtds.jdbc.Driver}
14:57:30,359 INFO Environment:506 - using CGLIB reflection optimizer
14:57:30,359 INFO Environment:536 - using JDK 1.4 java.sql.Timestamp handling
14:57:30,500 INFO Configuration:1159 - configuring from resource: /hibernate.cfg.xml
14:57:30,500 INFO Configuration:1130 - Configuration resource: /hibernate.cfg.xml
14:57:30,687 INFO Configuration:440 - Mapping resource: nl/LogString.hbm.xml
14:57:36,531 INFO HbmBinder:258 - Mapping class: nl.LogString -> LogString
14:57:36,703 INFO Configuration:440 - Mapping resource: nl/Site.hbm.xml
14:57:38,578 INFO HbmBinder:258 - Mapping class: nl.Site -> Site
14:57:38,578 INFO Configuration:1271 - Configured SessionFactory: null
14:57:38,578 INFO Configuration:851 - processing extends queue
14:57:38,578 INFO Configuration:855 - processing collection mappings
14:57:38,578 INFO Configuration:864 - processing association property references
14:57:38,578 INFO Configuration:893 - processing foreign key constraints
14:57:38,640 INFO Dialect:91 - Using dialect: org.hibernate.dialect.SQLServerDialect
14:57:38,671 INFO SettingsFactory:90 - Default batch fetch size: 1
14:57:38,671 INFO SettingsFactory:94 - Generate SQL with comments: disabled
14:57:38,671 INFO SettingsFactory:98 - Order SQL updates by primary key: disabled
14:57:38,671 INFO SettingsFactory:273 - Query translator: org.hibernate.hql.ast.ASTQueryTranslatorFactory
14:57:38,671 INFO ASTQueryTranslatorFactory:21 - Using ASTQueryTranslatorFactory
14:57:38,671 INFO SettingsFactory:106 - Query language substitutions: {}
14:57:38,687 INFO DriverManagerConnectionProvider:41 - Using Hibernate built-in connection pool (not for production use!)
14:57:38,687 INFO DriverManagerConnectionProvider:42 - Hibernate connection pool size: 20
14:57:38,687 INFO DriverManagerConnectionProvider:45 - autocommit mode: false
14:57:38,687 INFO DriverManagerConnectionProvider:80 - using driver: net.sourceforge.jtds.jdbc.Driver at URL: jdbc:jtds:sqlserver://localhost:1433/WebLog
14:57:38,687 INFO DriverManagerConnectionProvider:86 - connection properties: {user=sa, password=****}
14:57:38,921 INFO SettingsFactory:156 - Scrollable result sets: enabled
14:57:38,921 INFO SettingsFactory:164 - JDBC3 getGeneratedKeys(): enabled
14:57:38,921 INFO TransactionFactoryFactory:31 - Using default transaction strategy (direct JDBC transactions)
14:57:38,937 INFO TransactionManagerLookupFactory:33 - No TransactionManagerLookup configured (in JTA environment, use of read-write or transactional second-level cache is not recommended)
14:57:38,937 INFO SettingsFactory:176 - Automatic flush during beforeCompletion(): disabled
14:57:38,937 INFO SettingsFactory:179 - Automatic session close at end of transaction: disabled
14:57:38,937 INFO SettingsFactory:260 - Cache provider: org.hibernate.cache.EhCacheProvider
14:57:38,937 INFO SettingsFactory:187 - Second-level cache: enabled
14:57:38,937 INFO SettingsFactory:192 - Optimize cache for minimal puts: disabled
14:57:38,937 INFO SettingsFactory:199 - Structured second-level cache entries: enabled
14:57:38,937 INFO SettingsFactory:203 - Query cache: disabled
14:57:38,937 INFO SettingsFactory:214 - Statistics: disabled
14:57:38,937 INFO SettingsFactory:218 - Deleted entity synthetic identifier rollback: disabled
14:57:38,953 INFO SettingsFactory:232 - Default entity-mode: pojo
14:57:39,125 INFO SessionFactoryImpl:140 - building session factory
14:57:39,140 WARN Configurator:126 - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/D:/WebLogTask/lib/ehcache-1.1.jar!/ehcache-failsafe.xml
14:57:40,031 INFO SessionFactoryObjectFactory:82 - Not binding factory to JNDI, no JNDI name configured
14:57:40,031 INFO SessionFactoryImpl:366 - Checking 0 named queries
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space




Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 23, 2006 10:33 am 
Beginner
Beginner

Joined: Thu Feb 16, 2006 9:53 am
Posts: 24
I guess you can try Pagination or "Scrollable iteration" features :
http://www.hibernate.org/hib_docs/v3/reference/en/html_single/#objectstate-querying-executing-pagination

If your session is getting too big while iterating, you may also clear the session periodically. Session.clear() does "NOT close open iterators or instances of ScrollableResults."
Check Session Javadoc for further info.


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 2 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.