-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

View unanswered posts | View active topics


Board index » Projects » Search, Validator, Shards

All times are UTC - 5 hours [ DST ]


Forum locked This topic is locked, you cannot edit posts or make further replies.  Page 1 of 1
  [ 4 posts ] 
  Previous topic | Next topic 
Author Message
apachemaven
 Post subject: outofmemoryerror when indexing using hibernate-search.txt
PostPosted: Wed Dec 15, 2010 4:49 am 
Beginner
Beginner

Joined: Fri Nov 05, 2010 7:25 am
Posts: 21
Hi:

I am using hibernate-serach 3.2.1 in my appliction,and I create a util class for manually indexing:

---------------------------------
Code:
public class Indexer {
   public static void main(String[] args) {
      try {
         HibernateUtil.init();
         Search.getFullTextSession(HibernateUtil.getSessionFactory().getCurrentSession()).createIndexer().startAndWait();
      } catch (HibernateException e) {
         e.printStackTrace();
      } catch (InterruptedException e) {
         e.printStackTrace();
      }
   }
}

-----------------------------------

However I got the following exception,outofmemory,I know this exception,the indexing work depleted the memory of the machine.

My memory is 1G,and it is not possible to change my memory at the moment.

So I wonder if there is any parameter can be used to control the memory deplete in indexing work?

Thougth,I have told you guys it is a outofmemroyerror,but I think it maybe better to post the statck information:

-------------------------------
Code:
Exception in thread "Hibernate Search: identifierloader-1" java.lang.OutOfMemoryError: Java heap space
        at java.lang.AbstractStringBuilder.<init>(Unknown Source)
        at java.lang.StringBuffer.<init>(Unknown Source)
        at org.hibernate.criterion.SimpleProjection.getColumnAliases(SimpleProjection.java:74)
        at org.hibernate.loader.criteria.CriteriaQueryTranslator.getProjectedColumnAliases(CriteriaQueryTranslator.java:366)
        at org.hibernate.loader.criteria.CriteriaLoader.getResultColumnOrRow(CriteriaLoader.java:130)
        at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:647)
        at org.hibernate.loader.Loader.loadSingleRow(Loader.java:307)
        at org.hibernate.impl.ScrollableResultsImpl.prepareCurrentRow(ScrollableResultsImpl.java:254)
        at org.hibernate.impl.ScrollableResultsImpl.next(ScrollableResultsImpl.java:123)
        at org.hibernate.search.batchindexing.IdentifierProducer.loadAllIdentifiers(IdentifierProducer.java:136)
        at org.hibernate.search.batchindexing.IdentifierProducer.inTransactionWrapper(IdentifierProducer.java:103)
        at org.hibernate.search.batchindexing.IdentifierProducer.run(IdentifierProducer.java:91)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Exception in thread "Hibernate Search: entityloader-1" java.lang.OutOfMemoryError: Java heap space
        at com.mysql.jdbc.MysqlIO.unpackField(MysqlIO.java:765)
        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:416)
        at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2581)
        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1757)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2171)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2113)
        at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2275)
        at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
        at org.hibernate.loader.Loader.getResultSet(Loader.java:1849)
        at org.hibernate.loader.Loader.doQuery(Loader.java:718)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
        at org.hibernate.loader.Loader.doList(Loader.java:2294)
        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2172)
        at org.hibernate.loader.Loader.list(Loader.java:2167)
        at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
        at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1706)
        at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
        at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.loadList(IdentifierConsumerEntityProducer.java:133)
        at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.loadAllFromQueue(IdentifierConsumerEntityProducer.java:102)
        at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.run(IdentifierConsumerEntityProducer.java:84)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Exception in thread "Hibernate Search: entityloader-2" org.hibernate.InstantiationException: Could not instantiate entity: com.search.model.entity.Post
        at org.hibernate.tuple.PojoInstantiator.instantiate(PojoInstantiator.java:114)
        at org.hibernate.tuple.PojoInstantiator.instantiate(PojoInstantiator.java:123)
        at org.hibernate.tuple.entity.AbstractEntityTuplizer.instantiate(AbstractEntityTuplizer.java:606)
        at org.hibernate.persister.entity.AbstractEntityPersister.instantiate(AbstractEntityPersister.java:3888)
        at org.hibernate.impl.SessionImpl.instantiate(SessionImpl.java:1412)
        at org.hibernate.impl.SessionImpl.instantiate(SessionImpl.java:1401)
        at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1342)
        at org.hibernate.loader.Loader.getRow(Loader.java:1251)
        at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:619)
        at org.hibernate.loader.Loader.doQuery(Loader.java:745)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
        at org.hibernate.loader.Loader.doList(Loader.java:2294)
        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2172)
        at org.hibernate.loader.Loader.list(Loader.java:2167)
        at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
        at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1706)
        at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
        at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.loadList(IdentifierConsumerEntityProducer.java:133)
        at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.loadAllFromQueue(IdentifierConsumerEntityProducer.java:102)
        at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.run(IdentifierConsumerEntityProducer.java:84)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedConstructorAccessor3.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at org.hibernate.tuple.PojoInstantiator.instantiate(PojoInstantiator.java:111)
        ... 22 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.HashSet.<init>(Unknown Source)
        at com.search.model.entity.Post.<init>(Post.java:82)
        ... 26 more


-------------------------
Top
 Profile  
 
sanne.grinovero
 Post subject: Re: outofmemoryerror when indexing using hibernate-search.txt
PostPosted: Wed Dec 15, 2010 8:28 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hello,
are you storing the index in memory or on disk?
the MassIndexer is quite memory hungry, but also the Lucene index needs much memory, also the data you're loading might be quite complex so for each document your application might be loading from some short strings to a huge set of blobs.
I'd suggest to reduce the number of threads being used and also configure the index parameters to use less memory:

Code:
fullTextSession
.createIndexer()
.batchSizeToLoadObjects( 2 )
.cacheMode( CacheMode.IGNORE )
.threadsToLoadObjects( 2 )
.threadsForSubsequentFetching( 2 )
.batchSizeToLoadObjects( 5 )
.startAndWait();


Quote:
hibernate.search.default.indexwriter.batch.ram_buffer_size=16


And of course avoid the RAMDirectoryProvider, and make sure if you're using Hibernate's second level cache to have sensible limits setup for it.

_________________
Sanne
http://in.relation.to/

Top
 Profile  
 
apachemaven
 Post subject: Re: outofmemoryerror when indexing using hibernate-search.txt
PostPosted: Wed Dec 15, 2010 8:55 am 
Beginner
Beginner

Joined: Fri Nov 05, 2010 7:25 am
Posts: 21
s.grinovero wrote:
Hello,
are you storing the index in memory or on disk?
FSDirectory.

the MassIndexer is quite memory hungry, but also the Lucene index needs much memory, also the data you're loading might be quite complex so for each document your application might be loading from some short strings to a huge set of blobs.
I'd suggest to reduce the number of threads being used and also configure the index parameters to use less memory:

Code:
fullTextSession
.createIndexer()
.batchSizeToLoadObjects( 2 )
.cacheMode( CacheMode.IGNORE )
.threadsToLoadObjects( 2 )
.threadsForSubsequentFetching( 2 )
.batchSizeToLoadObjects( 5 )
.startAndWait();


Thanks,I will have a try.

But I wonder how should I know these parameters? Is there any docs? Since it is not a good idea to ask once meet these kind of problems.

Quote:
hibernate.search.default.indexwriter.batch.ram_buffer_size=16


And of course avoid the RAMDirectoryProvider, and make sure if you're using Hibernate's second level cache to have sensible limits setup for it.
Top
 Profile  
 
sanne.grinovero
 Post subject: Re: outofmemoryerror when indexing using hibernate-search.txt
PostPosted: Wed Dec 15, 2010 9:02 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Quote:
Thanks,I will have a try.

But I wonder how should I know these parameters? Is there any docs? Since it is not a good idea to ask once meet these kind of problems.

I hope you have found these:
http://docs.jboss.org/hibernate/stable/ ... ml_single/
But right I need to improve the docs in MassIndexer, so feedback is very welcome.

_________________
Sanne
http://in.relation.to/

Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  Page 1 of 1
  [ 4 posts ] 

Board index » Projects » Search, Validator, Shards

All times are UTC - 5 hours [ DST ]

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
cron
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.