Hibernate Books

All times are UTC - 5 hours [ DST ]



Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: Mass Indexer is extremely slow
PostPosted: Mon Mar 26, 2012 9:41 am 
Newbie

Joined: Mon Mar 26, 2012 9:12 am
Posts: 1
Hi,
I'm trying to build my index with the massIndexer, and i'm facing some strange problems. The indexed entity is very simple (i.e. One numeric field for ID and one String field) to index. When I launch the massindexer with startAndWait() function, the indexing takes eternity !!! I have like 300 000 entity to index and i can see that massIndexer can only index like 200 entity per minutes. Plus, I noticed in the console log that the hibernate query are running fast at the start, but after few seconds, the console output doesn't show anything. i.e., the first hundred (maybe more, maybe less) requests are super fast, but then it's "blocked" for at least 20s to 1mn. And so on for the rest of the indexing.
I've got the same results with the start() function. The field are annotated by
Code:
@DocumentId
for my id, and
Code:
@Field(name = "raisonSociale_raw", index = Index.NO, store = Store.YES)
for the string field.

Here is the code i'm using for indexing.
c is the entity class I want to index .
Code:
MassIndexer mi = getFullTextSession().createIndexer(c);
          mi.batchSizeToLoadObjects(30)//
          .threadsForSubsequentFetching(4)// .threadsToLoadObjects(10)//
          .purgeAllOnStart(true)// .optimizeAfterPurge(true)//
          .optimizeOnFinish(true).startAndWait();//


inb4 "this is a connection pool problem", this is not, i've increased the number of connection to 30 for minimum and 100 for max.
Any advice will be appreciated.


Top
 Profile  
 
 Post subject: Re: Mass Indexer is extremely slow
PostPosted: Mon Mar 26, 2012 10:58 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
Hi,
what database is it?

Could you monitor your heap ? I guess it's running out of memory; you might need to adjust the JVM parameters to give it more memory, or reduce the number of threads.

If it's MySQL, then you need to set some options on the driver or that JDBC driver will attempt to load the whole database in memory.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Mass Indexer is extremely slow
PostPosted: Thu Mar 29, 2012 4:43 am 
Regular
Regular

Joined: Thu Jun 16, 2011 12:03 pm
Posts: 94
Are your hibernate relationships setted to lazy or not?

i had this problem i had to change all my relationships with lazy=true. Massindexer was so slow.

and more, i had to use the manual indexer:

http://docs.jboss.org/hibernate/search/ ... index.html

Hibernator,


Top
 Profile  
 
 Post subject: Re: Mass Indexer is extremely slow
PostPosted: Thu Mar 29, 2012 5:46 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
the updated link is http://docs.jboss.org/hibernate/search/ ... htoindexes

Hibernator, did you identify why the MassIndexer was slow for you? It is supposed to be significantly faster than the manual one, unless you' re loading a very small data set.

lazyness of relationship does affect the indexing performance, but not significantly, unless it's loading entities which are not needed for the indexing process.

More often the problem is with MySQL because of it's bugs on scrolling large resultsets.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.