Mass Indexer is extremely slow

Gnarf95 · **Joined:** Mon Mar 26, 2012 9:12 am **Posts:** 1

Hi,
I'm trying to build my index with the massIndexer, and i'm facing some strange problems. The indexed entity is very simple (i.e. One numeric field for ID and one String field) to index. When I launch the massindexer with startAndWait() function, the indexing takes eternity !!! I have like 300 000 entity to index and i can see that massIndexer can only index like 200 entity per minutes. Plus, I noticed in the console log that the hibernate query are running fast at the start, but after few seconds, the console output doesn't show anything. i.e., the first hundred (maybe more, maybe less) requests are super fast, but then it's "blocked" for at least 20s to 1mn. And so on for the rest of the indexing.
I've got the same results with the start() function. The field are annotated by

Code:

@DocumentId

for my id, and

Code:

@Field(name = "raisonSociale_raw", index = Index.NO, store = Store.YES)

for the string field.

Here is the code i'm using for indexing.
c is the entity class I want to index .

Code:

MassIndexer mi = getFullTextSession().createIndexer(c);
          mi.batchSizeToLoadObjects(30)//
          .threadsForSubsequentFetching(4)// .threadsToLoadObjects(10)//
          .purgeAllOnStart(true)// .optimizeAfterPurge(true)//
          .optimizeOnFinish(true).startAndWait();//

inb4 "this is a connection pool problem", this is not, i've increased the number of connection to 30 for minimum and 100 for max.
Any advice will be appreciated.

sanne.grinovero · **Posted:** Mon Mar 26, 2012 10:58 am

Hi,
what database is it?

Could you monitor your heap ? I guess it's running out of memory; you might need to adjust the JVM parameters to give it more memory, or reduce the number of threads.

If it's MySQL, then you need to set some options on the driver or that JDBC driver will attempt to load the whole database in memory.

hibernator_11 · **Joined:** Thu Jun 16, 2011 12:03 pm **Posts:** 94

Are your hibernate relationships setted to lazy or not?

i had this problem i had to change all my relationships with lazy=true. Massindexer was so slow.

and more, i had to use the manual indexer:

http://docs.jboss.org/hibernate/search/ ... index.html

Hibernator,

sanne.grinovero · **Posted:** Thu Mar 29, 2012 5:46 am

the updated link is http://docs.jboss.org/hibernate/search/ ... htoindexes

Hibernator, did you identify why the MassIndexer was slow for you? It is supposed to be significantly faster than the manual one, unless you' re loading a very small data set.

lazyness of relationship does affect the indexing performance, but not significantly, unless it's loading entities which are not needed for the indexing process.

More often the problem is with MySQL because of it's bugs on scrolling large resultsets.