Hibernate Search +MassIndexer +entityloader-1 java.lang.Out

salemba · **Joined:** Tue Mar 02, 2010 4:28 pm **Posts:** 8

Hello,

I've been testing the new hibernate search release after reading a post about MassIndexer performance. I have a 1.2M entry in my database.

Usually when indexing my db, i use scrollable result and index entries one by one: I bring 10000 entries from db, then for loop to index them one by one.

But how can i use now MassIndexer?

I tried this way:

Code:

FullTextSession fts = this.getFullTextSession();
      try{
      MassIndexer msi= fts.createIndexer(Abonne.class);
         
         msi.batchSizeToLoadObjects( 1000 )
         .threadsForSubsequentFetching( 8 )
         .threadsToLoadObjects( 4 )
         .cacheMode(CacheMode.REFRESH)
         .limitIndexedObjectsTo(10000)
         .startAndWait();
      }catch(InterruptedException intExc){
         intExc.printStackTrace();
      }

but that gave me :

Code:

Exception in thread "Hibernate Search: identifierloader-1" java.lang.OutOfMemoryError: Java heap space

Thanks in advance for the great work Hibernate Team :)

sanne.grinovero · **Posted:** Fri Mar 05, 2010 4:32 am

Hello salemba,
from "java.lang.OutOfMemoryError" I can't be sure if that same thread was having trouble with memory, it just happened to ask for some memory when there wasn't available.
Just for reference: thread "Hibernate Search: identifierloader-1" means this thread was performing a task like what is implemented in sources as org.hibernate.search.batchindexing.IdentifierProducer

This class is scrolling on the identifiers only of your entity, using a batchsize of 100 and disabling the cache (the selected cachemode is not applied at this stage) and then pushing arrays if identifiers down the pipeline.

Is it possible that the identifier of you Abonne class is heavy on memory?

The implementation should use a fixed amount of memory - not dependent on the number of entities - but still the memory requirement is quite higher than a single scroller; did you set appropriate Xmx and Xms options to you JVM ?

please keep me updated, if we can improve implementation or documentation I'll do that

salemba · **Joined:** Tue Mar 02, 2010 4:28 pm **Posts:** 8

Hello,
Thank you for considering my post :)

Quote:

Is it possible that the identifier of you Abonne class is heavy on memory?

Actually i don't think so, here is the identifier:

Code:

@Id
   @DocumentId
   private String nab9;

it's limited to 9 character maximum; do you think it's really a problem?
this class contains about 35 string attributes.

my current jvm configuration :

Code:

-vmargs
-Dosgi.requiredJavaVersion=1.5
-XX:MaxPermSize=128m
-XX:PermSize=128m

-Xmn100m 
-Xms512m
-Xmx1024m

-XX:+TraceClassloading
-XX:+TraceClassUnloading  
-Djava.net.preferIPv4Stack=true
-XX:ParallelGCThreads={5}
-XX:+CMSClassUnloadingEnabled

I'm trying someworakround to get this batch update ineterfering with indexes, but I'm convinced that the solution is that MassIndexer: If i'm not missing any detail, post-update listeners do not listen to hibernateTemplate.bulkUpdate, so the only way to make lucene indexes up to date, is to reindex all the database after such modification.

Do you have any suggestion?

sanne.grinovero · **Posted:** Fri Mar 05, 2010 7:27 am

Quote:

Actually i don't think so, here is the identifier:
Code:
@Id
@DocumentId
private String nab9;

it's limited to 9 character maximum; do you think it's really a problem?
this class contains about 35 string attributes.

No that can't be a problem.

Quote:

my current jvm configuration :

Code:
-vmargs
-Dosgi.requiredJavaVersion=1.5

This looks like an eclipse JVM configuration, is that right?
note that these settings are NOT applied to the code you start from within eclipse, it's only used for your IDE an plugins, while tests are run in a forked JVM.
Look into "Run configurations..." -> Arguments -> VM Arguments

sanne.grinovero · **Posted:** Tue Mar 09, 2010 1:37 pm

hi salemba,
any news on this? Using MySQL maybe?
I just got notice that MySQL has a "feature" that will have it preload the whole resultset in the JVM memory, resulting in out-of-memory exception when the tables are big enough.
Notice that even while you're using .limitIndexedObjectsTo(10000) this has no effect on the resultset dimension, it's still loading the whole 1,2M identifiers in memory.

To have MySQL behave correctly use these options on the jdbc connection string:

Code:

jdbc:mysql://host/databasename?useCursorFetch=true&useServerPrepStmts=true

hope it helps, let me know

salemba · **Joined:** Tue Mar 02, 2010 4:28 pm **Posts:** 8

Thanks for the trick. That helped in a way :)

For the massindexer we let it as an alternative for the moment, i'll let you know as soon as we go back to it.

ps: Sorry for replying that late.