-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 
Author Message
 Post subject: java.lang.OutOfMemoryError During Mass Indexing
PostPosted: Fri Jul 07, 2017 10:55 am 
Newbie

Joined: Fri Jul 07, 2017 10:39 am
Posts: 1
When performing mass indexing with elasticsearch and the following parameters

fullTextSession.createIndexer().batchSizeToLoadObjects(25)
.cacheMode(CacheMode.IGNORE)
.threadsToLoadObjects(16)

I'am getting a java.lang.OutOfMemoryError: GC overhead limit exceeded exception.

The indexing runs fine for the first 100,000 records and then this exception occurs.

Can someone please provide advice on how to tweak the mass indexer to perform as quickly as possible with elasticsearch without this exception happening?

Thanks.


Top
 Profile  
 
 Post subject: Re: java.lang.OutOfMemoryError During Mass Indexing
PostPosted: Tue Jul 11, 2017 7:46 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Did you try the obvious: check how much memory is needed?

The index rebuild process is a very intensive operation and it will be significantly faster if you can allow it to scale up on the number of threads and use larger buffers. Using many buffers and threads of course consumes more memory, so make sure to allow your JVM to use a reasonable heap size.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: java.lang.OutOfMemoryError During Mass Indexing
PostPosted: Wed Aug 02, 2017 6:06 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hi Paul,

we've been working on performance recently and we now have a reasonable idea of what problem you might have reported.

See this PR if you're interested in the details:
- https://github.com/hibernate/hibernate-search/pull/1500

Or wait the next release, which is going to include all this work.

Thanks!

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: java.lang.OutOfMemoryError During Mass Indexing
PostPosted: Thu Aug 17, 2017 2:59 am 
Hibernate Team
Hibernate Team

Joined: Thu Aug 17, 2017 2:51 am
Posts: 3
For the record, the fix was released with Hibernate Search 5.8.0.CR1: http://in.relation.to/2017/08/16/hibern ... 5-8-0-CR1/


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.