When performing mass indexing with elasticsearch and the following parameters

fullTextSession.createIndexer().batchSizeToLoadObjects(25)
.cacheMode(CacheMode.IGNORE)
.threadsToLoadObjects(16)

I'am getting a java.lang.OutOfMemoryError: GC overhead limit exceeded exception.

The indexing runs fine for the first 100,000 records and then this exception occurs.

Can someone please provide advice on how to tweak the mass indexer to perform as quickly as possible with elasticsearch without this exception happening?

Thanks.

Did you try the obvious: check how much memory is needed?

The index rebuild process is a very intensive operation and it will be significantly faster if you can allow it to scale up on the number of threads and use larger buffers. Using many buffers and threads of course consumes more memory, so make sure to allow your JVM to use a reasonable heap size.

Hi Paul,

we've been working on performance recently and we now have a reasonable idea of what problem you might have reported.

See this PR if you're interested in the details:
- https://github.com/hibernate/hibernate-search/pull/1500

Or wait the next release, which is going to include all this work.

Thanks!

For the record, the fix was released with Hibernate Search 5.8.0.CR1: http://in.relation.to/2017/08/16/hibern ... 5-8-0-CR1/