The recommended approach to batch indexing is broken if ScrollableResults with eager fetching is done.

See: http://opensource.atlassian.com/project ... e/HHH-1803

After doing an upgrade to 3.1.1 and switching to using ScrollableResults instead of HQL I was happy to see a performance boost in the batch indexing.
That is until searching started to report random/missing results, at first I had thought this was a problem with Hibernate Search but it looks as if it
is related to a very old Hibernate issue.

I don't know if you guys have any sway in escalating this but by this not working it really hurts Hibernate Search and the recommended approach for indexing is broken with this problem.

*Note*
I had tried the patch that appeared to work for two others and rebuilding 3.2.2 GA but it did not solve the problem for me, only switching the collections to lazy fetching was I able to get around this problem. This very unfortunately increased my batch indexing time by a factor of 7.

from 3.1.0 to 3.1.1 there were some improvements on FullTextScrollableResults, but I quess you're meaning Hibernate Core's ScrollableResults performance which is not affected by Search, so switching away from HQL should be the only reason of your performance boost.

Thanks, this is interesting, I agree I'll try escalating this to someone more skilled on core.


If you want to push your batch indexing time I'd appreciate some feedback on current trunk, as there is a batch indexer using multiple threads. It should be faster and unaffected by this issues. Please see https://forum.hibernate.org/viewtopic.php?f=9&t=998976 for some instructions.

Thanks for getting back!

There were some problems using ScrollableResults with the previous (first) version of HibernateSearch for some reason which is why I ended up using just straight HQL. It may have been related to the need to align batch size with fetch size when clearing the session, with 3.1.1 you have flushToIndexes which wasn't available before.

Thanks for the update on the batch indexer, I'll take a look. The batch indexing is indeed a big area of concern at the moment which is why not being able to join fetch collections when using ScrollableResults properly is disappointing.

Totally agree, that's why I am in need for feedback. What is now in trunk is the result of several months of try&benchmark and much discussion on the dev list; It was designed to minimize the effect of latencies in entity and fetch loading, so benefits depend on your schema/model mapping and how much latency (network+disk+..) was affecting you.
So if you're having trouble it *might* give a huge improvement, or not. If it doesn't, I'd really like to know as we could yet improve something. My average expectation is around 10X, unless you're working on a local db and a very simple schema, then the overhead of additional complexity could even hurt.