I actually use lucene to index about 17 mio entries in a database (oracle). This is realized using a simple ScrollableResult of a projection. This takes about 2 hours to index the whole database. In the meantime the old index is still in use, so I don't need to shutdown the application. After it has rebuilt, I delete the old index and let lucene reopen the index now using the new index built before. The outage of the index search is then really small.

I actually consider moving to hibernate search. I let the MassIndexer index the database which would take > 9 hours calculating the time used for the first 500k rows. After playing around with some parameters it now indexed about 6 mio rows in 1 hour but then slowing down a bit. Hibernate Search takes a lot more time to re-index which makes my request even more important. Is it possible to do the rebuild asynchronous? Use the old index while the MassIndexer is re-indexing the whole database and when it's finished switch the index (maybe another location). I know this means loosing the database changes while indexing is in progress, but this would be ok.

Any ideas?

Hi,
regarding performance, Hibernate Search's MassIndexer is expected to be -generally- faster than the simple ScrollableResult option, but I guess you've tried 4.0.0.Final which is affected by excessive background commits, which are slowing it down conswiderably: https://hibernate.onjira.com/browse/HSEARCH-1019.

It's normal that is slows down a bit, as the cost of segments merges is related to the index size.

Currently there's an "async" option, but all it does it to work in background.. it will still wipe out the index, so your application can be used in the mean time but will be missing most results initially, gradually showing more results while it's approaching the finish.

In Search 3.x it was expected that indexing where to happen "offline", so it was not allowed to use the engine while index was being rebuilt - mostly targeting the use cases of upgrading an application, initial deployments, recovery after maintenance. Only since 4.0 it's allowed to use the application while a background MassIndexer is working, but the general pattern is still
1) wipe out the index (optionally can be skipped)
2) Add again all entities from the database, loading and processing them with multiple threads

So what I'd like to do is to add an option for phase 2) to use "Update" operations instead of "Add" operations, so that when one skips the initial wipe you don't end up with duplicates.

The limitation of this approach is that it will fail to delete entities from the index which are no longer in the database. Do you think this would still be acceptable for your case?
Otherwise we must add a third phase:
3) verify existance of all entities mentioned in the index - delete those not found in the database.

I think I could easily add the "Update" variant, but would need some more time for the third phase. What do you think of it?

P.S.
when running queries via Hibernate Search finding matches which are not found in the database is not a problem, they will be discarded from the results before being returned, but is obviously suboptimal in terms of performance and index size maintenance.

A wipe of the index is acceptable when you can rebuild your index in a couple minutes or even some seconds. When I can increase the speed to 1 hour I still got the problem of missing results while the index is rebuilt, which is not an option for me. So your update solution is really interesting for me. I don't delete much data, just mark it deleted (viewable in the application, but read only). Deleting from database is done just a couple times a year and in this case I may rebuild the index from scratch. Phase 3) is not top priority for me but an update mechanism without it would prevent other users from using it. So it would be great to see such a feature in Hibernate Search. As long as it doesn't exist, I'll not be able to switch to Hibernate Search.

Should I open a feature request on jira?

Yes please open a request on Jira.
I've already started some experiments and it's definitely doable, but I won't be able to work on it until next week.

Also the performance problem of the MassIndexer was solved yesterday, you could try version 4.1.0-SNAPSHOT to get a feel of the new speed.

done: https://hibernate.onjira.com/browse/HSEARCH-1032

perfect, thanks.