Firstly, I am something of an amateur with Hibernate Search & Lucene.

I have a database entity I am indexing that is also updated in the database by several scripts and non-Java applications, so I cannot rely on Hibernate Search in my application to dynamically update my database as it is not aware of all transactions.

So far, the only solution I have is to periodically run a complete re-index.

My question is, is there a way to tell Hibernate Search to periodically check for updates on my entity? And if not:

Can I use my created/updated fields on my entity record so Hibernate Search does not completely re-index everything (i.e. does not update records that have not been update since the last re-index).

Any other advice appreciate. Thanks.

ah, I've some experience with these kind of experiments, so you asked for my rant :-) :

Lucene's index is not containing the full representation of the entity, and the transformations applied are not bi-directional, so it's not possible to look into the index to find out if it's different than the database stored representation. Even if you made this possible by storing much more usually unneeded information in the index, it is far less efficient to read this data back than to batch-rebuild the index.

Another approach is to have a flag column containing a boolean "needsReindexing", this is the best solution I've seen, or alternatively using the version of the entity when using hibernate's optimistic locking and store the version information in the index; but using the version information you'll have to load the entities one by one, which again makes it inefficient.

My conclusion is that, unless you have a very small amount of updates compared to huge amount of indexed data, your best bet is to reindex it all periodically; which has btw many other benefits: like if you change analyzers, add new synonyms and/or stopwords, the existing index entries will be updated too.

In fact reindexing it all can be done quite fast, since latest beta this API is available which is the most efficient strategy we could think of:
http://in.relation.to/Bloggers/HibernateSearch32FastIndexRebuild

Great, thanks for your help.

I am loathe to run a daemon thread to take care of periodic re-indexing, my application is purely request driven so far and I am eager to keep it that way.

I haven't seen one, but is there a way to set an indexer to update periodically, or am I forced to roll my own here?

Thanks.

you should roll your own one, and possibly choose a time frame where you have less load (at night?)

Okay, sounds good.

Thanks for the advice!

Is there a way to only mass index of a particular shard ?

In our environment, I am generating an index that will be used by several applications.

I have to hot swap in an index generated from one application into another (for reasons outlined above).

Is the only way to hot-swap the indexes to bring down the target application, replace the indexes for it, then restart it?

Regards,

Steve

you could look into using the Master and Slave directoryProviders, they have appropriate logic to copy periodical snapshots to another dir (the master) and the other to take periodically a new snapshot from another dir (the slave) while switching from the first to the second to perform searches.

alternatively, just using rsync should be fine, if you're sure the "slave app" will never write to the index (you might want to disable the H.Search listener)