thanks for narrowing down the issue; indeed my suggestion doesn't apply in this case.
There was a similar issue reported a while ago, but this is the first time somebody happens to report that the master copied an incomplete file set, thanks for that!
I think we need
HSEARCH-152 so I'm proposing it for the next release.
this issue is not very hard to be solved in Hibernate Search 4 since the backend has a new design in which the IndexWriter and the DirectoryProvider can easily interact, but I'm afraid it's not easy to backport a fix for v. 3.1.0.
I think that your problem might be that the default deletion policy for segments in the Lucene Directory is
org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy, so you might have a race condition between segments being deleted and the copy going on; this is very unlikely since the master acquires a lock on the index while it's copying, and the same lock must be taken by the IndexWriter in the backend .. though merge operations might happen in a background thread.
Could you try verifying if my assumptions are correct, by opening the 3.1.0 sources and making sure the IndexWriter, when it's opened, is set to use the
org.apache.lucene.index.SerialMergeScheduler as MergeScheduler (instead of the default concurrent scheduler) ?
The class you would need to edit is AFAIR
org.hibernate.search.backend.Workspace. Alternatively you could change the deletion policy, likely a better performing solution but takes a bit more code so I'd try it out later when we know this is going in the right direction.
Please let me know, thanks