potential FileNotFoundException on slave copy

declan · **Joined:** Thu Jul 21, 2011 6:26 am **Posts:** 8

Still no joy. All servers - master/slaves were started at exactly the same time, refresh period is 120 seconds on all and still the same problem.

What else can I do?

sanne.grinovero · **Posted:** Tue Jul 26, 2011 10:56 am

Hi declan,
sorry for the delay.
I just looked at that code again, I'm wondering how the new ReaderProvider is affected by copies - could you please try setting up the "not-shared" reader provider? I think it might help, and it would help me to better understand what's happening in your case.

declan · **Joined:** Thu Jul 21, 2011 6:26 am **Posts:** 8

hi sanne,

it seems not a problem with the slave and the speed of the copy (as i mentioned that the index size is very small and its on a LAN)

the source of the problem is on the master, specifically when the master copies from indexBase to sourceBase, just listing the contents in 1 or 2 show that *sometimes* it a partial version of the index, the slave then copies this corrupt version of the index resulting in the exception.

i tried slowing down the master copy, syncing the slaves etc. with no success

we are using HS 3.1.0.GA (yes, we need to upgrade) but I can't right now, so don't know whether your suggestion still applies?

sanne.grinovero · **Posted:** Tue Jul 26, 2011 12:29 pm

thanks for narrowing down the issue; indeed my suggestion doesn't apply in this case.

There was a similar issue reported a while ago, but this is the first time somebody happens to report that the master copied an incomplete file set, thanks for that!
I think we need HSEARCH-152 so I'm proposing it for the next release.

this issue is not very hard to be solved in Hibernate Search 4 since the backend has a new design in which the IndexWriter and the DirectoryProvider can easily interact, but I'm afraid it's not easy to backport a fix for v. 3.1.0.

I think that your problem might be that the default deletion policy for segments in the Lucene Directory is org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy, so you might have a race condition between segments being deleted and the copy going on; this is very unlikely since the master acquires a lock on the index while it's copying, and the same lock must be taken by the IndexWriter in the backend .. though merge operations might happen in a background thread.

Could you try verifying if my assumptions are correct, by opening the 3.1.0 sources and making sure the IndexWriter, when it's opened, is set to use the org.apache.lucene.index.SerialMergeScheduler as MergeScheduler (instead of the default concurrent scheduler) ?
The class you would need to edit is AFAIR org.hibernate.search.backend.Workspace. Alternatively you could change the deletion policy, likely a better performing solution but takes a bit more code so I'd try it out later when we know this is going in the right direction.
Please let me know, thanks

declan · **Joined:** Thu Jul 21, 2011 6:26 am **Posts:** 8

Hi Sanne,

Apologies for not getting back sooner but I was on vacation.

Bizarrely the issue stopped occurring after a few days, the same way it started with no obvious changes. FYI, I did have to restart the master node at a random time one day and shortly after the slaves started misbehaving again. So you are correct in your previous post about the order of starting master and slaves and correctly configured refresh period. I understand it is important to give due consideration to a good refresh period but I think we should not have to worry about the order of when master and slaves start up.

Anyway, I have just upgraded to HS 3.4.0.Final. I understand the issue is still possible in HS 3.4, but you mentioned a a fix/workaround for it - what is that as a matter of interest? I plan to move away from the NFS master/slave copy anyway as I was never keen on that implementation and move to Infinispan.

sanne.grinovero · **Posted:** Mon Aug 29, 2011 5:54 am

Hi,
the best fix is IMO to implement HSEARCH-152, as mentioned in my previous post. That should be easier accomplished in v.4 than in v.3.x because the two components which need to interact for this are not as independent as they used to be, so easier to create a direct link between them to coordinate on the when+what to copy safely.

As a workaround, again as I mentioned above, I think setting the MergeScheduler to SerialMergeScheduler and also making sure you use a DelectionPolicy which avoid deleting the last-1 commit should avoid any issue.

Both should serve, I prefer HSEARCH-152 as it might use less disk space and because the SerialMergeScheduler is not the fastest option.

declan · **Joined:** Thu Jul 21, 2011 6:26 am **Posts:** 8

thanks, it's still a problem after upgrading, I think I will go with the second option, disk space is not an issue, does this look a reasonable way of implementing a DeletionPolicy to avoid the issue

Code:

public class KeepLastNDeletionPolicy implements IndexDeletionPolicy {
    int numToKeep;

    public KeepLastNDeletionPolicy(int numToKeep) {
      this.numToKeep = numToKeep;
    }

    public void onInit(List commits) throws IOException {
      // do no deletions on init
      doDeletes(commits, false);
    }

    public void onCommit(List commits) throws IOException {
      doDeletes(commits, true);
    }

    private void doDeletes(List commits, boolean isCommit) {
        if (isCommit)
        {
                int size = commits.size();
                for(int i=0;i<size-numToKeep;i++) {
                        ((IndexCommit) commits.get(i)).delete();
                }
        }
    }
  }

sanne.grinovero · **Posted:** Mon Aug 29, 2011 9:54 am

yes that looks like ok, but I'd recommend to write a test too.
I guess having some creativity to think about a proper test is the hardest part here :) You'll need to trigger a merge operation after you copied some files, but not yet all of them, to verify that the final index is correctly readable.