I think I may have uncovered a potential race condition in the master/slave file synchronization mechanism. Specifically, if the master instance switches out the source while the slave is copying over a file from that source AND the new source no longer has the file being copied, the slave will fail with the following stack:
Caused by: java.io.FileNotFoundException: C:\Users\mogleym\Scratch\csmc\mds\lucene\dev\master-copy\org.csmc.mds.model.Expertise\1\segments_1 (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
Since you probably don't want to have to implement locks on the source, I would suggest catching the FileNotFoundException on the slave side, and if thrown, simply deleting the corresponding slave file being synchronized. This should be a safe bet, since the file it's attempting to sync is actually no longer present in the source.
What do you guys think? And what workaround would you suggest in the meantime?