Lucene files not found in FSMasterDirectoryProvider's copy

mueller · **Joined:** Mon Mar 10, 2008 6:40 pm **Posts:** 114

I know I've written about this in the past, but this problem comes up fairly frequently for us and has happened in different servers and environments. So it's definitely not a networking or hardware issue. We basically get errors like this below:

Code:

2010-06-04 18:28:31,400 ERROR [pool-1-thread-1] FSMasterDirectoryProvider - Unable to synchronize source of /index_base/com.example.Person
java.io.FileNotFoundException: indexes/com.example.Person/_b9z.fdt (No such file or directory)
       at java.io.FileInputStream.open(Native Method)
       at java.io.FileInputStream.<init>(FileInputStream.java:106)
       at org.hibernate.search.util.FileHelper.copyFile(FileHelper.java:157)
       at org.hibernate.search.util.FileHelper.synchronize(FileHelper.java:148)
       at org.hibernate.search.util.FileHelper.synchronize(FileHelper.java:132)
       at org.hibernate.search.store.FSMasterDirectoryProvider$CopyDirectory.run(FSMasterDirectoryProvider.java:225)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)

The index filename changes with each error, but it's essentially the same problem that comes up. I can't tell if it's actually affecting our search operations...

When I previously posted about this, one of the Hibernate Search developers responded that it must be some kind of networking or hardware issue for us. It isn't, but odd that this isn't a known problem. We're using Hibernate Search v3.2. It seems to me that the copy operation from the base index directory to the source index directory is simply happening at the same time as Lucene is changing filenames and files. So this will naturally occur periodically with any high volume indexing. Are others experiencing this problem too? Is this another more serious problem? Does Hibernate Search recover from this properly?

sanne.grinovero · **Posted:** Sat Jun 05, 2010 5:02 am

Quote:

Does Hibernate Search recover from this properly?

Yes the code looks like it should be able to recover from this; you'll miss the updates from the failed copy, but they should be resynchronised on next scheduled copy.
The slaves shouldn't be affected AFAIK, but to be 100% sure could you please verify there are no errors on the slave nodes?

Would yo be able to write a simple test able to reproduce this? If there's a problem in Search I'll be glad to fix it, but being able to reproduce this issue is a requirement.

mueller · **Joined:** Mon Mar 10, 2008 6:40 pm **Posts:** 114

Hi Sanne, thanks for the quick reply. Those simple tests are so difficult for us sometimes, we have a ton of code we need to simplify. But I would assume it's easy to see if what I wrote makes sense. I'm running this on a centos 5 server. I'm not getting errors on the slave servers. But maybe the results they're returning aren't accurate.

sanne.grinovero · **Posted:** Sat Jun 05, 2010 5:23 am

Quote:

But I would assume it's easy to see if what I wrote makes sense.

well I believe what you say, but need clues about why and how, not sure what's causing you this specific error which I never saw before.
I have one idea: HSEARCH-152, but it's an old one and nobody ever complained that it was actually needed; I'd need a testcase to verify if that would be needed. BTW we where considering it just a possible improvement, not a requirement for this kind of replication, but might be wrong from what I see now.
If you can't make one I'll try myself, but that will have to wait two/three weeks as I'm leaving for a trip.