Hi,

I have a master and slave setup deployed as different war applications sharing a common filesystem for the sourcebase. The master is used for indexing while slave is used for searching.

The master updates the sourcebase every 2 mins

The slave gets refreshed every 5 mins

The settings are simple, I am not using JMS and Lucene's basic master/slave setup. Everything is working as expected however with one major issue

The index size seems to keep growing (which should not be the case, I mean the updates to the database is usually are minimal ) . I found out that during reindexing though the master indexes are purged, the slave indexes keep growing.


Following are my master and slave settings

MASTER

<property name="hibernate.search.default.sourceBase" value="/indexdir/default/sourcebase-master"/>
<property name="hibernate.search.postingtitle.directory_provider" value="filesystem-master" />
<property name="hibernate.search.default.directory_provider" value="filesystem-master" />
<property name="hibernate.search.default.refresh" value="120" />
<property name="hibernate.search.postingtitle.refresh" value="120" />


SLAVE

<property name="hibernate.search.default.sourceBase" value="/indexdir/default/sourcebase-master"/>
<property name="hibernate.search.default.indexBase" value="/indexdir/default/indexbase-slave"/>
<property name="hibernate.search.default.directory_provider" value="filesystem-slave" />
<property name="hibernate.search.default.refresh" value="300" />
<property name="hibernate.search.postingtitle.refresh" value="300" />

Hi,

do you do a full re-index all the time? How do you re-index? Do you use the mass indexer or FulltextSession#index? Do you optimize the index after re-indexing? Normally Lucene only marks deleted documents only as deleted and does not "clean out" the index until optimize is called.
Another issue could be the OS. Which one are you using? On Windows files might not get released/deleted.

--Hardy

Hi,

Thanks for your reply. I do a complete reindex by using manual indexing. The steps for the same are

1. purgeAll
2. flush to indexes
3. optimize


What I observed is as follows

1. The indexBase (shared by Master and Slave) seem to be perfect. It goes down to min MB and then grows after indexing
2. The sourceBase of Master does not seem to get deleted of the older data,it creates a new folder each time a reindex is done and the previous folders remain with a marker that says currentX where X is a number like current2

I am almost tempted to delete the sourceBase prior to reindexing after purgeAll is done , but seems like a bad idea, when this should happen automatically.

I am on Linux and yes I do keep getting "Unable to delete marker" or FileNotFoundException during index syncying

Thanks

Hi,

in case you are worried about the two directories there might just be a misunderstanding. Master and slave both utilize two directories (in case you see more than two there might indeed be a problem). A synchronization of the index is done via file copy between master and slave. While the copying is in progress the index is not usable. For this reason the synchronization occurs in a offline copy. One the synchronization is complete the sync index (directory) becomes active. See FSMasterDirectoryProvider.java for more details.

--Hardy