Hibernate Books

All times are UTC - 5 hours [ DST ]



Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: Master Slave setup not deleting older index
PostPosted: Thu Jul 05, 2012 4:17 am 
Regular
Regular

Joined: Tue May 17, 2011 1:45 am
Posts: 52
Hi,

I have a master and slave setup deployed as different war applications sharing a common filesystem for the sourcebase. The master is used for indexing while slave is used for searching.

The master updates the sourcebase every 2 mins

The slave gets refreshed every 5 mins

The settings are simple, I am not using JMS and Lucene's basic master/slave setup. Everything is working as expected however with one major issue

The index size seems to keep growing (which should not be the case, I mean the updates to the database is usually are minimal ) . I found out that during reindexing though the master indexes are purged, the slave indexes keep growing.


Following are my master and slave settings

MASTER

<property name="hibernate.search.default.sourceBase" value="/indexdir/default/sourcebase-master"/>
<property name="hibernate.search.postingtitle.directory_provider" value="filesystem-master" />
<property name="hibernate.search.default.directory_provider" value="filesystem-master" />
<property name="hibernate.search.default.refresh" value="120" />
<property name="hibernate.search.postingtitle.refresh" value="120" />



SLAVE

<property name="hibernate.search.default.sourceBase" value="/indexdir/default/sourcebase-master"/>
<property name="hibernate.search.default.indexBase" value="/indexdir/default/indexbase-slave"/>
<property name="hibernate.search.default.directory_provider" value="filesystem-slave" />
<property name="hibernate.search.default.refresh" value="300" />
<property name="hibernate.search.postingtitle.refresh" value="300" />


Top
 Profile  
 
 Post subject: Re: Master Slave setup not deleting older index
PostPosted: Thu Jul 05, 2012 11:32 am 
Hibernate Team
Hibernate Team

Joined: Thu Apr 05, 2007 5:52 am
Posts: 1689
Location: Sweden
Hi,

do you do a full re-index all the time? How do you re-index? Do you use the mass indexer or FulltextSession#index? Do you optimize the index after re-indexing? Normally Lucene only marks deleted documents only as deleted and does not "clean out" the index until optimize is called.
Another issue could be the OS. Which one are you using? On Windows files might not get released/deleted.

--Hardy


Top
 Profile  
 
 Post subject: Re: Master Slave setup not deleting older index
PostPosted: Thu Jul 05, 2012 6:56 pm 
Regular
Regular

Joined: Tue May 17, 2011 1:45 am
Posts: 52
Hi,

Thanks for your reply. I do a complete reindex by using manual indexing. The steps for the same are

1. purgeAll
2. flush to indexes
3. optimize


What I observed is as follows

1. The indexBase (shared by Master and Slave) seem to be perfect. It goes down to min MB and then grows after indexing
2. The sourceBase of Master does not seem to get deleted of the older data,it creates a new folder each time a reindex is done and the previous folders remain with a marker that says currentX where X is a number like current2

I am almost tempted to delete the sourceBase prior to reindexing after purgeAll is done , but seems like a bad idea, when this should happen automatically.

I am on Linux and yes I do keep getting "Unable to delete marker" or FileNotFoundException during index syncying

Thanks


Top
 Profile  
 
 Post subject: Re: Master Slave setup not deleting older index
PostPosted: Fri Jul 06, 2012 5:54 am 
Hibernate Team
Hibernate Team

Joined: Thu Apr 05, 2007 5:52 am
Posts: 1689
Location: Sweden
Hi,

in case you are worried about the two directories there might just be a misunderstanding. Master and slave both utilize two directories (in case you see more than two there might indeed be a problem). A synchronization of the index is done via file copy between master and slave. While the copying is in progress the index is not usable. For this reason the synchronization occurs in a offline copy. One the synchronization is complete the sync index (directory) becomes active. See FSMasterDirectoryProvider.java for more details.

--Hardy


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.