-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 
Author Message
 Post subject: Clustered Environment Sharing Index Files on Mounted Drive?
PostPosted: Wed Jan 13, 2010 12:31 pm 
Newbie

Joined: Tue Dec 15, 2009 3:49 pm
Posts: 6
I am working with a clustered environment where a few instances of an application will be sharing the same index. In order to save space on the machines where the applications reside, I want to be able to store the index files on a shared mounted drive. Then I want to point each application to that shared drive for index access. Are there any issues with this approach (i.e. - delayed response time, synchronization issues, etc..)? The data set will be in the hundred thousands.


Top
 Profile  
 
 Post subject: Re: Clustered Environment Sharing Index Files on Mounted Drive?
PostPosted: Thu Jan 14, 2010 3:41 pm 
Hibernate Team
Hibernate Team

Joined: Thu Apr 05, 2007 5:52 am
Posts: 1689
Location: Sweden
Technically it should be possible. Of course access over the network is always a little slower than a local disk. Synchronization wise the access to the Lucene directory will be protected by Lucene's own locking strategy. In the latest Hibernate Search release you can even configure the Lucene locking scheme. Depending on the amount of updates index updated might become a bottle neck so. An alternative could be a clustered Search setup using JMS or JGroups.

Another question is how you mount the shared drive. Are you for example using NFS? I vaguely remember reading somewhere that this could cause problems. You should check the Lucene forums for that.

--Hardy


Top
 Profile  
 
 Post subject: Re: Clustered Environment Sharing Index Files on Mounted Drive?
PostPosted: Thu Jan 14, 2010 5:07 pm 
Newbie

Joined: Tue Dec 15, 2009 3:49 pm
Posts: 6
Thanks for your response. I think I have also read in forumsn to use SAN over NFS. We have not decided how to mount the drive yet..


Top
 Profile  
 
 Post subject: Re: Clustered Environment Sharing Index Files on Mounted Drive?
PostPosted: Fri Jan 15, 2010 11:17 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hi,
if you have each instance writing updates to the shared index you could have problems with locking; especially NFS can't guarantee correct working of the file locks used by Lucene.
You have two options:

use the JMS approach (or similar approach using JGroups backend since latest beta release), so that you have only one single node doing the writes and the others doing the reading. Application would scale fairly well because you offload the indexwriters from the main nodes, and actually the single writing node can be setup as having "exclusive write access", which makes it perform better.

An alternative is to use a different LockFactory; I've written about the different LockFactories available on Lucene's wiki: http://wiki.apache.org/lucene-java/AvailableLockFactories
We're currently working on an Infinispan based Directory; it's too early to use the whole Directory in production but you could use the Infinispan approach for Locking only, it's extremely safe and efficient.
Of course you're welcome to try the Directory too, it should work AFAIK and needs feedback. It will definitely make the operations and management easier than shared filesystems, and theoretically perform better.

You mention that you want to share indexes to "save space" ..?? How big are the indexes? And incidentally you can't set it up to use a local index in a clustered environment, or each node will end up having a different incomplete index.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.