Hi Guys,
Just reading up on HS and just have a few questions on it which werent so clear to me in the documentation.

In the master/slave paradigm what happens if the master dies a death?? Will another node take over responsibility for updating the slaves?? ( I guess not judging by the configuration properties in the docs)

A shared directory is needed for copying of the master copy to the slaves, is that true(I can see by the configuration it is but just want to hear it)?? Any other options for data transfer??

Ok here is my scenario........I have 2 machines with 2 nodes on each which the front end runs on, I have a seperate back office system which does most of the updates on data that I want to index. Having a shared mount between the two systems is not a runner!! I would say Im allowed a minimum of a 15 minute delay before changes in the back office should be reflected in the front end. If I dont have a shared directory between systems then what would be the best way to keep my index up to date using hibernate search??

1. Obvious one is to run a scheduled task and check each rows timestamp to see if it has changed, if so then update the index.
2. Any way slaves can update there own index themselves instead of waiting for the copied version??

Any other options open to me with HS??


Thanks guys,
LL

Correct. There is no master failover procedure. On the other hand, no data gets lost. The index update jobs will just be kept in the JMS queue (if properly configured)



Yes, the JMS Master/Slave approach requires a shared file system. Other options are to work with a version of HSearch build from source. We just recently added support for clustering using JGroups and Infinispan (not sure of the latter is committed yet). Of course that's cutting edge, but we it would be great to get some feedback on this new clustering possibilities.

Other than that a scheduled task would of course work or you could maybe use rsync to sync the master directories to the slaves. This would not require a shared file system. That's just an idea though.

--Hardy

We had initially thought about having a back up of the master node, however we came to the same conclusion that the messages would be on the queue if the master node failed. Worse case if the queue went down then we could manually kick off the indexing on the database.

Also, JMS queues can be configured with a backup queue server. I guess it comes all down on how much effort you want to put into this. For example, is your database itself clustered?

--Hardy

Thanks for the feedback guys.

Dont think the queue option is a runner, without a shared file store between machines it aint an option.


Ok if i run with a scheduled task idea. I have two machines with 2 nodes on each, lets configure each of the two nodes to use the same directory on the same machine. So two nodes on the same machine, share the same index.

Will probably need 2 types of scheduled tasks. 1 that indexes at night and takes a considerable amount of time and the other that just indexes every 15 minutes but just checks if timestamp has changed at record level, so a partial index of just records that have been updated.

Question though is where to do the indexing??

Scenario 1 Offline
We have a seperate application that the 2 types of scheduled tasks run in. It generates the full index and keeps a most up to date version, when finished it needs to get the latest index online.........question is though, how to do the copy without anyone noticing it?? i.e users who are currently searching the current index. Actually this must be like how the master sends its version to the slaves. How do you guys drop it in??

Scenario 2 Online
On each of the two machines configure one of the nodes as responsible for keeping the index up to date. This activates our scheduled tasks. (Dont like doing double the work here and dont like having this process in the front end nodes)

Scenario 3 Offline but online ;)
Like scenario 1 but the application runs on both machines but connected to the same file index as front end nodes and updates index directly, no need for copying. (Again doing double work but no risk to front end nodes from indexing process)

Would prefer to run with the scenario 1 but how to get the latest index in without effecting users?? Other options??

Thanks for the input guys,
LL

Hi Guys,
When you replace the slaves index with the newest index do users experience any outage?? I see that in lucene there is no update allowed on the index, you must delete the old version/record and replace with the new one but what happens if you have users currently searching against the old index and your application is currently deleting it??

Also if your index is big and I assume it could take a while(havent actually tested myself yet) before the new index is copied over/inserted......does this cause any downtime for users at all??

Couldnt find anything on this and have yet to give it a try myself.

Cheers guys,
LL

Hi,

there is not downtime for the slaves. The JMS Master/Slave configuration works with two copies of the index on the slave side. One active one and one passive one. On index synchronization the latter gets updated via file copy operations. Only once the passive index is updated to the latest master version an index switch occurs and the passive index becomes the active one (and vice versa of course).

Also not the whole index is copied every time. Only the changed files gets synchronized.

--Hardy

Hi Hardy,

Ah I didnt know that, I didnt see from the configuration that the slave directory actually has two copies, an active and passive. Nice solution though. Thanks for that.

Cheers,
LL