Hibernate Books

All times are UTC - 5 hours [ DST ]



Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: Hibernate search clustering latency
PostPosted: Tue Oct 04, 2011 8:37 am 
Newbie

Joined: Tue Oct 04, 2011 8:27 am
Posts: 3
Hi,
I am trying to minimize the indexes replication latency (in JMS master/slave the default refresh interval is half hour as described at [1]).
I have found interesting articles [2] and [3]:

"Latency: the slave nodes need refreshing their local index folders after a fixed period. So there is a latency between master and slave to see a newly indexed item" [2]

and

"To that end, HIbernate Search doesn't rely on JBossCache or any in-memory partitioning / caching schemes and instead relies on JMS and each JVM's local disk in order to provide up-to-date indexing across a cluster with simultaneous low latency" [3]

It's seems that solution described at [2] can provide up to date indexes with low latency by replaying the lucene actions on each local lucene index using JMS topic. What do you think about this solution?


[1] http://docs.jboss.org/hibernate/stable/ ... ms-backend
[2] http://linhtech.wordpress.com/2010/10/0 ... mentation/
[3] http://stackoverflow.com/questions/8858 ... ate-search


Top
 Profile  
 
 Post subject: Re: Hibernate search clustering latency
PostPosted: Wed Oct 05, 2011 5:32 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
Hi,
[3] is outdated: indeed we never made an integration with JBoss Cache, as this technology evolved now into a new project called Infinispan. We do have Infinispan integration which can be configured in several ways.

You can get zero latency between writes and reads via a sync backend to an Infinispan stored index: this would have an immediate write on network level and all written data would be immediately visibile to other nodes.
Since there's network performance involved, usually it is configured as async, meaning that your write performance won't slow down too much and provide rather quick visibility of changes to other nodes (but no guarantees); still even if you need sync this network based Directory is quicker than the FSDirectory, assuming you have a reasonable network infrastructure and are not competing with special purpose SSD arrays.

Note that when using the Infinispan Directory we still suggest to setup JMS as master/slave: only one node is allowed to write in the index; a network wide lock is acquired and if another node happens to try acquire this lock the index will be save but the node will likely experience timeouts in acquisition: better to send off all work to a master node and have it efficiently write on behalf of all nodes in the scope of a single lock.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Hibernate search clustering latency
PostPosted: Sat Dec 17, 2011 12:04 am 
Newbie

Joined: Fri Dec 16, 2011 6:18 pm
Posts: 1
But what happens in the event your master node goes down (maintenance, hardware failure)? It would seem that during that time no updates to the index can occur. Is there any automatic conversion of an existing slave to become the new master? If a configuration is using JGroups instead of JMS for the master/slave, are the update messages just lost?

Thanks,
Adam Harris


Top
 Profile  
 
 Post subject: Re: Hibernate search clustering latency
PostPosted: Wed Dec 21, 2011 1:01 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
Hi Adam,
yes JMS can provide a higher level of resilience than JGroups by persisting unprocessed messages, still JGroups can be configured with some degree of guarantee of delivery as well (but messages are lost if sender crashes before the receiver reconnects).

With both options, you don't necessarily have a single master. Taking JMS as an example, you could have a secondary master listen to the same queue; this makes copy-around of the reference index a bit more complex, possibly using the Infinispan Directory would be simpler for that as it keeps all nodes in sync having the same index.

There is no automatic election from slave to master, but we have plans for such improvements by better integrating the Infinispan and JGroups modules; if you need such an architecture feel free to help out, or you can build on top of the above mentioned components.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.