I face some challenge when introduce hibernate search within our jboss clusters, hope to get some advice.

Our system has two cluster, one is kind of provisioning cluster with two nodes, all the data changes are towards this cluster, so call provision cluster. Then there is another cluster handles google like search request based on hibernate search, no data update operation should be done in this cluster, so call traffic cluster which could have 2~n nodes.

To get both high availability and scalability, i intent to use master/slave mode in hibernate search. Since there is no hard drive available on traffic cluster (of douse NFT is available, but i don't think it is a good choice to store index), so i intent to use infinispan as index directory provider, backend can be either JMS or jgroups.

My problem is master node(s). i want to choice either whole traffic cluster or whole provisioning cluster as master to update index. It seems you can't config more than once node as master with hibernate search today (no matter JMS or jgroups backend), otherwise it throws "org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock" exception. As i understood, the first master node keeps the lock when opens index writer, so the rest of node(s) in the cluster get such exception. i don't want to differentiate nodes in the same cluster (i.e only config one as master), this could cause single point of failure. So I wonder if there is any way to auto elect the master in a cluster and re-elect in the cluster if the previous master node crashed.

I see there is some experiment feature in hibernate-search 4.3 for auto election if uses jgroups backend, but we are a commercial product, so i am afraid of such experiment feature.

Thank you for your advice.

Hi,
as you say the project today includes some experimental support for "automatic" master election but that needs more work to be fully reliable, I expect this will need some more months.

Other than "automatic" you have the option to control the election yourself: JMS queues can be configured with persistent storage (so that the master node going down can be restarted without losing updates in the downtime), or you can setup multiple consumers.
In case of multiple consumers you need to control yourself (with custom code or integrations) how to achieve a "passive master" to get elected to active master, and force-release the lock in that case. Usually the external code/integrations are in a better position to make such a decision as it might be connected with your monitor tooling, or directly under manual control "admin needs to push the button".
In both configurations, using JMS would prevent you to lose any updates.

Interesting, please let me know if you're willing to participate in the development or testing of these features: that's usually the best path to get a very solid implementation.

Hi Sanne,

Thx for replying. Now i set "exclusive_index_use" as false to enable multiple masters in the same cluster, of course with JMS as backend as you suggested. The index update is not very frequent in our system, so lose some efficient is not a problem for us.

Btw, one additional question, when there is an entity create/update/delete, the master node will do partial index update (i.e update this impacted entity) or rebuild whole index again? If it is just a partial index update, then it is very fast, so it is kind of near realtime search, right?

BR/Keven