Hello,

I have a web application that runs on two non-clustered nodes (with a load-balancer in front of it). Now my main problem is that once I update an entity the search index needs to get updated on both nodes. I tried to do it over infinispan with and without jgroups-master-slave. The only configuration that is working is, when I manually assign master and slave. But this is not an option for me, in our environment the ear is auto-deployed on both nodes, so there can be no difference.
With jgroups set to auto and even without jgroups at all with only I always get a Lock-Exception:
Work Type org.hibernate.search.backend.UpdateLuceneWork
: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@23e2066b
at org.apache.lucene.store.Lock.obtain(Lock.java:84) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:146) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:113) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:101) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]

It is also not working with

or



I think it's because of an error in AutoNodeSelector: If there is only one node, it selects the first node, if the second node comes up it selects the second node without releasing the first node's lock on the index, I guess.
It is working, if I do the following:

Deploy on node 1

Now node 1 is master and has the lock

deploy on node 2

now node 2 is master but gets exception on updates of entities

redeploy on node 1

now node 1 is master again, it all works fine

But of course this is not possible in production. ;-)

So here are some ideas:

It should be possible to have a non shared ram-index on each node, with jgroups to tell all nodes to update their index so you won't have any locking problems. This solution would work without any infinispan clustering.

It should be possible to overide AutoNodeSelector or define NodeSelection programmtically

Do you have any other ideas on how to solve it?

I found out that if I don't use a MassIndexer on startup, the index is not locked by one node anymore.

But how should I build my initial index if the Massindexer is not working correctly?

No, I was not a MassIndexer fault: If I start one node, update an entity, start the second one, then the second node gets LockingExceptions again. Obviously there is really an issue when the index-master-node changes after the first master has locked the index.

Hi,
interesting description. Essentially if I undestood you correcty, the MassIndexer takes long to complete, and in parallel the master role changed to a secondary node?
I've created https://hibernate.atlassian.net/browse/HSEARCH-1554

The "Automatic master election" strategy is indeed experimental, and I guess your feedback will need to be incorporated before I remove that warning from the documentation.

A nice solution would be to fix AutoNodeSelector to be conservative about changing an existing master node, but by doing so the policy becomes statefull and needs coordination with the other nodes. Not an easy fix.

I would recommend using the configuration to separately configure the master/slave nodes; if you have to deploy the same EAR you could differentiate the configuration option depending on some environment property, maybe the hostname?

Consider that you can configure Hibernate Search also programmatically.

Hello Sanne,

thanks for your answer. No it was not because it took to long, the indexing was finished when I started node 2. It was because that way node 1 had written into the index before node 2 was even startet.

I just have tested other options to verify this behaviour. One was not to index intially at all:
1. start node 1
2. update entity on node 1, so that it gets indexed for the first time: Node 1 is master and updated the index.
3. start node 2:
4. update entity anywhere: now node 2 is master and gets same lock exception because node 1 has lock on it.
5. redeploy on node 1
6. update entity anywhere: now node 1 is master again, it all works fine, because this node already has the lock.

If I first start both nodes before I initially index an entity it works fine, because the second node is master and also the first to lock the index. But in production I cannot guarantee that both nodes are started before an entity gets updated.

I also tried a very simple configuration:


I debbuged the code and was wondering why there is always an ExclusiveIndexWorkspaceImpl used instead of a shared one (within LuceneBackendResources).
I debugged some more and found out that the MaskedProperty class always hides the exclusive_index_use option.
You can see that by setting a breakpoint in IndexManagerHolder, line 322: now the toString of
rootCfg is = {​INCIDENT_INDEX.​exclusive_index_use=false, default.directory_provider=infinispan, hcore.session_factory=null, ​default.​exclusive_index_use=false}
but the one of globalProperties is = {directory_provider=infinispan}

So I guess, there is an error, too. What do you think? If the indexes were just shared it should all work.

How can I change Index properties programmatically? I don't know right now, how I could programmatically define a jgroups master and a slave, I didn't find this in the reference guide.

Ok, forget about the debugging point: It's kind of embarrassing for me. ;-)
I copied the configuration from your reference and there have been special (NON_ANSI) characters in it:


These are not visible in eclipse, also not in my post above...

Now it's working with a shared index, also with jgroups. But you have to enable exclusive index use to get it working.

Thanks I'll need to test that, travelling now.

The programmatic configuration feature is something inherited from Hibernate ORM:

http://docs.jboss.org/hibernate/orm/4.3/manual/en-US/html_single/#configuration-programmatic

Almost forgot: you can also inject environment properties directly in the configuration file using this syntax:



and run the JVM with -Drobots.max_merge_docs=42

Ah good to see that :)
(My replies above where written before I saw your update)

I actually had the same problem once, didn't realize the special characters where coming from the documentation. I'm looking forward to provide a better validation of configuration properties in 5.0.


Yes that's expected. Again we could do a better job at validating configurations.