-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 9 posts ] 
Author Message
 Post subject: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Mon Mar 10, 2014 1:02 pm 
Expert
Expert

Joined: Thu Jan 08, 2009 6:16 am
Posts: 661
Location: Germany
Hello,

I have a web application that runs on two non-clustered nodes (with a load-balancer in front of it). Now my main problem is that once I update an entity the search index needs to get updated on both nodes. I tried to do it over infinispan with and without jgroups-master-slave. The only configuration that is working is, when I manually assign master and slave. But this is not an option for me, in our environment the ear is auto-deployed on both nodes, so there can be no difference.
With jgroups set to auto
Code:
<property name="hibernate.search.default.worker.backend" value="jgroups"/>)
and even without jgroups at all with only
Code:
<property name="hibernate.search.default.directory_provider" value="infinispan"/>
I always get a Lock-Exception:
Work Type org.hibernate.search.backend.UpdateLuceneWork
: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@23e2066b
at org.apache.lucene.store.Lock.obtain(Lock.java:84) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:146) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:113) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:101) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.4.2.Final.jar:4.4.2.Final]

It is also not working with
Code:
<property name="hibernate.search.​default.​exclusive_index_use" value="false"/>

or
Code:
<property name="hibernate.search.default.locking_strategy" value="none"/>



I think it's because of an error in AutoNodeSelector: If there is only one node, it selects the first node, if the second node comes up it selects the second node without releasing the first node's lock on the index, I guess.
It is working, if I do the following:
    Deploy on node 1
    Now node 1 is master and has the lock
    deploy on node 2
    now node 2 is master but gets exception on updates of entities
    redeploy on node 1
    now node 1 is master again, it all works fine

But of course this is not possible in production. ;-)

So here are some ideas:
    It should be possible to have a non shared ram-index on each node, with jgroups to tell all nodes to update their index so you won't have any locking problems. This solution would work without any infinispan clustering.
    It should be possible to overide AutoNodeSelector or define NodeSelection programmtically

Do you have any other ideas on how to solve it?


Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 4:53 am 
Expert
Expert

Joined: Thu Jan 08, 2009 6:16 am
Posts: 661
Location: Germany
I found out that if I don't use a MassIndexer on startup, the index is not locked by one node anymore.

But how should I build my initial index if the Massindexer is not working correctly?


Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 6:30 am 
Expert
Expert

Joined: Thu Jan 08, 2009 6:16 am
Posts: 661
Location: Germany
No, I was not a MassIndexer fault: If I start one node, update an entity, start the second one, then the second node gets LockingExceptions again. Obviously there is really an issue when the index-master-node changes after the first master has locked the index.


Last edited by mmerder on Thu Jan 08, 2015 4:39 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 7:52 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hi,
interesting description. Essentially if I undestood you correcty, the MassIndexer takes long to complete, and in parallel the master role changed to a secondary node?
I've created https://hibernate.atlassian.net/browse/HSEARCH-1554

The "Automatic master election" strategy is indeed experimental, and I guess your feedback will need to be incorporated before I remove that warning from the documentation.

A nice solution would be to fix AutoNodeSelector to be conservative about changing an existing master node, but by doing so the policy becomes statefull and needs coordination with the other nodes. Not an easy fix.

I would recommend using the configuration to separately configure the master/slave nodes; if you have to deploy the same EAR you could differentiate the configuration option depending on some environment property, maybe the hostname?

Consider that you can configure Hibernate Search also programmatically.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 9:11 am 
Expert
Expert

Joined: Thu Jan 08, 2009 6:16 am
Posts: 661
Location: Germany
Hello Sanne,

thanks for your answer. No it was not because it took to long, the indexing was finished when I started node 2. It was because that way node 1 had written into the index before node 2 was even startet.

I just have tested other options to verify this behaviour. One was not to index intially at all:
1. start node 1
2. update entity on node 1, so that it gets indexed for the first time: Node 1 is master and updated the index.
3. start node 2:
4. update entity anywhere: now node 2 is master and gets same lock exception because node 1 has lock on it.
5. redeploy on node 1
6. update entity anywhere: now node 1 is master again, it all works fine, because this node already has the lock.

If I first start both nodes before I initially index an entity it works fine, because the second node is master and also the first to lock the index. But in production I cannot guarantee that both nodes are started before an entity gets updated.

I also tried a very simple configuration:
Code:
<property name="hibernate.search.default.directory_provider" value="infinispan"/> 
         <property name="hibernate.search.​default.​exclusive_index_use" value="false"/> 
         <property name="hibernate.search.​INCIDENT_INDEX.​exclusive_index_use" value="false"/> 


I debbuged the code and was wondering why there is always an ExclusiveIndexWorkspaceImpl used instead of a shared one (within LuceneBackendResources).
I debugged some more and found out that the MaskedProperty class always hides the exclusive_index_use option.
You can see that by setting a breakpoint in IndexManagerHolder, line 322: now the toString of
rootCfg is = {​INCIDENT_INDEX.​exclusive_index_use=false, default.directory_provider=infinispan, hcore.session_factory=null, ​default.​exclusive_index_use=false}
but the one of globalProperties is = {directory_provider=infinispan}

So I guess, there is an error, too. What do you think? If the indexes were just shared it should all work.

How can I change Index properties programmatically? I don't know right now, how I could programmatically define a jgroups master and a slave, I didn't find this in the reference guide.

_________________
-----------------
Need advanced help? http://www.viada.eu


Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 10:17 am 
Expert
Expert

Joined: Thu Jan 08, 2009 6:16 am
Posts: 661
Location: Germany
Ok, forget about the debugging point: It's kind of embarrassing for me. ;-)
I copied the configuration from your reference and there have been special (NON_ANSI) characters in it:
Code:
<property name="hibernate.search.​default.​exclusive_index_use" value="false"/> 


These are not visible in eclipse, also not in my post above...

Now it's working with a shared index, also with jgroups. But you have to enable exclusive index use to get it working.

Code:
<property name="hibernate.search.default.directory_provider" value="infinispan"/> 
<property name="hibernate.search.default.exclusive_index_use" value="false"/> 
<property name="hibernate.search.INCIDENT_INDEX.exclusive_index_use" value="false"/>
<property name="hibernate.search.default.worker.backend" value="jgroups"/>


Last edited by mmerder on Thu Jan 08, 2015 4:39 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 1:39 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Thanks I'll need to test that, travelling now.

The programmatic configuration feature is something inherited from Hibernate ORM:

http://docs.jboss.org/hibernate/orm/4.3/manual/en-US/html_single/#configuration-programmatic

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 2:00 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Almost forgot: you can also inject environment properties directly in the configuration file using this syntax:

Code:
hibernate.search.Robots.indexwriter.max_merge_docs = ${robots.max_merge_docs}


and run the JVM with -Drobots.max_merge_docs=42

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Hibernate Search 4.4.2 on two standalone JBoss EAPs 6.2
PostPosted: Tue Mar 11, 2014 2:24 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Quote:
Ok, forget about the debugging point: It's kind of embarrassing for me. ;-)
I copied the configuration from your reference and there have been special (NON_ANSI) characters in it:


Ah good to see that :)
(My replies above where written before I saw your update)

I actually had the same problem once, didn't realize the special characters where coming from the documentation. I'm looking forward to provide a better validation of configuration properties in 5.0.

Quote:
But you have to enable exclusive index use to get it working.

Yes that's expected. Again we could do a better job at validating configurations.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 9 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.