Hibernate Search and Infinispan Directory lock timeout issue

hmb · **Joined:** Fri May 11, 2012 5:55 am **Posts:** 4

Hi all,

we have been using Hibernate Search for quite a while now and are very happy about its easy of use. Recently we have been looking into preparing our application for a load balanced setup. We are running on Amazon Web Services (AWS) and our configuration for all instances of the application is identical. We thought that a master / slave setup was therefore unsuitable (at least until the auto detection / master election is available) and we choose to try the Infinispan approach. We use the default Infinispan configuration with a custom JGroups configuration (JDBC_PING) file to setup the cluster. The cluster seems to form correctly since the index from the first server becomes available on the second server that starts. Changes from the first server also become direct available on the second server. However, if I try to change something on the second server an exception occurs:

Code:

ERROR 11:39:44.296 default o.h.s.exception.impl.LogErrorHandler - HSEARCH000058: Exception occurred org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@2a10b73d
Primary Failure:
   Entity nl.usmedia.domain.model.Entity  Id 403071  Work Type  org.hibernate.search.backend.AddLuceneWork

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@2a10b73d
   at org.apache.lucene.store.Lock.obtain(Lock.java:84) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]
   at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:127) ~[hibernate-search-engine-4.1.1.Final.jar:4.1.1.Final]
   at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:102) ~[hibernate-search-engine-4.1.1.Final.jar:4.1.1.Final]
   at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:119) ~[hibernate-search-engine-4.1.1.Final.jar:4.1.1.Final]
   at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99) ~[hibernate-search-engine-4.1.1.Final.jar:4.1.1.Final]
   at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) ~[hibernate-search-engine-4.1.1.Final.jar:4.1.1.Final]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_03]
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_03]
   at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_03]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_03]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_03]
   at java.lang.Thread.run(Thread.java:722) [na:1.7.0_03]
ERROR 11:39:44.296 default o.h.s.b.i.l.LuceneBackendQueueTask - HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!

This exception occurs after a slight delay, but consistently after every change.

The search related properties that are provided to the EntityManager are:

Code:

        hibernate.search.default.directory_provider=infinispan
        hibernate.search.infinispan.configuration_resourcename=development-hibernatesearch-infinispan.xml
        hibernate.search.​default.​exclusive_index_use = false
        hibernate.search.lucene_version = LUCENE_35

We have been trying to find people with similar issues, but most seem to be related to heavy load, restarts, or other (in my (humble) opinion) unrelated issues. Does anybody know what we are doing wrong, or how we could solve this issue?

We are using:

Hibernate 4.1.3.Final
Hibernate Search 4.1.1.Final

Thanks a lot!,

with kind regards,

Hidde.

sanne.grinovero · **Posted:** Sun May 13, 2012 7:56 pm

Hi,
there are two problems to consider when needing to replicate the index:

1) Make sure that both applications can read/search the same shared index.
This can be solved either by using:
A) the master/slave directories, which do a periodic copy via some shared filesystem
B) via the Infinispan Directory.

2)Making sure that only one node attempts to write at the same time
This needs to configure a specific backend, you have three options:
I) don't configure anything, and assume that each node will be able to acquire the lock. Only suitable if you know a single node writes to the indexes by application design
II) use JMS to have one node send his changes to be applied by the other node (master)
III) use JGroups to replace JMS (again you have a master - might be simpler to configure but is less reliable as there as queue provides no transactional persistence)

It seems you solved the first problem only, by sharing the index via an Infinispan Directory. I'm glad you figured out the documentation to configure it on EC2 BTW ;)

For problem 2), it seems you are in scenario I : not configured a master backend. And I guess this wasn't a conscious choice?
I'd suggest to configure a JMS queue between the two nodes.
I guess we should cleanup the documentation to clarify this is a dual-ingredients recipe.. any suggestion?

BTW since you're looking into load-balancing I guess you're interested in better performance; consider that using the JMS or JGroups backend you can set exclusive_index_use=true which is much more efficient.

hmb · **Joined:** Fri May 11, 2012 5:55 am **Posts:** 4

Hi Sanne,

thanks a lot for your reply. I read somewhere that Infinispan could be an alternative for the master/slave backend, so I thought I did not need it. I guess I assumed that once the first node was done writing, it would release the lock allowing the second node to write to the index. I now understand that I still need to configure a JMS or JGroups backend to get rid of the locking issues and that I can set the exclusive index use back to true.

The reason I was still in scenario 2-I was conscious (although possibly flawed): The nodes are started and stopped based on the load they endure. However, the first node started is usually also the first one stopped which means that the master will no longer be available. I do have some servers outside this cluster which are more static and I'll try to assign one of those the master role.

I am currently out of the office for a couple of days, but I will have a look at the documentation once I'm back.

Thanks a lot for your help,

with kind regards,

Hidde.

hmb · **Joined:** Fri May 11, 2012 5:55 am **Posts:** 4

Hi Sanne,

I forgot all about my promise to get back to you on the documentation. My apologies. I have have read through chapter 3 a little an I think I would expect some heads up in 3.3.1. (Infinispan Directory configuration). Possibly around the third paragraph where the docs talk about the Infinispan Directory being considered one directory and that updates will be propagated throughout the cluster. Something along the line of:

Quote:

Please note that (even though Infinispan takes care of distributing your updates) a master/slave backend [refer to 3.4?] has to be configured to make sure only one node writes to the index.

I hope this helps,

with kind regards,

Hidde.

sanne.grinovero · **Posted:** Tue Jun 12, 2012 11:36 am

Hi,
good point. I'll update it.

Thanks!
Sanne