LockObtainFailedException and configuring Lucene LockFactory

zentropy · **Joined:** Wed Dec 05, 2007 9:17 pm **Posts:** 4

Hello,

I am using Hibernate version 3.0.0.GA, and I am running into a couple of issues whenever my index is updated.

Each developer is running his/her own instance of Tomcat, and we are all sharing the same mount drive that hosts the Lucene indexes. This will be very similar to production, where multiple Tomcat instances will share the same indexes on a single mount point.

Hibernate Search is used in my application to search for products. Whenever a new product is created (or an existing product is edited and saved), I (semi-consistently) get the following stack trace:

Code:

18:56:33-217 ERROR te.transaction.JDBCTransaction   exception calling user Synchronization
org.hibernate.search.SearchException: Unable to open IndexWriter for class com.mycompany.product.metadata.SomeMetadata
        at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:135)
        at org.hibernate.search.backend.impl.lucene.LuceneWorker.add(LuceneWorker.java:68)
        at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:62)
        at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:43)
        at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.run(LuceneBackendQueueProcessor.java:98)
        at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:132)
        at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.afterCompletion(PostTransactionWorkQueueSynchronization.java:50)
        at org.hibernate.transaction.JDBCTransaction.notifyLocalSynchsAfterTransactionCompletion(JDBCTransaction.java:243)
        at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:121)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:558)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:319)

        (snip)

Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@\\some_drive\search_index\SomeMetadata\write.lock
        at org.apache.lucene.store.Lock.obtain(Lock.java:70)
        at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:598)
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:410)
        at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:118)
        ... 76 more

The new product or product changes get saved into the DB, but the index is not updated.

Looking on the shared drive, the lock file is indeed still there:

\\some_drive\search_index\SomeMetadata\write.lock

I looked and saw some posts on the Lucene forum that the default SimpleFSLock class should be swapped out in favor of NativeFSLockFactory.

Do you think that this is what's causing the error? If so, how do I go about changing out the default SimpleFSLock? I couldn't find this in the Hibernate Search docs. Is this something that's exposed through Hibernate Search?

My second issue has to do with throughput. When I save a product, the response time is around 20-30 seconds. Do you think this is due to the fact that I am using the synchronous worker execution mode? Should I change that to use asynchronous?

Thank you in advance for your help!

-Frank

emmanuel · **Posted:** Fri Mar 14, 2008 10:47 am

hi
The lock configuration is not exposed by Hibernate Search. can you open a JIRA issue so that we can add it.

More globally, your architecture will probably not scale (depends how many inserts/updates and how many // tomcats). Lucene needs a pessimistic lock when it updates the index, so the lock is shared by all tomcat instances.

asynchronous will help to hide the lock wait period to the user but does not really solve the fundamental problem. You should use the MDB backend (or some similar architecture to scale your app).

http://www.hibernate.org/hib_docs/search/reference/en/html_single/#d0e422

http://www.hibernate.org/hib_docs/search/reference/en/html_single/#jms-backend

zentropy · **Joined:** Wed Dec 05, 2007 9:17 pm **Posts:** 4

Hi Emmanuel,

Thank you very much for your quick reply. :)

We will have two Tomcat instances running on production. The original thinking is to use the configuration documented in Hibernate Search:

http://www.hibernate.org/hib_docs/search/reference/en/html/search-architecture.html#d0e408

... of having the two Tomcats sharing the same search index.

There will be less than 20 users, making edits and saving Products with the application. The DB insert/update runs before the search index update within the same transaction. Since this is a administrative Web app with a low number of users, I would have thought that the above setup would be sufficient.

Our Entity relationships span several layers deep. Do you think that the more complex the relationships, the longer it takes to perform incremental indexing?

One key requirement that I have is to reflect the Product insert/updates in the Search results as soon as possible. After reading the Hibernate Search docs re. JMS, it seems that this may be achieved by configuring a low number for the hibernate.search.default.refresh property. However, I would like to simplify the design to not use a third "master" instance to perform updates to the index.

Can you tell me if doing the following is possible with Hibernate Search:

1) 2 Tomcat instances, one running a JMS queue.

2) Both instances accepts search requests.

3) Both instances accepts Product insert/edit requests

4) Insert/edits are published to the JMS queue on the designated instance

5) The designated instance is solely responsible for writing to the index

6) There will only be one index that is shared between the two Tomcat instances (i.e. no master/slave setup, and hence no copying of index files)

Do you think this is possible?

Also, the documentation mentioned that I could use AbstractJMSHibernateSearchController without a MDB (we are running Tomcat) - how would I go about doing so? We are using Spring - can I just configure it as a Spring bean and point (somehow) Hibernate Search to it?

Best,

-Frank

emmanuel · **Posted:** Mon Apr 07, 2008 5:21 pm

Sorry for the long delay

Yes I'm pretty sure you can do the described architecture. I have to admit I have not tested it since it requires a NFS system and will make indexing slower.

I don't know Spring enough to know if it has some sort of JMS queue reader that triggers a bean call, but if it does, then yes you will just ahve to extend the hibernate search class.
At that point though (Servlet container, + component model library + JMS queue + Message Driven Bean), I would probably go to a vanilla app server to avoid too many config overhead and testing.