I'm running Hibernate Search in a clustered environment and when a particular entity is deleted I keep running into the following exception which has a fatal impact on the application.

This is pretty urgent so if someone has any ideas it would be appreciated.

I'm hoping it might be something as simple as increasing the acquire lock timeout. If so how would you go about doing this?

16:23:50,864 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[167936]:PERSISTENT, de
liveryId=0
javax.ejb.EJBTransactionRolledbackException: Unable to remove from Lucene index: Submissi
on#74640
at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:87)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
at $Proxy301.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate
$run_N8003352271541955702.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate
$run_N8003352271541955702.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.hibernate.search.SearchException: Unable to remove from Lucene index: Subm
ission#74640
at org.hibernate.search.backend.impl.lucene.LuceneWorker.remove(LuceneWorker.java:109)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:80)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:46)
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.run(LuceneBackendQueueProcessor.java:98)

at org.hibernate.search.backend.impl.jms.AbstractJMSHibernateSearchController.onMessage(AbstractJMSHibernateSear
chController.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
at org.jboss.seam.intercept.EJBInvocationContext.proceed(EJBInvocationContext.java:44)
at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:101)
at org.jboss.seam.intercept.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:50)
at sun.reflect.GeneratedMethodAccessor763.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerIntercep
tor.java:54)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
... 23 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@C:\Java\jboss-4.2.2\se
rver\tat\data\searchindex-master\Submission\write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:85)
at org.apache.lucene.index.DirectoryIndexReader.acquireWriteLock(DirectoryIndexReader.java:250)
at org.apache.lucene.index.IndexReader.deleteDocument(IndexReader.java:725)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.remove(LuceneWorker.java:104)
... 47 more

Hi,

could you give some more information? How do your master/slave configuration look like? And how does the code look like between opening and closing the Hibernate session?

Are you saying that even when you manually remove the lock file and restart the application it will fail again at exactly the same Submission?

--Hardy

Master and slave configuration is similar to that described in the Hibernate Search in Action book.

Restarting the application really wont be an option in our case and shouldn't be in most deployments.

Deleting the the index lock resolves the fatal problem with the application however the index is not updated to reflect the removed entity.

I'm not sure offhand how or why this lock was left around and never cleaned up.

There must be a clean solution since I don't want to have to get into writing hacks where I need to programmatically remove locks myself followed by some form of reindex to get the index back into sync.

I did some looking and found this:

"If you are already doing that (single writer) correctly, the other
common cause is that this is a leftover lock file (for example if the
JVM crashed or was killed or even if you didn't close a previous
writer before the JVM exited). There is a better locking
implementation (NativeFSLockFactory) that correctly frees the lock
when the JVM crashes so you may want to use that one instead if you
hit this often (but first explain the root cause of your crashes!). "

I am extending AbstractJMSHibernateSearchController MDB and implemented as described in the book I'm assuming that even if there are more then one MDB processing the queue HS will handle the synchronization.

If this assumption is correct then it might be possible that the application had a fatal error or was killed at the point the update was taking place and the above quoted scenario occurred. If so then this mentioned NativeFSLockFactory might be the appropriate solution and I would think this change would need to be implemented in HibernateSearch since I don't see an option to configure this.

Hi,

I was not suggesting that you implement some sort of hack. I was just trying to get to the bottom of this. I've seen this left over lock files before as a result of a JVM crash.

NativeFSLockFactory might be a solution. I recommend you open a Jira issue for this.

--Hardy

Thanks I have opened HSEARCH-284.