Lucene / Hibernate Search Lock Exception

wizche · **Joined:** Wed Aug 14, 2013 3:19 am **Posts:** 7

I've started a thread on stackoverflow, but I was suggested to move the discussion here:

http://stackoverflow.com/questions/18085526/lucene-hibernate-search-lock-exception

UPDATE:
After changing lock policy to native (and rebuild a clean new index) I now receive the following exception (instead of the NPE with the simple locking strategy). The concurrency seems to not be correctly handled:

Code:

09:42:48,274 ERROR LogErrorHandler:82 - Exception occurred org.apache.lucene.store.LockReleaseFailedException: [b]Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /opt/myapp/item_index/myapp.item.domain.Item/write.lock[/b]

org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /opt/myapp/item_index/myapp.item.domain.Item/write.lock
        at org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:294)
        at org.apache.lucene.index.IndexWriter.unlock(IndexWriter.java:4277)
        at org.hibernate.search.backend.Workspace.forceLockRelease(Workspace.java:284)
        at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:127)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Thanks for the help!

sanne.grinovero · **Posted:** Wed Aug 14, 2013 4:33 am

Hi,
ok that's giving a better clue.
It looks like you're having two Hibernate Search instances running, could you verify that? You really want a single engine dealing with the index exclusively!

I've never seen such an error reported before, I suspect the fact that you're extending the FullTextIndexEventListener could be triggering old bug https://hibernate.atlassian.net/browse/HSEARCH-597, getting you possibly a second indexing engine registered automatically when it fails to recognize the existing one.

You might want to try disabling auto-registration

Code:

hibernate.search.indexing_strategy = manual

http://docs.jboss.org/hibernate/search/3.4/reference/en-US/html_single/#search-configuration-event

wizche · **Joined:** Wed Aug 14, 2013 3:19 am **Posts:** 7

Thanks Sanne, how can I check if I have two Hibernate Search instances running?

I don't get why after each lucene query with matches, the processWork is being called on each item in the result set. Do you modify the index for each matched item?

sanne.grinovero · **Posted:** Wed Aug 14, 2013 4:56 am

Quote:

how can I check if I have two Hibernate Search instances running?

With a debugger ;)

Quote:

I don't get why after each lucene query with matches, the processWork is being called on each item in the result set. Do you modify the index for each matched item?

No that's not normal. The processWork is triggered only on dirty entities (entities on which you have applied some change), but it's possible you have something wrong in your entities which makes them flagged as dirty just after loading, for example if you have some computed fields but you forgot to annotate them with @Transient.

wizche · **Joined:** Wed Aug 14, 2013 3:19 am **Posts:** 7

With default configuration (previous to your suggestion) in the Thread stack I see:

After disabling event indexing (and removing locking strategy native), I could not reproduce the error anymore. Furthermore I cannot spot any Hibernate Search thread anymore.

sanne.grinovero · **Posted:** Wed Aug 14, 2013 5:53 am

What do you mean?
I'm not sure if you're saying that the problem is resolved, or if you completely disabled Hibernate Search.

Regarding the debugger: I would look at how many SearchFactory implementations you have registered in the ORM event listeners chain.

A simpler solution would be to verify that Search works fine when you remove your custom event listener, that really is the source of all trouble.

wizche · **Joined:** Wed Aug 14, 2013 3:19 am **Posts:** 7

Quote:

What do you mean?
I'm not sure if you're saying that the problem is resolved, or if you completely disabled Hibernate Search.

I mean I set the

Code:

<prop key="hibernate.search.indexing_strategy">manual</prop>

And after that no processWork is triggered (nor my custom indexer). Search work, but index are not updated anymore (which is supposedly the purpose of this property). And the problem didn't appeared.

Quote:

Regarding the debugger: I would look at how many SearchFactory implementations you have registered in the ORM event listeners chain.

Can you explain further where do I find the event listeners chain? (class?)

Quote:

A simpler solution would be to verify that Search works fine when you remove your custom event listener, that really is the source of all trouble.

sanne.grinovero · **Posted:** Wed Aug 14, 2013 7:17 am

Quote:

And after that no processWork is triggered (nor my custom indexer). Search work, but index are not updated anymore (which is supposedly the purpose of this property). And the problem didn't appeared.

How are you registering your custom event listener?

wizche · **Joined:** Wed Aug 14, 2013 3:19 am **Posts:** 7

sanne.grinovero wrote:

Quote:

And after that no processWork is triggered (nor my custom indexer). Search work, but index are not updated anymore (which is supposedly the purpose of this property). And the problem didn't appeared.

How are you registering your custom event listener?

I'm setting the eventListeners property in my SessionFactory like this:

Code:

      
   <bean id="itemSessionFactory"
      class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
      <property name="dataSource" ref="itemDataSource" />
      <property name="mappingResources">.......</property>
      <property name="hibernateProperties">
         <ref bean="hibernateItemProperties" />
      </property>
      <!-- event listener used for hibernate search -->
      <property name="eventListeners">
         <map>
            <entry key="post-update">
               <bean class="myapp.search.eventlistener.HibernateItemEventListener" />
            </entry>
            <entry key="post-insert">
               <bean class="myapp.search.eventlistener.HibernateItemEventListener" />
            </entry>
            <entry key="post-delete">
               <bean class="myapp.search.eventlistener.HibernateItemEventListener" />
            </entry>
            <entry key="post-collection-recreate">
               <bean class="myapp.search.eventlistener.HibernateItemEventListener" />
            </entry>
            <entry key="post-collection-remove">
               <bean class="myapp.search.eventlistener.HibernateItemEventListener" />
            </entry>
            <entry key="post-collection-update">
               <bean class="myapp.search.eventlistener.HibernateItemEventListener" />
            </entry>
         </map>
      </property>
   </bean>

wizche · **Joined:** Wed Aug 14, 2013 3:19 am **Posts:** 7

Could you please explain me how can I identify which property is making my item goes dirty? Where does Lucene/Hibernate decide the entity is dirty? If we could avoid the processWork to be triggered after each search we could solve the multiple index writer problem for sure.

The indexed properties, are 100% not changed after a search.

wizche · **Joined:** Wed Aug 14, 2013 3:19 am **Posts:** 7

I think I solved it.

The problem was not the Hibernate Search itself, it was Atomikos which killed threads after the concurrent transaction threshold has been exceeded (probably leaving the file descriptor open). I increased the concurrent transaction limit (it was 50) and the problem was gone.

Despite this, I still don't understand why is hibernate search opening an index writer each time I perform a lucene query.