Lock not released by MassIndexer HSearch 4

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

Hi,

I'm testing against Hibernate Search 4.1.0Alpha1 and I have a problem with the lock not being released.
First of all, I'm using lockstrategy is native, which doesn't seem to work. If I kill the Tomcat process, the lock file is still there (Windows 7).

Code:

<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
        <property name="dataSource" ref="dataSource"/>
        <property name="configLocation" value="classpath:hibernate.cfg.xml"/>
        <property name="hibernateProperties">
            <value>
               hibernate.bytecode.provider=javaassist
               hibernate.connection.driver_class=${jdbc.driverClassName}
               hibernate.connection.is-connection-validation-required=true
               hibernate.jdbc.batch_size=30
                hibernate.dialect=${hibernate.dialect}
                hibernate.use_sql_comments=false
                hibernate.query.substitutions=true
                  hibernate.generate_statistics=true
                hibernate.search.default.directory_provider=filesystem

                hibernate.search.default.locking_strategy=native
                hibernate.search.default.indexBase=${hibsearch.hibernate_index_dir}
                hibernate.search.analyzer=nl.project.dao.hibernate.search.MyStandardAnalyzer
                hibernate.search.autoregister_listeners=true
[...]

Second, MassIndexer creates a lock when it starts running but after it completes, it's still there.

The code for calling MassIndexer is

Code:

fts.createIndexer(clazz).purgeAllOnStart(true)
                  .optimizeOnFinish(true).threadsToLoadObjects(3)
                  .batchSizeToLoadObjects(30)
                  .threadsForSubsequentFetching(12).startAndWait()

Any suggestions?

Marc

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

Right, eternal locks are default unless you set
hibernate.search.default.exclusive_index_user=false

Doesn't seem like an awesome default to me.

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

Hmm, after MassIndexing, Hibernate keeps a write.lock file in the directory, even if
hibernate.search.default.exclusive_index_user=false

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

The lock also leads to subsequent MassIndexer failures:

Code:

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@h:\java\projects\indexDir\nl.msw.compraventa.model.Offer\write.lock
   at org.apache.lucene.store.Lock.obtain(Lock.java:84)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)
   at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:125)
   at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:100)
   at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:114)
   at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:101)
   at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:69)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

sanne.grinovero · **Posted:** Thu Feb 02, 2012 4:09 pm

The property name is exclusive_index_use , does not have a final r

I don't know what Windows7 should do with the "native" lock; It uses a NIO file lock, so it's up to the operating system to cleanup. If our process is killed, we can't do much anymore to clean the lock :-/

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

Ok, thanks. Not sure if you saw my comment on JIRA but it does seem like the MassIndexer never releases it's write lock even with setting correctly applied.

sanne.grinovero · **Posted:** Mon Feb 06, 2012 12:13 pm

Quote:

Not sure if you saw my comment on JIRA but it does seem like the MassIndexer never releases it's write lock even with setting correctly applied.

Thanks! I'll add a test and fix that soon.

lovelyliatroim · **Joined:** Thu Oct 08, 2009 10:34 am **Posts:** 55

Hey Guys,

Just stepping up from 3.4 to 4.1 and i seem to be having the same issue with regard the "write.lock" file still being present after a mass index!!

Maybe i have missed some special config when stepping up !!

My config looks like so

Quote:

#Hibernate Search properties

#Master Directory config
## DirectoryProvider
# (remote) master location where information is copied to
hibernate.search.default.sourceBase=C:/lucene/shared

# local master location
hibernate.search.default.indexBase=C:/lucene/master

# refresh every half hour,defined in secs
hibernate.search.default.refresh=120

# appropriate directory provider
hibernate.search.default.directory_provider=filesystem-master

I dont think i need more config for the upgrade(I might have missed something).

Like i said, after i do a mass index on an entity the "write.lock" file remains after it!! I see this on XP!!

Cheers,
LL

sanne.grinovero · **Posted:** Mon Apr 30, 2012 9:11 am

Hi,
if you're still having the lock, and you're using native locking, it very likely means the application which acquired the lock is still running.

Please check it's not running in background? And don't forget to shutdown Hibernate correctly! (using SessionFactory#close or javax.persistence.EntityManagerFactory#close() depending if you are using plain Hibernate or JPA)

lovelyliatroim · **Joined:** Thu Oct 08, 2009 10:34 am **Posts:** 55

Quote:

Please check it's not running in background? And don't forget to shutdown Hibernate correctly! (using SessionFactory#close or javax.persistence.EntityManagerFactory#close() depending if you are using plain Hibernate or JPA)

Sorry for late reply......buts surely you mean the entityManager.close and not the factory!! If the factory then how do i bypass this, i dont want to tear up and tear down the factory everytime my app has indexed!!

lovelyliatroim · **Joined:** Thu Oct 08, 2009 10:34 am **Posts:** 55

Ok i think it is just a step up issue and change in behaviour

exclusive_index_use is defaulted to true which means that the write.lock is exists till the death(ore factory is teared down and recreated), i was use to the 3.x version seeing the write.lock file being removed after it had finished indexing. And it appears in the 4.x that if this is set to true then it will exist until the factory is destroyed.

If i set exclusive_index_use to false then it is the same behaviour i was use to in the 3.x version.

I guess it make no difference to the clients in a master/slave setup that they now have a write.lock file in their index i this is set to "true"??

sanne.grinovero · **Posted:** Tue Jul 24, 2012 11:48 am

Hi,
yes on all your findings and questions.

It's not a problem for Hibernate Search that the lock file is still there, since all its internal functions are routed to the same IndexWriter instance for best performance.

You should set exclusive_index_use=false only if you have other applications writing to the index, so that Hibernate Search acquires the lock for the shortest time possible. Generally I think that should still be avoided, because if you have two applications writing at the same time and often contending the lock you end up with timeout exceptions.