Hibernate Search version: 3.0.1 GA

I am getting org.hibernate.search.SearchException: Unable to open IndexReader for class .. , when I am trying to run a Junit Test to update Index from a database. This happening when my application is deployed in Jboss server which is accessing the same Index. I would like to know the configurations I missed so that I can access, update the same index in the case Multiple session factory threads accessing the same index(as is the case with the application and test ).

Do need to extend the ReaderStrategy ?

Or is it due to coruptions of index (see the stack trace, what is Unknownformat: 7)?

Plese give me some directions in this case.

this the only configuration i have in my hibernate.cfg.xml

<property name="search.default.directory_provider">org.hibernate.search.store.FSDirectoryProvider</property>
name="search.default.indexBase">/home/application/index</property>


This is the stack trace.

2009-01-09 13:10:54.0800 ERROR main org.hibernate.transaction.JDBCTransaction - exception calling user Synchronization
org.hibernate.search.SearchException: Unable to open IndexReader for class com.testapp.domain.Task
at org.hibernate.search.backend.Workspace.getIndexReader(Workspace.java:84)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.remove(LuceneWorker.java:92)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:80)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:46)
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.run(LuceneBackendQueueProcessor.java:98)
at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:149)
at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.afterCompletion(PostTransactionWorkQueueSynchronization.java:50)
at org.hibernate.transaction.JDBCTransaction.notifyLocalSynchsAfterTransactionCompletion(JDBCTransaction.java:289)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:152)
at com.dao.TaskDAO.createTaskIndex(TaskDAO.java:1711)
at com.commandtest.IndexUpdateTest.testTaskUpdate(TaskIndexUpdateTest.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
at org.junit.runners.Suite.runChild(Suite.java:115)
at org.junit.runners.Suite.runChild(Suite.java:23)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:116)
at org.junit.runner.JUnitCore.run(JUnitCore.java:107)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:88)
at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:54)
at org.junit.runner.JUnitCore.main(JUnitCore.java:46)
Caused by: org.apache.lucene.index.CorruptIndexException: Unknown format version: -7
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:211)
at org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:68)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:636)
at org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:63)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:209)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:192)
at org.hibernate.search.backend.Workspace.getIndexReader(Workspace.java:80)
... 37 more

This should be the problem; the index is either corrupted and you should delete it, or it has been created or updated by a newer version of Lucene.
Indexes are (usually) forward compatible (you can update Lucene) but not often backwards.
Search 3.1 uses Lucene to 2.4, Search 3.0.1 used Lucene 2.3
Could you have mixed some version or sharing the index directory with another version?

I did never experience directly an index corruption, this happens if you have filesystem problem or if you had to kill your application, so I would check for versions first unless you have had to kill it.

Thanks for the reply.

My scenario is , my application is running with an index and database. Occasionally I get sql dump for one of the indexed objects. I update the database with the dump and run a Junit test (mass indexing of that indexed object) manually to update the index from command prompt. The problem is only when I delete the corresponding index directory of the object I am able to run Junit test. If I updated an object of that type from Application and again run the Junit test I am getting the above exception. I am using same cfg.xml configuration for both application and test. Is the index corruption is due to combination of manual(test) and synchronous(with in the application) approach? How to maintain a stable index when multiple applications ,tests access/update same index, I know Lucene do pessimistic lock on Index while writing but why corruption taking place?

I am using the same jars to run both the test and the application.

Sorry, I don't get this. Why should you need to delete the index?
Did you try the purge and purgeAll methods in H.Search?

I have just modified the Junit test code like this:


@SuppressWarnings("unchecked")
public void createTaskIndex()
{
if (mLogger.isInfoEnabled())
{
mLogger.info("Entered createTaskIndex method of TaskDAO");
}

int BATCH_SIZE = 1000;
int FETCH_SIZE = 100;
try
{
Session sessionH = HibernateUtil.getSession();
FullTextSession session = Search.createFullTextSession(sessionH);
Transaction tx = session.beginTransaction();
//purge and flush
session.purgeAll( Task.class );
session.flush();
session.getSearchFactory().optimize( Task.class ); // exception occuring at this place


Criteria query = session.createCriteria( Task.class )
.setResultTransformer(

CriteriaSpecification.DISTINCT_ROOT_ENTITY) // Minimize cache interaction
.setCacheMode(CacheMode.IGNORE)
.setFetchSize(FETCH_SIZE) //Align batch size and JDBC fetch size
.setFlushMode(FlushMode.MANUAL);
//Disable flush

ScrollableResults scroll = query.scroll(
ScrollMode.FORWARD_ONLY); //Scroll in forward mode only

int batch = 0;
scroll.beforeFirst();
while ( scroll.next() ) {
batch++;
session.index( scroll.get(0) );
if (batch % BATCH_SIZE == 0) {

session.flush(); // flushtoindexs() not available in 3.0.1
session.clear();
}
}


//
// this is the code before modification..
// List<Task> taskList = session.createQuery("from Task as task").setFirstResult(pStartIndex).setMaxResults(
// 1000).list();
// for (Task task : taskList)
// {
// fullTextSession.index(task);
// }
tx.commit();
}
catch (HibernateException he)
{
mLogger.error("Caught Lucene query parsing exception and message is::", he);
throw new InfrastructureException(he);
}
if (mLogger.isInfoEnabled())
{
mLogger.info("Leaving createTaskIndex() method of TaskDAO");
}
}


I am getting Indexwriter exception this time but cause is same : corrupted index ,
Occuring when i re-run the test after updating a task through application.


org.hibernate.search.SearchException: Unable to open IndexWriter for class com.appl.domain.Task
at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:135)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:126)
at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(LuceneWorker.java:49)
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.run(LuceneBackendQueueProcessor.java:98)
at org.hibernate.search.impl.SearchFactoryImpl.optimize(SearchFactoryImpl.java:247)
at com.dao.TaskDAO.createTaskIndex(TaskDAO.java:1755)
at com.test.IndexUpdateTest.testTaskUpdate(IndexUpdateTest.java:18)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:38)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Caused by: org.apache.lucene.index.CorruptIndexException: Unknown format version: -7
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:211)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:254)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:636)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:251)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:710)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:503)
at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:118)
... 25 more

This isn't working correctly, the flush() does flush only changes to DB not to the index, so you can control DB and index separately.
Instead of flushing you should commit the transaction and close the session; then you may begin a new one for the indexing.

Or upgrade to version 3.1 to make use of flushToIndexes(), that feature was not available at 3.0.1.

Thank you , I have modified the code as suggested but the exception remains the same.

@SuppressWarnings("unchecked")
public void createTaskIndex()
{
if (mLogger.isInfoEnabled())
{
mLogger.info("Entered createTaskIndex method of TaskDAO");
}

int BATCH_SIZE = 1000;
int FETCH_SIZE = 100;
try
{
Session sessionH = HibernateUtil.getSession();
FullTextSession session = Search.createFullTextSession(sessionH);
Transaction tx = session.beginTransaction();
//purge and flush
session.purgeAll( Task.class );
//session.flush();
tx.commit();
sessionH.close();

sessionH = HibernateUtil.getSession();
session = Search.createFullTextSession(sessionH);
tx = session.beginTransaction();

session.getSearchFactory().optimize( Task.class ); // **** Exception here


Criteria query = session.createCriteria( Task.class )
.setResultTransformer(

CriteriaSpecification.DISTINCT_ROOT_ENTITY) // Minimize cache interaction
.setCacheMode(CacheMode.IGNORE)
.setFetchSize(FETCH_SIZE) //Align batch size and JDBC fetch size
.setFlushMode(FlushMode.MANUAL);
//Disable flush

ScrollableResults scroll = query.scroll(
ScrollMode.FORWARD_ONLY); //Scroll in forward mode only

int batch = 0;
scroll.beforeFirst();
while ( scroll.next() ) {
batch++;

if(!sessionH.isOpen()){
sessionH = HibernateUtil.getSession();
session = Search.createFullTextSession(sessionH);
tx = session.beginTransaction();
}

session.index( scroll.get(0) );
if (batch % BATCH_SIZE == 0) {

tx.commit();
sessionH.close();
}
}


//
//
// List<Task> taskList = session.createQuery("from Task as task").setFirstResult(pStartIndex).setMaxResults(
// 1000).list();
// for (Task task : taskList)
// {
// fullTextSession.index(task);
// }
tx.commit();
}
catch (HibernateException he)
{
mLogger.error("Caught Lucene query parsing exception and message is::", he);
throw new InfrastructureException(he);
}
if (mLogger.isInfoEnabled())
{
mLogger.info("Leaving createTaskIndex() method of TaskDAO");
}
}
}

It looks like the two Lucene instances don't "see" the lock at the correct time.
Are you running JBoss and the unittest on the same machine? NFS/Samba/other network FS are known to be broken with lock files.

I had a similar requirement once, I moved the indexing code to the application server and created a "index administration panel" having a button which triggers (re)indexing of the entities: one button per entity type; quite useful for the admin.

Yes I am running both in the same machine

I'm sorry I'm running out of ideas.
It looks as a locking issue, most probably a Lucene issue, you should ask to the Lucene forum..
can't you update to latest Search and Lucene 2.4? maybe it has been fixed already, in version 2.4 some corruption issues were fixed.
You may want to search for "corrupt" on this page:
http://issues.apache.org/jira/browse/LUCENE?report=com.atlassian.jira.plugin.system.project:changelog-panel

Also you probably want to double check your classpath, please be sure you are not using different lucene versions in the test or server.

Sorry about this Sanne. It was my mistake. The problem was due to a old lucene jar in the class path. I removed it and its working fine. Thanks for the quick responses.

no problem, thanks for letting us know!