We started to get the following error after we shipped.
We did not see it when we tested, only when we released to production.

We rebuilt our indexes and the problem when away for a bit, but then returned.

We go alot of these "Unable to open Lucene IndexReader" through the night and the we started to get Too Many Files open errors.

I've also pasted our mvn deps at the bottom...

................

Caused by: org.hibernate.HibernateException: could not init listeners
at org.hibernate.event.EventListeners.initializeListeners(EventListeners.java:205)
at org.hibernate.cfg.Configuration.getInitializedEventListeners(Configuration.java:1396)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1385)
at org.hibernate.cfg.AnnotationConfiguration.buildSessionFactory(AnnotationConfiguration.java:954)
at com.conducive.util.hbm.SessionFactoryBuilder.buildDefaultSessionFactory(SessionFactoryBuilder.java:42)
at com.conducive.wicket.PRManagerApplication.buildSessionFactory(PRManagerApplication.java:472)
at com.conducive.wicket.PRManagerApplication.init(PRManagerApplication.java:275)
... 32 more
Caused by: org.hibernate.search.SearchException: Unable to open Lucene IndexReader
at org.hibernate.search.reader.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:119)
at org.hibernate.search.reader.SharingBufferReaderProvider.initialize(SharingBufferReaderProvider.java:96)
at org.hibernate.search.reader.ReaderProviderFactory.createReaderProvider(ReaderProviderFactory.java:75)
at org.hibernate.search.impl.SearchFactoryImpl.<init>(SearchFactoryImpl.java:183)
at org.hibernate.search.event.ContextHolder.getOrBuildSearchFactory(ContextHolder.java:53)
at org.hibernate.search.event.FullTextIndexEventListener.initialize(FullTextIndexEventListener.java:102)
at org.hibernate.event.EventListeners$1.processListener(EventListeners.java:198)
at org.hibernate.event.EventListeners.processListeners(EventListeners.java:181)
at org.hibernate.event.EventListeners.initializeListeners(EventListeners.java:194)
... 38 more
Caused by: java.io.FileNotFoundException: <omitted>_fn2.fnm (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(Unknown Source)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:94)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:691)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:68)
at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:599)
at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:104)
at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:74)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:704)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:314)
at org.hibernate.search.reader.SharingBufferReaderProvider.readerFactory(SharingBufferReaderProvider.java:169)
at org.hibernate.search.reader.SharingBufferReaderProvider$PerDirectoryLatestReader.<init>(SharingBufferReaderProvider.java:243)
at org.hibernate.search.reader.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:114)
... 46 more

.......


<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>hibernate3-maven-plugin</artifactId>
<version>2.2</version>
<configuration>
<components>
<component>
<name>hbm2ddl</name>
<implementation>annotationconfiguration</implementation>
</component>
</components>
<componentProperties>
<configurationfile>/src/main/resources/hibernate.cfg.xml
</configurationfile>
<outputfilename>schema.sql</outputfilename>
<export>false</export>
<update>true</update>
</componentProperties>
</configuration>
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.5</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>${hibernate.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-annotations</artifactId>
<version>${hibernate.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>${hibernate.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-ehcache</artifactId>
<version>${hibernate.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>${hibernate.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-search</artifactId>
<version>3.2.0.CR1</version>
<exclusions>
<exclusion>
<groupId>org.hibernate</groupId>
<artifactId>ejb3-persistence</artifactId>
</exclusion>
</exclusions>
</dependency>
<!--
<dependency> <groupId>org.hibernate</groupId>
<artifactId>hibernate-commons-annotations</artifactId>
<version>3.3.4.ga</version> </dependency> <dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-tools</artifactId>
<version>3.2.4.GA</version> </dependency>
-->
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>4.0.2.GA</version>
</dependency>
</dependencies>
</plugin>

this is quite frequent on web servers as file handlers are limited and as each network connection also takes a file handler.
First thing make sure you've set a "server oriented" limit in your operating system parameters (I guess it's linux?)

Second thing, you should make sure the index is optimized frequently to limit the effective number of files created; read Hibernate Search's manual for optimization strategies.

your current stacktrace looks like it's not able to start, check for pending .lock files in the index directory. If you're sure no process is using the index you should delete these lock files.

BTW the final version was released some days ago with some bugfixes you might want to include (but AFAIK not related to this problem)

The error was happening at run time before I restarted..

I think the TOO Many Open files happened because the indexer wasn't closing file handles because it was crashing.

My guess is that the indexer has a concurrency problem. When it gets under load it starts to complain about files not existing, EOF, etc..


Here are a few examples:


2010-05-07 14:09:05,194 ERROR [Hibernate Search: Directory writer-1] impl.LogErrorHandler - Exception occurred org.hibernate.search.SearchException: Exception while closing IndexWriter

org.hibernate.search.SearchException: Exception while closing IndexWriter
at org.hibernate.search.backend.Workspace.closeIndexWriter(Workspace.java:195)
at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:113)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: read past EOF
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:80)
at org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:73)
at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:111)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:691)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:667)
at org.apache.lucene.index.DocumentsWriter.applyDeletes(DocumentsWriter.java:956)
at org.apache.lucene.index.IndexWriter.applyDeletes(IndexWriter.java:5207)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4370)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4209)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4200)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2195)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2158)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2122)
at org.hibernate.search.backend.Workspace.closeIndexWriter(Workspace.java:191)
... 7 more
...............

2010-05-07 13:58:10,772 WARN [TP-Processor26] wicket.PRManagerApplication - Forwarding from an not-secured page to a secured page using an instantiated Page object is not allowed. Forwarding to https using original url. PageClasss:class com.conducive.ui.adminPages.credentials.AdminLogin URI:/longview_admin/wicket:interface/:2::INewBrowserWindowListener::
2010-05-07 13:58:47,649 ERROR [Hibernate Search: Directory writer-1] lucene.PerDPQueueProcessor - Unexpected error in Lucene Backend:
org.hibernate.search.SearchException: Unable to open IndexWriter
at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:159)
at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:103)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@/var/log/buzzstream/lucene/indexes/com.conducive.data.pojo.SocialNetwork/lucene-ebe7e3f30343c8bd7b4a1b1d83f7d044-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:85)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1550)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1079)
at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:152)
... 7 more

......................

this is a good clue; if this happened at startup then it just means you should delete the log; I assume that's not the case:

You're either affected by HSEARCH-515, which is a very good reason to switch to the final version, or you're having multiple applications sharing the same index.
When more than one application is writing concurrently to the same index, and the load is high, it might happen that one of them times out on lock aquisition; in this scenario we can't do much about it, I highly discourage such a setup.
If you're the only app using this index, or the only app writing to this index, you should really enable exclusive_index_use as described on the manual, it will prevent the need to aquire/release the lock at each index update and greatly boost performance.

We rolled back to the previous version of hibernate and hibernate search..

Is the exclusive_index_use a new concept? I'm not finding it in my classpath.

I have pasted our current versions below.

I have several daemons as well as an app that are all currently creating indexes. Even though the daemons are generating entities and thus generating indexes, I don't really require the indexes that would be generated by the daemons as they are leaf nodes and not part of the object graph that would actual be searched.

Based on the versions I'm currently using, is it possible to configure my daemons to NOT rewrite indexes but let the app keep writing them? Would this just be a matter of setting a system property for
public static final String INDEXING_STRATEGY = "hibernate.search.indexing_strategy";
If so, what would I set it to, it is undocumented. I tried to find usages of it, but it didn't have any.

Also, If I can do this, then I'm curious if I can do something similar to "exclusive_index_use" in the versions that I'm using:

<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>3.3.2.GA</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-annotations</artifactId>
<version>3.4.0.GA</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>3.4.0.GA</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-ehcache</artifactId>
<version>3.3.2.GA</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>3.3.2.GA</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-search</artifactId>
<version>3.1.1.GA</version>
</dependency>

it's a new configuration option, Search for it on the latest reference documentation


you should make sure that each index is written by only one Search instance to avoid contention timeouts. Remember that each entity can be configured to write to a different index, so in your described scenario it would be fine to enable that option if you can check for exclusive access (if you're sure that each daemon writes a different type of object)


not sure what your daemons do, but you could setup the master/slave approach as described on reference, so that you have only one master application which writes to the indexes. In version 3.2 we added a JGroups transport for master/slave, in addition to the JMS transport.


no

So how do I configure I configure the daemons to not write a search index?

Would this work:

<property name="hibernate.search.indexing_strategy" value="manual" />

Yes that's a good solution. You could have different apps use different users to enforce security at os level by using file access permissions.

What version of hibernate was the new release of hibernate search tested with?

Hibernate Search 3.2 requires Hibernate Core 3.5.0 or 3.5.1 (3.5.x)

Ok..

I'm currently use 3.1.1.GA Hibernate Search and I can't find any reference to hibernate.search.indexing_strategy when I search the source. Except the static variable on the Environment class. I cannot find any usage of the static variable. Where is this actually used?

I never could find it in the source, but it does seem to work.

Although it still writes to the index at startup, it doesn't write to it after that.

D/

cool.
FYI it's org.hibernate.search.impl.SearchFactoryImpl which reads this variable at initialization, but doesn't contain the string, it refers to Environment.INDEXING_STRATEGY

Yeah, I figured. I was searching on both the string and the variable name. I also did a find usage from eclipse and didn't find it. Very strange.