Persistant Indexing with Infinispan Problems

TAUnderwood · **Joined:** Mon Jul 11, 2011 5:16 pm **Posts:** 12

Hi Everyone,

I'm having issues getting my index to be persistant when using Infinispan. When not trying to use persistance it works just fine however as soon as I start trying to use a FileCacheStore for persistance I start getting exceptions.

It works for about 2 hours when this exception begins to be thrown:

Code:

[2011-08-29 11:30:53,425] ERROR FileCacheStore.java:317 Hibernate Search: indexwriter-154 ) ISPN000063: Exception while saving bucket Bucket{entries={_4o.fdt|M|cnwk.foreman.model.SoftwareDownload=ImmortalCacheEntry{key=_4o.fdt|M|cnwk.foreman.model.SoftwareDownload, value=ImmortalCacheValue{value=FileMetadata{lastModified=1314642653425, size=32768}}}}, bucketId='1509281792'}
java.io.FileNotFoundException: /var/opt/fullTextStore/LuceneIndexesMetadata/1509281792 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
   at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.createChannel(FileCacheStore.java:494)
   at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.write(FileCacheStore.java:472)
   at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:315)
   at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)
   at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)
   at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)
   at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:195)
   at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:210)
   at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
   at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:82)
   at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
   at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
   at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
   at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:214)
   at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:162)
   at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
   at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:114)
   at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
   at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
   at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)
   at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
   at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
   at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77)
   at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
   at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
   at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274)
   at org.infinispan.CacheImpl.put(CacheImpl.java:515)
   at org.infinispan.CacheSupport.put(CacheSupport.java:51)
   at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:206)
   at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)
   at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)
   at org.apache.lucene.index.FieldsWriter.abort(FieldsWriter.java:121)
   at org.apache.lucene.index.StoredFieldsWriter.abort(StoredFieldsWriter.java:90)
   at org.apache.lucene.index.DocFieldProcessor.abort(DocFieldProcessor.java:71)
   at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:421)
   at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:729)
   at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2042)
   at org.hibernate.search.backend.impl.lucene.works.AddWorkDelegate.performWork(AddWorkDelegate.java:76)
   at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace.doWorkInSync(DirectoryProviderWorkspace.java:96)
   at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace$AsyncIndexRunnable.run(DirectoryProviderWorkspace.java:144)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:680)

I know that the error says I am opening too many files for my OS but I have already doubled the limit and it's still giving me the error.

Here is my configuration:

hibernate.cfg.xml:

Code:

 <property name="hibernate.search.default.directory_provider">infinispan</property>
        <property name="hibernate.search.infinispan.configuration_resourcename">infinispan.cfg.xml</property>

infinispan.cfg.xml

Code:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
        xmlns="urn:infinispan:config:4.2">

    <!-- *************************** -->
    <!-- System-wide global settings -->
    <!-- *************************** -->

    <global>

        <!-- Duplicate domains are allowed so that multiple deployments with default configuration
            of Hibernate Search applications work - if possible it would be better to use JNDI to share
            the CacheManager across applications -->
        <globalJmxStatistics
                enabled="true"
                cacheManagerName="HibernateSearch"
                allowDuplicateDomains="true"/>

        <!-- If the transport is omitted, there is no way to create distributed or clustered
            caches. There is no added cost to defining a transport but not creating a cache that uses one,
            since the transport is created and initialized lazily. -->
        <transport
                clusterName="HibernateSearch-Infinispan-cluster"
                distributedSyncTimeout="50000">
            <!-- Note that the JGroups transport uses sensible defaults if no configuration
                property is defined. See the JGroupsTransport javadocs for more flags -->
        </transport>

        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
            Hibernate Search takes care to stop the CacheManager so registering is not needed -->
        <shutdown
                hookBehavior="DONT_REGISTER"/>

    </global>

    <!-- *************************** -->
    <!-- Default "template" settings -->
    <!-- *************************** -->

    <default>

        <locking
                lockAcquisitionTimeout="20000"
                writeSkewCheck="false"
                concurrencyLevel="500"
                useLockStriping="false"/>

        <lazyDeserialization
                enabled="false"/>

        <!-- Invocation batching is required for use with the Lucene Directory -->
        <invocationBatching
                enabled="true"/>

        <!-- This element specifies that the cache is clustered. modes supported: distribution
            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
            Lucene indexes, but make sure you have enough memory to store the index in your heap.
            Also distribution scales much better than replication on high number of nodes in the cluster. -->
        <clustering
                mode="replication">

            <!-- Prefer loading all data at startup than later -->
            <stateRetrieval
                    timeout="60000"
                    logFlushTimeout="30000"
                    fetchInMemoryState="true"
                    alwaysProvideInMemoryState="true"/>

            <!-- Network calls are synchronous by default -->
            <sync
                    replTimeout="20000"/>
        </clustering>

        <jmxStatistics
                enabled="true"/>

        <eviction
                maxEntries="-1"
                strategy="NONE"/>

        <expiration
                maxIdle="-1"/>

    </default>

    <!-- ******************************************************************************* -->
    <!-- Individually configured "named" caches.                                         -->
    <!--                                                                                 -->
    <!-- While default configuration happens to be fine with similar settings across the -->
    <!-- three caches, they should generally be different in a production environment.   -->
    <!--                                                                                 -->
    <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->
    <!-- should be enabled, and maybe distribution is desired.                           -->
    <!-- ******************************************************************************* -->

    <!-- *************************************** -->
    <!--  Cache to store Lucene's file metadata  -->
    <!-- *************************************** -->
    <namedCache name="LuceneIndexesMetadata">

        <clustering mode="replication">
            <stateRetrieval
                    fetchInMemoryState="true"
                    logFlushTimeout="30000"/>
            <sync replTimeout="25000"/>
        </clustering>
        <loaders preload="true">
            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">
                <properties>
                    <property name="location" value="/var/opt/fullTextStore"/>
                </properties>
            </loader>
        </loaders>
    </namedCache>

    <!-- **************************** -->
    <!--  Cache to store Lucene data  -->
    <!-- **************************** -->
    <namedCache name="LuceneIndexesData">

        <clustering mode="replication">
            <stateRetrieval
                    fetchInMemoryState="true"
                    logFlushTimeout="30000"/>
            <sync
                    replTimeout="25000"/>
        </clustering>
        <loaders>
            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">
                <properties>
                    <property name="location" value="/var/opt/fullTextStore"/>
                </properties>
            </loader>
        </loaders>
    </namedCache>

    <!-- ***************************** -->
    <!--  Cache to store Lucene locks  -->
    <!-- ***************************** -->
    <namedCache
            name="LuceneIndexesLocking">
        <clustering
                mode="replication">
            <stateRetrieval
                    fetchInMemoryState="true"
                    logFlushTimeout="30000"/>
            <sync
                    replTimeout="25000"/>
        </clustering>
    </namedCache>

</infinispan>

and lastly here is my indexing code:

Code:

ftSession.createIndexer().batchSizeToLoadObjects(25).cacheMode(CacheMode.NORMAL).threadsToLoadObjects(5).threadsForIndexWriter(3).startAndWait();

I am using Hibernate Search 4.4, Lucene: 3.1.0, Infinispan 5.0.0 and jGroups: 2.12.1.3.

I'm new to all of this so any help would be great.

sanne.grinovero · **Posted:** Mon Aug 29, 2011 4:30 pm

Hi,

Quote:

I am using Hibernate Search 4.4, Lucene: 3.1.0, Infinispan 5.0.0 and jGroups: 2.12.1.3.

Hibernate Search 4.4 doesn't exist.. do you mean 4.0.0.Alpha1 ?
If it's 3.4, then are you sure it's Infinispan 5.0 ? Is it 5.0.0.FINAL ?
Just asking as we introduced support (and tests) for Infinispan 5 only since Hibernate Search version 4.0x (officially, but I've seen others reporting it works fine).

Could you dump a list of open files, and see if the problem is related to Infinispan's CacheLoader, rather than another component leaking file handles ?

TAUnderwood · **Joined:** Mon Jul 11, 2011 5:16 pm **Posts:** 12

Sorry you are correct. I meant 3.4.0.Final for Hibernate Search and 5.0.0.Final for Infinispan.

As far as the open file dump goes I have one I am more than willing to provide however it is extremely large (at least 1000 files are listed mostly from the java process) what is the best way to provide it? I am not sure on the conventions of the forums and since it is so large I didnt want to straight throw it in to my post.

Thanks for your help!

sanne.grinovero · **Posted:** Mon Aug 29, 2011 5:37 pm

maybe you could extract from statistics from the file? I don't need to read it all especially if it's so big, we only want to know where the majority of your file handles are gone.
This is likely an Infinispan issue, so to share your file you could open an issue on the Infinispan issue tracker and attach the file there.

TAUnderwood · **Joined:** Mon Jul 11, 2011 5:16 pm **Posts:** 12

Thanks Sanne,

I am going to guesstimate some statistics but it is a very large margin so I hope you can see the pattern:

80% open files belong to java
5% belong to Google (chrome is my default browser)
5% belong to intelliJ
and 10% belong to various system tools (finder, etc)

Of the files opened by java
2-3% are various .jar files and other java files
95% belong to the cachestore

I'll see about opening a file on the issue tracker.

If you have any additional thoughts I would greatly appreciate it!

Thanks!

sanne.grinovero · **Posted:** Tue Aug 30, 2011 3:58 am

thank you, yes please open an issue, these things need to be tracked.
https://issues.jboss.org/browse/ISPN

Last questions, what's the absolute number of open file handles belonging to the cachestore, and how many files do you see in the cachestore?

I need to understand if the amount of open files is too high for your settings - which might need an improvement anyway - or if it's a crazy amount out of control, meaning we have to find a resources leak.
The important information would be to know if there are more open file handles than there are files.

TAUnderwood · **Joined:** Mon Jul 11, 2011 5:16 pm **Posts:** 12

Thanks again Sanne,

I have opened the issue at:https://issues.jboss.org/browse/ISPN-1362

As far as the numbers you requested there are 10160 files that are listed as open in the cache store and a total of 10178 files in the cache store.

Let me know if you need any additional information.

sanne.grinovero · **Posted:** Tue Aug 30, 2011 1:53 pm

thank you very much, I just adjusted the title slightly.
It seems to me the issue is either a) it creates too many files b)it should not try to keep them all open.
At least it's not leaking file handles.

As a workaround until that's fixed, I'd suggest you to try another CacheLoader. I've personally used the JDBC implementation a lot, it's quite handy to store the index in the same database as your entities as you can do a single backup taking both (and one less data store to manage).

TAUnderwood · **Joined:** Mon Jul 11, 2011 5:16 pm **Posts:** 12

Thanks Sanne,

I'll try the JDBC cacheloader.