Using LuceneStoreConfigurationBuilder from Infinispan 6.0

jhb · **Joined:** Thu Oct 31, 2013 10:04 am **Posts:** 5

Hi,

We have a server application using Lucene 4.5 and currently relies on shared storage or rsync to achieve primary/backup HA. Note that for now, only one instance will be writer at a time.

While checking on Infinspan 6.0 I found out that it is now providing a LuceneStoreConfigurationBuilder and I expect it to persist Lucene data files to local FS in a better way than SingleFileStoreConfigurationBuilder. I'm trying to mix info I find in online doc, JavaDoc, xml files and unit tests I find but did not manage to find a working configuration.

On top of Infinispan directory I'm also using Lucene TrackingIndexWriter, SearcherManager and ControlledRealTimeReopenThread but doubt it relates to my problems with LuceneStrore.

When I start primary node, a Infinispan-IndexStore dir is created but always stays empty, also Infinispan-SingleFileStore contains index.<name>.meta.dat. Rebuild of index starts pulling data from DB but nothing is ever written to IndexStore.

Once I start backup node, index.<name>.meta.dat is replicated but node then fails with file not found because nothing was replicated to IndexStore. See stack trace at the end of this post.

Switching to using SingleFileStore for data is a bit better as data is created to backup node, but I expect performance to be quite low as our indexes can reach 20GB or more.

Can someone share a working cluster configuration with persistent storage and eviction (to save RAM) for Infinispan 6.0 ?

Thank you!

--
See sample source below for my configuration. Note that I don't have any customization of jgroups, I let defaults from what is bundled with infinispan distribution.

Code:

    
    public void setupGlobalConfiguration(String clusterName) {
        GlobalConfigurationChildBuilder cfg = GlobalConfigurationBuilder.defaultClusteredBuilder().clusteredDefault();
        cfg.globalJmxStatistics()
                .enable()
                .cacheManagerName("Lucene")
                .allowDuplicateDomains(Boolean.TRUE)
            .transport()
                .clusterName(clusterName)
                .distributedSyncTimeout(50000)
            .shutdown()
                .hookBehavior(ShutdownHookBehavior.DONT_REGISTER);

        manager = new DefaultCacheManager(cfg.build());
    }
    
    public Directory getDirectory(String indexName) {
        //Metadata configuration
        ConfigurationBuilder 
        configuration = getBaseConfig();
        configuration
            .persistence()
                .passivation(false)
                .addStore(SingleFileStoreConfigurationBuilder.class)
                .shared(true)
                .fetchPersistentState(true)
            .clustering()
                .cacheMode(CacheMode.REPL_SYNC)
                .sync().replTimeout(25000)
                .stateTransfer()
                    .fetchInMemoryState(true)
                    .timeout(30000)
        ;

        manager.defineConfiguration(indexName + ".meta", configuration.build());
        Cache<?, ?> metadataCache = manager.getCache(indexName + ".meta");
        
        //Data configuration
        configuration = getBaseConfig();
        configuration
            .eviction()
                .maxEntries(-1)
                .strategy(EvictionStrategy.NONE)
            .expiration()
                .lifespan(-1)
            .persistence()
                .passivation(false)
                .addStore(LuceneStoreConfigurationBuilder.class)
                .autoChunkSize(110)
                .shared(true)
                .fetchPersistentState(true)
            .clustering()
                .cacheMode(CacheMode.REPL_SYNC)
                .sync().replTimeout(25000)
            .stateTransfer()
                .fetchInMemoryState(true)
                .timeout(30000)
        ;

        manager.defineConfiguration(indexName + ".data", configuration.build());
        Cache<?, ?> chunksCache = manager.getCache(indexName + ".data");

        //Lock configuration
        configuration = getBaseConfig();
        configuration
            .clustering()
                .cacheMode(CacheMode.REPL_SYNC)
                .sync().replTimeout(25000)
            .stateTransfer()
                .fetchInMemoryState(true)
                .timeout(30000)
        ;
        
        manager.defineConfiguration(indexName + ".locks", configuration.build());
        Cache<?, ?> distLocksCache = manager.getCache(indexName + ".locks");
        
        Directory directory = DirectoryBuilder
                .newDirectoryInstance(metadataCache, chunksCache, distLocksCache, indexName)
                .create();

        return directory;
    }

    private ConfigurationBuilder getBaseConfig() {
        ConfigurationBuilder configuration = new ConfigurationBuilder();
        configuration
            .locking()
                .lockAcquisitionTimeout(20000)
                .writeSkewCheck(false)
                .concurrencyLevel(500)
                .useLockStriping(false)
            .transaction()
                .transactionMode(TransactionMode.NON_TRANSACTIONAL)
                .invocationBatching().disable()
            .clustering()
                .sync()
                    .replTimeout(20000)
            .clustering()
                .cacheMode(CacheMode.REPL_SYNC)
                .stateTransfer()
                    .fetchInMemoryState(true)
                    .timeout(480000)

            .eviction()
                .maxEntries(-1)
                .strategy(EvictionStrategy.NONE)
            .expiration()
                .maxIdle(-1)
             ;
        
        return configuration;
    }

Code:

...
Caused by: java.io.FileNotFoundException: Q:\build\dist\server\Infinispan-IndexStore\index.contact\_50.si (The system cannot find the path specified)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
   at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:193)
   at org.infinispan.lucene.cachestore.DirectoryV4Adaptor.openInput(DirectoryV4Adaptor.java:43)
   at org.infinispan.lucene.cachestore.DirectoryLoaderAdaptor.loadIntern(DirectoryLoaderAdaptor.java:245)
   at org.infinispan.lucene.cachestore.DirectoryLoaderAdaptor.access$300(DirectoryLoaderAdaptor.java:32)
   at org.infinispan.lucene.cachestore.DirectoryLoaderAdaptor$LoadVisitor.visit(DirectoryLoaderAdaptor.java:297)
   at org.infinispan.lucene.ChunkCacheKey.accept(ChunkCacheKey.java:71)
   at org.infinispan.lucene.cachestore.DirectoryLoaderAdaptor.load(DirectoryLoaderAdaptor.java:186)
   ... 53 more

sanne.grinovero · **Posted:** Thu Oct 31, 2013 2:18 pm

Hi,
that question belongs to the Infinispan forums:
https://community.jboss.org/en/infinispan?view=discussions

But I happen to know about it as I've implemented the LuceneLoader myself as a contribution to the Infinispan project, as I needed it as a way to efficiently import large indexes in Infinispan;
the problem is that it's a CacheLoader not a CacheStore; in other words, it currently only supports loading.

Looks like someone after me contributed the configuration builder but has chosen a misleading name, sorry about that, I'll ping the Infinispan team about this.

If you are interested to contribute storage capabilities for it, it shouldn't be too complex and I'm more than happy to provide some guidance; I would like to do it but couldn't find the time yet.

jhb · **Joined:** Thu Oct 31, 2013 10:04 am **Posts:** 5

Hi Sanne,

Sorry for posting on the wrong site! It seems I'm really lucky that you happen to be the one who created this loader.

Now it's clear why it's not working. So SingleFileStore is the only reasonable way of persisting a Lucene index ?

I'll take a look at a "multiple file store" implementation but I doubt I'll have time to seriously work on it.

Thanks a lot for your help!

sanne.grinovero · **Posted:** Thu Oct 31, 2013 5:59 pm

Any of the Infinispan CacheStores should work well for this purpose; for example a good candidate is the one based on Cassandra, as it's very efficient with writes (not so much at reading but for that you have the in-memory cache).
Yes I the new SingleFileStore is probably a good choice too.

I might be working on the storage support sometimes soon as I would like to make that too, but can't promise on the times. Indeed you where lucky I just happened to know about this question :) But I also read the Infinispan forums.

BTW such advanced usage questions are often better discussed on IRC, you'll find most of the developers on Freenode, channel #infinispan

jhb · **Joined:** Thu Oct 31, 2013 10:04 am **Posts:** 5

Cassandra will be a good option for the future for us but right now we are still storing data in SQL DB, so deployment would be too complex for our customers.

Do you think I should move the discussion to Infinispan forum so others can profit from the discussion ?

Good to know about the IRC channel, I'll join if I make anything useful out of Lucene storage or have questions.

Thanks!

sanne.grinovero · **Posted:** Fri Nov 01, 2013 5:17 am

Quote:

Do you think I should move the discussion to Infinispan forum so others can profit from the discussion ?

Quote:

Cassandra will be a good option for the future for us but right now we are still storing data in SQL DB, so deployment would be too complex for our customers.

You might want to use the JDBCCacheStore then, probably better suited than the filesystem based ones.

Quote:

Good to know about the IRC channel, I'll join if I make anything useful out of Lucene storage or have questions.

Sure, feel free to ping me there. I'm travelling today and weekend, but normally I'm connected.

sanne.grinovero · **Posted:** Fri Nov 01, 2013 5:19 am

Actually I have a question for you.
How come you thought to post on this forums for this question? I'm wondering if we have some misleading message / doc somewhere, if something needs to be clarified. Possibly some stale wiki page somewhere?

jhb · **Joined:** Thu Oct 31, 2013 10:04 am **Posts:** 5

sanne.grinovero wrote:

Quote:

Cassandra will be a good option for the future for us but right now we are still storing data in SQL DB, so deployment would be too complex for our customers.

You might want to use the JDBCCacheStore then, probably better suited than the filesystem based ones.

According to documentation I thought JDBCCacheStore was not expected to have good performance. We have a mostly write intensive load with docs from 0.5KB to 500KB and currently have speeds about 2K docs/s, should I expect similar performance from JDBCCacheStore ?

sanne.grinovero wrote:

Actually I have a question for you.
How come you thought to post on this forums for this question? I'm wondering if we have some misleading message / doc somewhere, if something needs to be clarified. Possibly some stale wiki page somewhere?

No it's my fault :) I was trying to replicate the same cache configuration as what Hibernate Search is using and found a post in this forum about some custom Infinispan configuration of Hibernate storage. I had many tabs open and thought I was on Infinispan forums !

sanne.grinovero · **Posted:** Fri Nov 01, 2013 11:39 am

Quote:

According to documentation I thought JDBCCacheStore was not expected to have good performance. We have a mostly write intensive load with docs from 0.5KB to 500KB and currently have speeds about 2K docs/s, should I expect similar performance from JDBCCacheStore ?

You're probably right, the JDBC one isn't very fast and I also doubt it would be able to match your speed requirements. Let's continue on the Infinispan forums.

jhb · **Joined:** Thu Oct 31, 2013 10:04 am **Posts:** 5

Hi Sanne,

I posted a reply on : https://community.jboss.org/message/845674

Regards,