You mean performance is good enough, right?
Running the standard Hibernate Search file-based DirectoryProvider, the JMS throughput is great. When we switch to InfinispanDirectoryProvider with the file store we're running into the throughput issues.
That's very unexpected as in all tests we did the Infinispan one is faster
I believe it was faster when we had used the FileCacheStore in Infinispan. Using a more reliable store like BDBJE or JDBM has been slower.
The readBytes in which class? could you describe the frames you see, or share some screenshots or JProfiler snapshot?
I don't have a snapshot, but I can easily make one. I can't remember where specifically the method was, I just remember it being in the Infinispan code. This is what lead us to increasing the chunk size and setting eviction to a higher maxEntries.
...generally a good idea to prevent chunking to happen by playing with low enough max_merge_docs, as an example.
I don't think we've messed with the max merge docs parameters yet.
What caught my attention is that you're using a JdbmCacheStore, this one is not well known for it's performance. Could you try a run without using any CacheStore? If we find that's the cause, you could then try different cachestores and configure them as async cachestores: you're still going to be fault tolerant in a clustered configuration as while you write on disk asynchronously, the values will be replicated in sync to other nodes.
We will try to run without a CacheStore at all and see what performance is like. The drawback is, however, our indexes are around 20GB so not having a Store behind them makes them not persistent. But for testing purposes, we're going to try and see if there is a difference. We have turned on async with 40 threads and saw a little bit of an increase in performance for a short burst, then it seemed to hang once those 40 threads got busy. May need more threads as we have 200+ listener threads in JMS.
Ultimately we want to upgrade to JBoss 7 but being stuck in JBoss EAP 5 has made going past Infinispan 4 very difficult. I tried for almost a day getting it working but ran into problems with JBoss log manager being a pain. That's another post but it is why we're at Infinispan 4 not 5.
We'll try different CacheStores. Which one do you recommend? Have you had any experience with in-memory or alike dbs with the JDBC CacheStore? Our servers have a lot of storage compared to our Oracle RAC instance so we'd have to stand up a standalone db for the JDBC. We tried Sleepycat and were very disappointed with the record locking issues and frequent corruption with synchronization.