Hello,
We are using Hibernate Search on WebShphere 8.5 and index is in directory. We are intending to use IBM extreme scale for storing index in memory.

What are the necessary changes we will need to incorporate in our application for using index from memory data grid ? Has any used IBM extreme scale for index caching and clustering ?

Thanks
Pradeep

Hello,
interesting, I had to code a very similar support for Infinispan (aka JBoss Data Grid) which I think should be very similar to Extreme Scale.
I should warn you that while it initially seemed an easy task as you just have to "store files", the locking specifics in a highly concurrent environment made it much more complex than what I initially expected.. especially if you want it to be efficient you need to be rather expert in the workings of the storage.

I'm not aware of other Hibernate Search users to have attempted this on IBM extreme scale; in addition to Infinispan I heard it works on Voldemort and on Ehcache, all three using a similar design.

To plug in a Directory implementation in Hibernate Search, that's trivial you just need to implement a DirectoryProvider. You can find an example in the /infinispan module in the source code :

https://github.com/hibernate/hibernate-search

Specifically, this is the main code:
https://github.com/hibernate/hibernate-search/blob/master/infinispan/src/main/java/org/hibernate/search/infinispan/spi/InfinispanDirectoryProvider.java

Most of the code deals with A) initialization of the index B) parsing configuration tuning parameters to pass them to the Infinispan Directory - a complexity which you might not need in a POC, so this part should be very simple.

The more complex part is to actually implement the Lucene Directory.
The one we created for Infinispan was contributed into the Infinispan project (as it's open source), and you can find it in here:

https://github.com/infinispan/infinispan

Specifically, in this Maven module:
https://github.com/infinispan/infinispan/tree/master/lucene/lucene-directory

If you're willing to share your experiments and contribute them to Hibernate Search, I'll be happy to help. It would be nice to support IBM's stuff as well.
Feel free to ask for more details, but I'm not familiar with extreme scale so I can only describe what would be required from the point of view of Hibernate Search and/or Apache Lucene.

Sanne,
Thanks you very much responding. We are also new the extreme scale and can share what we learned.

There is some lucene directory implementation I found on internet. Is this good starting point ?

http://www.nofluffjuststuff.com/blog/billy_newport/2010/07/apache_lucene_store_plugin_for_ibm_websphere_extreme_scale

I only had a quick look so I don't know how much work it would need (I understand it's just an example so it might need some testing) but yes that looks like an excellent starting point.

Bear in mind that it's also a bit outdated, you'll have to update it to the Lucene 4 API as a first step, if you want to use it with the latest Hibernate Search 5 (which I highly recommend).