As my dataset grows >1 billion documents, I need to distribute indexing and searching of my lucene indexes. Currently I implement a sharding strategy, but all the processing is done within one jvm on one machine. Is there a way to spread the shards to different jvm's on different machines?


Thanks in advance.

I would definitely look at Infinispan (if i had the requirement for distributed search). I downloaded it a couple of weeks ago and started to have a play around with it and it very good. There is a Lucene directory provider that comes with the distribution so it would be worth having a look. I do believe there will be an integration with Hibernate Search and Infinispan (Sanne Grinovero is the best person who can comment on this).

Another option is the master slave configuration. This configuration allows a single master who is responsible for adding documents to the index while slave nodes have a read only copy of the index to perform searches. When the slave nodes update an object there is a message that is placed on a queue for the master process to index the document. Periodically the master index changes are pushed to the slave node. The online documentation or Hibernate Search in Action provides more indepth details.

Hope that helps.

hi amin-mc, thanks for the nice introduction.
True, we're going to create a new module for Hibernate Search to integrate the new distributed Directory, it's scheduled for beta3 of 3.2.0, which is the next iteration.

The Infinispan Directory is meant to cope with huge indexes which need easy and reliable distribution across several nodes, especially suited for interactive updates as Hibernate Search typically does. But in case you have many nodes wanting to write to the index at same time you still will have contention issues, the best design is to combine the master/slave setup with an Infinispan Directory, so I'd suggest you to configure the architecture for that design and then eventually switch the Directory when your dataset will demand it. Combine it all with sharding if possible to better distribute the load.