Hi,

Can any body has any numbers on the speed of the indexing process using HSearch ? I have the indexing process running on an IBM blade and my numbers are as below:

Records : 10 million+
Time taken : 18 days.

My indexable entity is relatively big, but the indexed fields are just 5. This bean has 8 associated objects and I am NOT indexing them.

Any help is highly appreciated.

[Update: http://www.hibernate.org/15.html,
Many people try to benchmark Hibernate. All public benchmarks we have seen so far had (and most still have) serious flaws.

it seems that, I've asked a pretty stupid question !!! ]

--Best Regards
Anil.[/b]

Hi,
I don't consider your question as stupid, having to reindex all data is a real world problem.
I've had a similar problem:

Records: 4 Million
Time taken: 3 months (estimate after 3 days)

Mine was probably slower than yours because a "single" record is actually a very complex entity, with a dozen of many-to-many objects going in the same index.

We solved this problem with some tweaking and recoding:
1)correct lazy/eager fields on the entities
2)caching for entities which where referred to very often (look-up tables and similar)
3)More threads: one reading PKs, 30 read the entities, 10 convert from entities to Lucene Documents, one writes to the index (and each step has a 1000 elements queue connecting to the next)
(The number of threads really depends on the machine, database, network performance and cache configuration)
4)Some custom-build StringBridges for the more complex fields
4)Tweaking the indexwriter parameters (Lucene 2.3) to merge lots of documents in the index at the same time (not one by one).

Time taken: 20 minutes.

I just finished updating my code to H.Search 3.0.1 and will share it as soon as it is "cleaned up" and testable; Unfortunately I'll have to do it in my hobby time, so my estimate is to release some alpha code in half march.

My current code uses the inner-API of Hibernate Search, the idea is to submit a patch, so I have to change some design.
I could use some help in benchmark and testing setup to verify it all, I'm studying the test s in Hibernate Search now.

Generally speaking when indexing millions of records (in the tens) takes half a day or more, something is very likely to be wrong and generally it's due to an improper fetching strategy (ie. how you read data from the database).

Disclaimer, time heavily depends on the complexity of the dataset.