Hi folks
I'm trying to index an entity with 5M rows in development server.
I was receiving a java heap space exception.
I cannot load the app giving it more than 1024m to xmx or xms attributes.

so, I was thinking indexing by steps.
Can I do the indexing issue in 10 steps of 200K rows (modifying the criteria with limit and offset)
Can I try this way or im losing something?

thanks in advance.

I use hibernate 3.2.6, hibernate search 3.0.1

I have been using batch_size = 200
and using the "especially efficient way to index" that is exposed in chapter 6 of the reference.

Anyway it expcepts heap size before indexing. It stop in the while querying the list();
thank u.

Hi,
having "batch_size = 200" is too high in my opinion, but should not be relevant to the memory problem unless you are loading huge data graphs.

Are you sure you are clearing the session periodically in your indexing code?
Did you try enabling hibernate query logging, to verify what you are loading?
I'm indexing millions of entities without the need to change my xmx settings (using default low values), so I think this could be a mapping issue.

thanks for your answer, I have fixed the problem using scrolls properly.
now the problem is the time. The indexing batch is stimated in 23hours and a half.

I have tunned some options, but it doesnt improve too much.
Can I index partially today and the rest tomorrow?.

do I have to get the commit instruction to save the created indexes?
If i just stop the server while batch is running Would I lose the changes?
how can I study the Lucene? files they are all binary.

Of course, you control the indexing process; however you'll have to find a way to define were to stop and then restart at same position.
You could also leave it running, you can stil use an application while it's indexing.


When using the index() API the Lucene backend will flush some indexed documents periodically, so you don't have to wait for commit() to see something in the index, but without guarantees. If you need to be sure all entities you indexed are flushed to the index you have to commit. This shouldn't be a problem as you control the commit() yourself. In 3.1.0 there is a "flushToIndexes()" method; you'll have to upgrade to have more control.


Look for Luke in the FAQ http://www.hibernate.org/446.html .

thank u very much and sorry for my ignorance.
I just read the reference attachment the source code of 3.0.1
I will study hard before posting again.

no problem, you're welcome.