I'm trying to use hibernate search to index about 9000 objects with 9,000,000 nested objects.

I get this exception after 1 hour of processing:

[java] java.lang.OutOfMemoryError: Java heap space
[java] 01:09:15,128 ERROR AssertionFailure:22 - an assertion failure occured (this may indicate a bug in Hibernate)
[java] org.hibernate.annotations.common.AssertionFailure: Access a Sealed WorkQueue whcih has not been sealed
[java] at org.hibernate.search.backend.WorkQueue.getSealedQueue(WorkQueue.java:38)
[java] at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:120)
[java] at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.afterCompletion(PostTransactionWorkQu
eueSynchronization.java:50)
[java] at org.hibernate.transaction.JDBCTransaction.notifyLocalSynchsAfterTransactionCompletion(JDBCTransaction.java:243)

[java] at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:121)
[java] at com.broadlane.portal.contracts.ContractManager.indexContracts(Unknown Source)
[java] at com.broadlane.portal.contracts.ContractManager.main(Unknown Source)
[java] 01:09:15,128 ERROR JDBCTransaction:246 - exception calling user Synchronization
[java] org.hibernate.annotations.common.AssertionFailure: Access a Sealed WorkQueue whcih has not been sealed
[java] at org.hibernate.search.backend.WorkQueue.getSealedQueue(WorkQueue.java:38)
[java] at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:120)
[java] at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.afterCompletion(PostTransactionWorkQu
eueSynchronization.java:50)
[java] at org.hibernate.transaction.JDBCTransaction.notifyLocalSynchsAfterTransactionCompletion(JDBCTransaction.java:243)

[java] at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:121)
[java] at com.broadlane.portal.contracts.ContractManager.indexContracts(Unknown Source)
[java] at com.broadlane.portal.contracts.ContractManager.main(Unknown Source)
[java] Indexed 7034 contracts

I tried to increase the heapsize to 1MB, but the indexing fails a little bit latter.

Is there a recommended strategy to do the original indexing when dealing with millions of objects? Like break up the indexing in several pieces instead of doing everything in one query like this:

Session session = HibernateUtil.getSessionFactory().getCurrentSession();
FullTextSession fullTextSession = Search.createFullTextSession(session);
Transaction tx = fullTextSession.beginTransaction();
List contracts = session.createCriteria(Contract.class).list();
int i = 0;
for (Object contract : contracts) {
fullTextSession.index(contract);
i++;
}
tx.commit();

Thanks,
-Christophe

Yes there is
http://www.hibernate.org/hib_docs/search/reference/en/html_single/#search-batchindex
Use beta3 or above

Thanks Emmanuel.

I tried:

Session session = HibernateUtil.getSessionFactory().getCurrentSession();
FullTextSession ftSession = Search.createFullTextSession(session);
Transaction tx = ftSession.beginTransaction();
ScrollableResults result = ftSession.createCriteria(Contract.class).scroll( ScrollMode.FORWARD_ONLY );
int index = 0;
while (result.next()) {
index++;
ftSession.index(result.get(0));
if (index % batchSize == 0) session.clear();
}
tx.commit();

But for some reason if I don't comment out the session.clear() I get a JDBC error right away:

[java] 13:55:37,165 ERROR LazyInitializationException:19 - failed to lazily initialize a collection of role: com.test.Contract.programs, no session or session was closed
[java] org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: com.test.Contract.programs, no session or session was closed
[java] at org.hibernate.collection.AbstractPersistentCollection.throwLazyInitializationException(AbstractPersistentCollec
tion.java:358)
[java] at org.hibernate.collection.AbstractPersistentCollection.throwLazyInitializationExceptionIfNotConnected(AbstractPe
rsistentCollection.java:350)
[java] at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:343)
[java] at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86)
[java] at org.hibernate.collection.PersistentBag.iterator(PersistentBag.java:249)
[java] at org.hibernate.search.engine.DocumentBuilder.buildDocumentFields(DocumentBuilder.java:474)
[java] at org.hibernate.search.engine.DocumentBuilder.getDocument(DocumentBuilder.java:420)
[java] at org.hibernate.search.engine.DocumentBuilder.addWorkToQueue(DocumentBuilder.java:333)
[java] at org.hibernate.search.backend.impl.BatchedQueueingProcessor.prepareWorks(BatchedQueueingProcessor.java:121)
[java] at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.beforeCompletion(PostTransactionWorkQ
ueueSynchronization.java:44)

I think I read somewhere that scrollable resultset does not work with MS SQL server, could that be the reason?

Thanks,
-Christophe

I would say that, somehow, your batchSize is different than the one specified in your config file.

Silly me I was missing that in my hibernate.cfg:

<property name="hibernate.search.worker.batch_size">100</property>

It's indexing fine right now, we'll see how long it takes to index 9,000 objects with 9,000,000 nested objects...

Thanks Emmanuel.