Hello

Hibernate Search 3.4.0.Final
Hibernate 3.6.3

I have around 8 lacs of records to be indexed. MassIndexer does work due to data discrepancies.
And I have following code to run.


private Integer indexMetadataList(ScrollableResults metadataList, FullTextSession fullTextSession,
Integer totalIndicesCreated) {
while (metadataList.next()) {
try{
fullTextSession.index(metadataList.get(0));
totalIndicesCreated++;
if(totalIndicesCreated % 5 == 0){
fullTextSession.flushToIndexes();
fullTextSession.clear();
}
} catch(ObjectNotFoundException ex){
// ignore ObjectNotFoundException. This happens only when there are no associated rows.
}

}

return totalIndicesCreated;
}

private ScrollableResults getAppsBatched(Session session, Integer limit, int offset) {

log.info("Retrieving data from {} to {}" , offset, offset + appsChunkSize);

ScrollableResults scrollbleResults = session.createQuery("Select m from Metadata m, Apps a where m.apps.appid = a.appid")
.setMaxResults(limit).setFetchSize(limit).setFirstResult(offset).setCacheable(false)
.scroll(ScrollMode.FORWARD_ONLY);

log.info("Retrieved data from {} to {}" , offset, offset + appsChunkSize);

return scrollbleResults;
}


private void indexApps() throws Exception{
Session session = null;
FullTextSession fullTextSession = null;
int totalIndicesCreated = 0;

try {
//42250;
ScrollableResults metadataList = null;

session = sessionFactory.getCurrentSession();
fullTextSession = Search.getFullTextSession(session);
fullTextSession.setFlushMode(FlushMode.MANUAL);
fullTextSession.setCacheMode(CacheMode.IGNORE);

do {

try {

metadataList = getAppsBatched(session, appsChunkSize, offset);

log.info("Creating indices from {} to {}", offset, offset + appsChunkSize);
totalIndicesCreated = indexMetadataList(metadataList, fullTextSession, totalIndicesCreated);

log.info("Indices created from {} to {}", offset, offset + appsChunkSize);
session.evict(Metadata.class);

} catch(Exception ex){
log.error("Exception creating indices...", ex);
}



} while(offset < maxLimit ); // Use this for testing



} catch (Exception e) {
throw e;
}
// MassIndexer fails because of stale data where referenced row does not exist for appid in apps_readonly table.
/*try{
session = sessionFactory.openSession();
fullTextSession = Search.getFullTextSession(session);
fullTextSession.createIndexer(Metadata.class)
.batchSizeToLoadObjects( 5 )
.threadsForSubsequentFetching( 2 )
.threadsToLoadObjects( 250 )
.threadsForIndexWriter( 5 )
.cacheMode(CacheMode.NORMAL) // defaults to CacheMode.IGNORE
.startAndWait();


}*/

log.info("Total indices created count {}", totalIndicesCreated);
}

Please help me out.

Hi, welcome.
some hints:

1) Which database is it? MySQL's driver ignores the request for scrollableresults, it basically fetches the whole data in a single query. Yes that's extremely stupid, as it might not fit in memory, but you can experiment with some driver parameters to "fix" that.

2) In the catch block instead of ignoring the error you should make sure you close the Session and replace it with a new one: Hibernate's Sessions should be discarded after an error.

3) Can you try scrolling on the data but commenting out the

So we can undersand if you're going in OOM because of the indexing or because of the data loading from the database

4) I'd strongly advise to fix the data on the database, so you can use the MassIndexer. You should be able to do that easily with some SQL statements? like to remove all references to invalid foreign keys.

5) Use the formatting options of the forum, so we can read your code better ;)