-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 3 posts ] 
Author Message
 Post subject: Hibernate Search, Index creation memory leak
PostPosted: Fri Jun 12, 2009 6:49 am 
Newbie

Joined: Wed Jun 13, 2007 5:44 am
Posts: 16
Hi.

I'm trying to create a lucene index of about 3.4 million entities. No matter what I've tried, I still have some memory leaks. How should I optimize this index's creation?
I'm running Hibernate Search with Seam 2.1.1.GA.
My versions:
Hibernate 3.3.1.GA
Hibernate EntityManager 3.4.0.GA
Hibernate Search 3.1.1.GA

I'm running a Postgres database.

Here's the code I'm using

Code:
      int step = 5000;
      int indexed = 0;
      // Get the min and max Id
      long minId = ((Number) entityManager.createQuery("select min(id) from Book b").getSingleResult()).longValue();
      long maxId = ((Number) entityManager.createQuery("select max(id) from Book b").getSingleResult()).longValue();
      boolean fetch = true;
      FullTextSession fullTextSession = Search.getFullTextSession((Session) entityManager.getDelegate());
      while (fetch) {
         List<Book> books = entityManager.createQuery(
               "select b from Book b where id >=:minId and id <:maxId and role=:role").setParameter("minId", minId).setParameter("maxId", minId + step)
               .setParameter("role", Role.STOCK).getResultList();
         // Index every books
         for (Book book : books) {
            fullTextSession.index(book);
            indexed++;
         }
         books = null;

         fullTextSession.flushToIndexes();
         fullTextSession.clear();
         entityManager.clear();
         minId += step;

         // Every books were fetched
         if (minId > maxId) {
            fetch = false;
         }
      }


Top
 Profile  
 
 Post subject: Re: Hibernate Search, Index creation memory leak
PostPosted: Sat Jun 13, 2009 6:13 am 
Hibernate Team
Hibernate Team

Joined: Thu Apr 05, 2007 5:52 am
Posts: 1689
Location: Sweden
Hi,

what do you mean with memory leak? Does your indexing complete or do you get a OutOfMemoryException?

Regarding the actual code have a look here - http://docs.jboss.org/hibernate/stable/ ... batchindex . This is the recommended way of batch indexing where you basically use ScrollableResults and manual transaction management.

Depending on your heap size settings it might alsp help to change the JVM settings.

--Hardy


Top
 Profile  
 
 Post subject: Re: Hibernate Search, Index creation memory leak
PostPosted: Sat Sep 19, 2009 1:20 am 
Beginner
Beginner

Joined: Thu Jun 21, 2007 1:47 pm
Posts: 46
Nevermind, my worker replacement wasn't updated when I updated to search 3.1.1 ...


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 3 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.