-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 8 posts ] 
Author Message
 Post subject: Mass Indexer out of Memory Problem
PostPosted: Thu Aug 18, 2011 3:51 am 
Newbie

Joined: Thu Aug 18, 2011 3:32 am
Posts: 4
I am using MassIndexer for indexing about 27,000,000 data.But it throws out of memory exception.
if I limit indexer 4,000,000 data it works fine.I need some advice for the configuration

Thanks.


Code:
Exception in thread "Hibernate Search: identifierloader-1" java.lang.OutOfMemoryError: Java heap space


Code:
FullTextSession fts = this.getFullTextSession();
      try{
      MassIndexer msi= fts.createIndexer(PhoneBook.class);
         
         msi.batchSizeToLoadObjects( 50)
         .threadsForSubsequentFetching( 1 )
         .threadsToLoadObjects( 1 )
         .cacheMode(CacheMode.IGNORE)
         .startAndWait();
      }catch(InterruptedException intExc){
         intExc.printStackTrace();
      }


My current JVM configuration

Code:
-XX:MaxPermSize=128m
-XX:PermSize=128m

-Xmn100m
-Xms512m
-Xmx1024m


Top
 Profile  
 
 Post subject: Re: Mass Indexer out of Memory Problem
PostPosted: Thu Aug 18, 2011 8:56 am 
Hibernate Team
Hibernate Team

Joined: Thu Apr 05, 2007 5:52 am
Posts: 1689
Location: Sweden
-Xmx1024m - that seems awful little memory for an application indexing 27,000,000 million documents. Lucene and freetext search in general are resource (especially memory wise) intensive applications, especially if you scale to large data sets. Is 1 GB all you can give this process?


Top
 Profile  
 
 Post subject: Re: Mass Indexer out of Memory Problem
PostPosted: Thu Aug 18, 2011 2:02 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Also, is that MySQL ? It's driver will ignore our request for a proper scrollable result and will eagerly load the full list of identifiers in memory. I've reported this years back but they didn't fix it yet AFAIK.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Mass Indexer out of Memory Problem
PostPosted: Fri Aug 19, 2011 2:34 am 
Newbie

Joined: Thu Aug 18, 2011 3:32 am
Posts: 4
I can set -Xmx4096m but it is not enough.Is there another solution in the configuration for indexing large data set.


Top
 Profile  
 
 Post subject: Re: Mass Indexer out of Memory Problem
PostPosted: Fri Aug 19, 2011 3:06 am 
Newbie

Joined: Thu Aug 18, 2011 3:32 am
Posts: 4
s.grinovero wrote:
Also, is that MySQL ? It's driver will ignore our request for a proper scrollable result and will eagerly load the full list of identifiers in memory. I've reported this years back but they didn't fix it yet AFAIK.


we stared in mysql because of scrollable result bug we change to oracle.


Top
 Profile  
 
 Post subject: Re: Mass Indexer out of Memory Problem
PostPosted: Fri Aug 19, 2011 4:25 am 
Hibernate Team
Hibernate Team

Joined: Thu Apr 05, 2007 5:52 am
Posts: 1689
Location: Sweden
So even if you setting the heap size to 4 GB and switch to another database you are still running out of memory? Do you have any idea on why the application runs out of memory. Did you attach a profiler to see what objects are consuming all the space?
How much data (fields) are there per entity? Looking at the size of the documents you might be able to determine the minimum memory requirements for Lucene. Of course the problem could also be on the database (Hibernate) side, but this is something a profiler would tell you.
Other than that I think that the MassIndexer is the way to go. It tries to handle your resources in the most efficient way. Which version of Hibernate Search are you using? Are you using the latest version?

--Hardy


Top
 Profile  
 
 Post subject: Re: Mass Indexer out of Memory Problem
PostPosted: Fri Aug 19, 2011 7:33 am 
Newbie

Joined: Thu Aug 18, 2011 3:32 am
Posts: 4
Quote:
Class Name - Allocated Objects Bytes Allocated Bytes Allocated Objects Allocated
char[] 39.94% 1587999016 38130259
java.lang.String 30.68% 1219811104 38119097
com.etcbase.cas.model.phonebook.PhoneBook 11.04% 438739600 4218650
java.lang.Long 5.86% 233131992 9713833
java.util.HashMap$Entry 3.45% 137127872 4285246
org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer 2.29% 91122768 1265594
int[] 2.27% 90140104 67678
java.util.HashMap$Entry[] 0.92% 36772520 38601
com.etcbase.cas.model.lookup.PhoneBookType_$$_javassist_44 0.59% 23624384 421864
com.etcbase.sfc.model.lookup.ContactType_$$_javassist_101 0.59% 23624384 421864
com.etcbase.sfc.model.lookup.ValidityStatus_$$_javassist_105 0.59% 23624384 421864
byte[] 0.32% 12792760 34145
org.objectweb.asm.Item 0.16% 6351912 113427




version
<hibernate.search.version>3.4.0.Final</hibernate.search.version>
<lucene.version>3.3.0</lucene.version>

The table size which we want to index :5.6 GB

The fields for indexing are : (firstName,lastName,cityName,districtName)
Code:
package com.etcbase.cas.model.phonebook;

import java.util.Date;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.FetchType;
import javax.persistence.ManyToOne;
import javax.persistence.Temporal;
import javax.persistence.TemporalType;
import javax.persistence.Transient;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlTransient;

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.hibernate.search.annotations.Analyzer;
import org.hibernate.search.annotations.Field;
import org.hibernate.search.annotations.FieldBridge;
import org.hibernate.search.annotations.Index;
import org.hibernate.search.annotations.Indexed;

import com.etcbase.cas.model.lookup.PhoneBookType;
import com.etcbase.cas.util.LookupBridge;
import com.etcbase.sfc.model.common.SfcEntity;
import com.etcbase.sfc.model.lookup.ContactType;
import com.etcbase.sfc.model.lookup.ValidityStatus;

@Entity
@Indexed(index = "phoneBook")
@Analyzer(impl = StandardAnalyzer.class)
@XmlRootElement()
public class PhoneBook extends SfcEntity {

   private static final long serialVersionUID = 2673638618514022474L;

   private PhoneBookType phoneBookType;
   private ContactType contactType;
   private String phoneNumberText;
   private String firstName;
   private String lastName;
   private String cityName;
   private String originalCityName;
   private String districtName;
   private String originalDistrictName;
   private String addressText;
   private ValidityStatus validityStatus;
   private PhoneBookQueue createdByPhoneBookQueue;
   private PhoneBookQueue lastModifiedByPhoneBookQueue;
   private PhoneBookQuery createByPhoneBookQuery;
   private PhoneBookQuery lastModifiedByPhoneBookQuery;
   private Date lastModifiedDateTime;

   @ManyToOne(fetch = FetchType.LAZY)
   @XmlTransient
   public PhoneBookType getPhoneBookType() {
      return phoneBookType;
   }

   @ManyToOne(fetch = FetchType.LAZY)
   @XmlTransient
   public ContactType getContactType() {
      return contactType;
   }

   @Field(name = "phoneNumberText")
   @Column(length = 20)
   public String getPhoneNumberText() {
      return phoneNumberText;
   }

   @Column(length = 80)
   @Field(name = "firstName")
   public String getFirstName() {
      return firstName;
   }

   @Field(name = "lastName")
   public String getLastName() {
      return lastName;
   }

   @Column(length = 20)
   @Field(name = "cityName")
   public String getCityName() {
      return cityName;
   }

   @Column(length = 40)
   @Field(name = "districtName")a
   public String getDistrictName() {
      return districtName;
   }

   @XmlTransient
   public String getAddressText() {
      return addressText;
   }

   @ManyToOne(fetch = FetchType.LAZY)
   @FieldBridge(impl = LookupBridge.class)
   @Field(name = "validityStatus")
   @XmlTransient
   public ValidityStatus getValidityStatus() {
      return validityStatus;
   }

   @ManyToOne(fetch = FetchType.LAZY)
   @XmlTransient
   public PhoneBookQueue getCreatedByPhoneBookQueue() {
      return createdByPhoneBookQueue;
   }

   @ManyToOne(fetch = FetchType.LAZY)
   @XmlTransient
   public PhoneBookQueue getLastModifiedByPhoneBookQueue() {
      return lastModifiedByPhoneBookQueue;
   }

   @Transient
   public String getPhoneBookTypeEnumValue() {
      return this.getPhoneBookType() == null ? null : this.getPhoneBookType().getEnumValue();
   }

   @Transient
   public String getContactTypeEnumValue() {
      return this.getContactType() == null ? null : this.getContactType().getEnumValue();
   }

   @ManyToOne(fetch = FetchType.LAZY)
   @XmlTransient
   public PhoneBookQuery getCreateByPhoneBookQuery() {
      return createByPhoneBookQuery;
   }

   @ManyToOne(fetch = FetchType.LAZY)
   @XmlTransient
   public PhoneBookQuery getLastModifiedByPhoneBookQuery() {
      return lastModifiedByPhoneBookQuery;
   }

   @XmlTransient
   public String getOriginalCityName() {
      return originalCityName;
   }

   @XmlTransient
   public String getOriginalDistrictName() {
      return originalDistrictName;
   }

   @Temporal(TemporalType.TIMESTAMP)
   public Date getLastModifiedDateTime() {
      return lastModifiedDateTime;
   }

   @Override
   @Transient
   @XmlTransient
   public String getUid() {
      return uid;
   }

   @Override
   @Transient
   @XmlTransient
   public Long getId() {
      return super.getId();
   }
   
   
   public void setPhoneBookType(PhoneBookType phoneBookType) {
      this.phoneBookType = phoneBookType;
   }

   public void setContactType(ContactType contactType) {
      this.contactType = contactType;
   }

   public void setPhoneNumberText(String phoneNumberText) {
      this.phoneNumberText = phoneNumberText;
   }

   public void setFirstName(String firstName) {
      this.firstName = firstName;
   }

   public void setLastName(String lastName) {
      this.lastName = lastName;
   }

   public void setCityName(String cityName) {
      this.cityName = cityName;
   }

   public void setDistrictName(String districtName) {
      this.districtName = districtName;
   }

   public void setAddressText(String addressText) {
      this.addressText = addressText;
   }

   public void setValidityStatus(ValidityStatus validityStatus) {
      this.validityStatus = validityStatus;
   }

   public void setCreatedByPhoneBookQueue(PhoneBookQueue createdByPhoneBookQueue) {
      this.createdByPhoneBookQueue = createdByPhoneBookQueue;
   }

   public void setLastModifiedByPhoneBookQueue(PhoneBookQueue lastModifiedByPhoneBookQueue) {
      this.lastModifiedByPhoneBookQueue = lastModifiedByPhoneBookQueue;
   }

   public void setOriginalCityName(String originalCityName) {
      this.originalCityName = originalCityName;
   }

   public void setOriginalDistrictName(String originalDistrictName) {
      this.originalDistrictName = originalDistrictName;
   }

   public void setCreateByPhoneBookQuery(PhoneBookQuery createByPhoneBookQuery) {
      this.createByPhoneBookQuery = createByPhoneBookQuery;
   }

   public void setLastModifiedByPhoneBookQuery(PhoneBookQuery lastModifiedByPhoneBookQuery) {
      this.lastModifiedByPhoneBookQuery = lastModifiedByPhoneBookQuery;
   }

   public void setLastModifiedDateTime(Date lastModifiedDateTime) {
      this.lastModifiedDateTime = lastModifiedDateTime;
   }
}


Top
 Profile  
 
 Post subject: Re: Mass Indexer out of Memory Problem
PostPosted: Mon Aug 22, 2011 5:30 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hi,
it looks like you're experiencing a memory leak. You're not using a second level cache right?

BTW, unrelated with the issue but all ValidityStatus, PhoneBookType, ContactType look like perfect candidates to enable caching on them.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 8 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.