Hibernate Community • View topic

View unanswered posts | View active topics

Board index » Hibernate & Java Persistence » Hibernate Users

All times are UTC - 5 hours [ DST ]

Performance issues

Page 1 of 1

[ 4 posts ]

Previous topic | Next topic

Author

Message

webguy

Post subject: Performance issues

Posted: Wed Jan 21, 2009 5:57 am

Newbie

Joined: Thu Feb 07, 2008 8:30 am
Posts: 7

Hi,

Hibernate version: 3.2.5.ga

Mysql 5

I'm having some performance issues with hibernate. My project is based on appfuse-light with struts2-spring-hibernate. I have an import file called import.xml that among other things contains data of regions from which you can see below.

There are about 16000 of these elements. When starting the import it goes really fast. After a few seconds I have about 1000 of them stored in the database. But as the import goes on it gets slower and slower. In the end it only imports about 10 objects per second and in total it takes about 30 minutes to import all of these objects. I'm not sure why this is. Does anyone have any ideas? Below i provide some code snippets. If you need any more information just ask for it and I will try to provide it.

Any suggestions on how I can speed things up are most appreciated.

This is what the import data looks like:

Code:

<COUNTRIES>
<COUNTRY>
  <CO_RECNO>6</CO_RECNO>
  <CO_CODE>AD</CO_CODE>
  <CO_DESC>ANDORRA</CO_DESC>
</COUNTRY>
.....
.....
.....
</COUNTRIES>

Iterating through all elements:

Code:

List<Element> countryList = root.getChild("BASISDATA").getChild("GEOGRAPHY").getChild("COUNTRIES").getChildren();
for (Element country : countryList) {
    regionService.updateRegionFromElement(country, Region.LEVEL_0);
}

Creating the region object and save it:

Code:

public Region createRegion(String name, String smId, Region parent, String zipcode) {
    Region region = regionManager.createRegion();
    region.setName(name);
    region.setSmId(smId);
    if (parent != null) {
      region.setParent(parent);
    }
    if (zipcode != null) {
      region.setCode(zipcode);
    }
    region.setChildren(new HashSet<Region>());
    region.setCompanies(new HashSet<Company>());
    regionManager.saveRegion(region);
    return region;
  }

applicationContext-hibernate.xml

Code:

<property name="initialPoolSize"><value>10</value></property><property name="minPoolSize"><value>10</value></property>
<property name="maxPoolSize"><value>100</value></property>
<property name="maxStatements"><value>0</value></property>
 <property name="properties">

<property name="hibernateProperties">
            <props>
                <prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
                <prop key="hibernate.hbm2ddl.auto">update</prop>
            <prop key="hibernate.cache.provider_class">org.hibernate.cache.OSCacheProvider</prop>
            <prop key="hibernate.cache.use_second_level_cache">true</prop>
            <prop key="hibernate.cache.use_query_cache">true</prop>
            </props>
        </property>

Region.hbm.xml hibernate-mapping

Code:

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
    "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
    "http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">

<hibernate-mapping>

  <class name="org.medusa.model.Region" table="region" optimistic-lock="version">
    <id name="id"
        column="regionId"
        type="java.lang.Long"
        unsaved-value="null"
        >
      <generator class="native" />
    </id>
    <property name="version" />
    <property name="smId" />
    
    <property
        name="name"
        type="java.lang.String"
        update="true"
        insert="true"
        access="property"
        column="name"
        length="100"
        not-null="false"
        unique="false"
        />
    <property
        name="code"
        type="java.lang.String"
        update="true"
        insert="true"
        access="property"
        column="code"
        length="30"
        not-null="false"
        unique="false"
        />
    <many-to-one
        name="parent"
        class="org.medusa.model.Region"
        cascade="none"
        outer-join="auto"
        column="parentId"
        />

    <set
        name="children"
        table="region"
        lazy="true"
        inverse="true"
        cascade="none"
        sort="unsorted"
        order-by="name"
        >
      <cache usage="read-write" />
      <key column="parentId" />
      <one-to-many class="org.medusa.model.Region" />

    </set>

    <set
        name="companies"
        lazy="true"
        inverse="true"
        cascade="none"
        sort="unsorted"
        >
      <cache usage="read-write" />
      <key column="regionId" />
      <one-to-many class="org.medusa.model.Company" />

    </set>

      </class>
</hibernate-mapping>

Top

webguy

Post subject:

Posted: Wed Jan 21, 2009 8:26 am

Newbie

Joined: Thu Feb 07, 2008 8:30 am
Posts: 7

I tried to skip the save method and then it runs much faster. This is the save method used:

getHibernateTemplate().saveOrUpdate(region)

Is there any other way to save the objects faster?

Top

nordborg

Post subject:

Posted: Wed Jan 21, 2009 8:57 am

Expert

Joined: Wed Mar 03, 2004 6:35 am
Posts: 1240
Location: Lund, Sweden

I think you are seeing this because you somewhere in your loop has a query. This is a guess only since your post doesn't show any query, but I have myself experience the exact same behaviour.

It all happens because with default settings Hibernate will execute a Session.flush() before a query is executed. And a flush() means that all objects in the session are dirty-checked. Which means that everything gets slower and slower as more objects are added to the session. There are at least two possible solutions:

* Evict objects from the session once they have been processed or call Session.clear() at regular intervals.
* Use a different flush mode, for example FlushMode.COMMIT.

For me, the second approach works best, but it may have some drawbacks if, for example, the file can contain duplicates. Then you must keep track of new records so that you don't call you createRegion() multiple times for the same region.

Another solution is to use a StatelessSession but it has a lot of limitations and will, for example, not work if you depend on cascades, uses collections and other "advanced" stuff.

Top

webguy

Post subject:

Posted: Wed Jan 21, 2009 9:22 am

Newbie

Joined: Thu Feb 07, 2008 8:30 am
Posts: 7

Thanks a lot for your suggestions.

I found the following link below that explains batch processing and is precisely what you suggested in your first solution.

http://www.hibernate.org/hib_docs/reference/en/html/batch.html

Code:

Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
   
for ( int i=0; i<100000; i++ ) {
    Customer customer = new Customer(.....);
    session.save(customer);
    if ( i % 20 == 0 ) { //20, same as the JDBC batch size
        //flush a batch of inserts and release memory:
        session.flush();
        session.clear();
    }
}
   
tx.commit();
session.close();

Top

Page 1 of 1

[ 4 posts ]

Board index » Hibernate & Java Persistence » Hibernate Users

All times are UTC - 5 hours [ DST ]

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum