Hello,
Im using hibernate and spring on java 1.6.
When i try to save a big set of 100,000 records i get stuck for 20 minutes! while hibernate loads the data.
I can see that the GC is working very hard.
I tried to disable the second cache and it stills slow.

this is the definition of the set:

<set name="paths" lazy="false" cascade="all-delete-orphan">
<key column="path_group_id" not-null="true"/>
<one-to-many class="src.PathSegment"/>
</set>

i didn't find a way to tell hibernate to save the data in batches all i could find it how to batch up lots of queries but no one bit query.

Thanks,
Omer C

On such large persistent collections hibernate suggest to use the extra-lazy feature.
Here's an article about the usage of this feature:

http://sites.google.com/a/pintailconsultingllc.com/java/hibernate-extra-lazy-collection-fetching

Thanks for the reply.
I checked the lazy issue but the thing is that if i work in lazy =true/extra then it slows down all the non db related application stuff.
The thing is that i tried replacing hibernate with jdbc and did the adding the set by myself (100,000 inserts) and it was done in 6 seconds ! while hibernate spends about 20 minutes on loading the data alone...

When the GC is working very hard, then it is most probably not hibernate which spends the 20 minutes on loading the data, most time will be spent by the GC. Probably your JVM has problem with holding all 100,000 objects in memory, so as first thing please examine what's your current -Xmx setting and take account of eventually increasing this value.



How did you recognize this !?

My Xmx is very high (6GB), much higher than what is needed for the amount of data I want to save (I also check the free memory in the system before starting to save the data).

I get this stack trace while the thread is stuck on saving the set (I did kill -QUIT to get the thread stack trace without killing it):

java.lang.Thread.State: RUNNABLE
at java.util.HashMap.transfer(HashMap.java:484)
at java.util.HashMap.resize(HashMap.java:463)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at java.util.HashSet.add(HashSet.java:200)
at org.hibernate.util.IdentityMap.entrySet(IdentityMap.java:207)
at org.hibernate.engine.StatefulPersistenceContext.getOwnerId(StatefulPersistenceContext.java:1104)
at org.hibernate.property.BackrefPropertyAccessor$BackrefGetter.getForInsert(BackrefPropertyAccessor.java:139)
at org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropertyValuesToInsert(AbstractEntityTuplizer.java:294)
at org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyValuesToInsert(PojoEntityTuplizer.java:239)
at org.hibernate.persister.entity.AbstractEntityPersister.getPropertyValuesToInsert(AbstractEntityPersister.java:3696)
at org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:290)
at org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:204)
at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:144)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:210)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:195)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:117)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:93)
at org.hibernate.impl.SessionImpl.fireSaveOrUpdate(SessionImpl.java:534)
at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:526)
at org.hibernate.engine.CascadingAction$5.cascade(CascadingAction.java:241)
at org.hibernate.engine.Cascade.cascadeToOne(Cascade.java:291)
at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:239)
at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:192)
at org.hibernate.engine.Cascade.cascadeCollectionElements(Cascade.java:319)
at org.hibernate.engine.Cascade.cascadeCollection(Cascade.java:265)
at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:242)
at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:192)
at org.hibernate.engine.Cascade.cascade(Cascade.java:153)
at org.hibernate.engine.Cascade.cascade(Cascade.java:120)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.cascadeOnUpdate(DefaultSaveOrUpdateEventListener.java:380)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.performUpdate(DefaultSaveOrUpdateEventListener.java:352)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsDetached(DefaultSaveOrUpdateEventListener.java:246)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:112)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:93)
at org.hibernate.impl.SessionImpl.fireSaveOrUpdate(SessionImpl.java:534)
at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:526)
at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:522)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.orm.hibernate3.HibernateTemplate$CloseSuppressingInvocationHandler.invoke(HibernateTemplate.java:1293)
at $Proxy9.saveOrUpdate(Unknown Source)

My configuration is like so:

<!-- The class that holds the set -->
<hibernate-mapping>
<class name="src.PathsGroup" table="path_groups" lazy="false">

<id name="id" column="path_group_id" unsaved-value="null">
<generator class="sequence">
<param name="sequence">
path_groups_path_group_id_seq
</param>
</generator>
</id>

<set name="paths" lazy="false" cascade="all-delete-orphan">
<key column="path_group_id" not-null="true"/>
<one-to-many class="src.PathSegment"/>
</set>

</class>
</hibernate-mapping>




<!-- The object in the set -->
<hibernate-mapping>
<class name="src.PathSegment" table="path_group_paths" lazy="false">

<id name="id" column="path_group_paths_id" unsaved-value="null">
<generator class="seqhilo">
<param name="sequence">path_group_paths_path_group_paths_id_seq</param>
<param name="max_lo">10000</param>
</generator>
</id>

<!-- int value -->
<property name="pathId" column="path_db_id"/>

</class>
</hibernate-mapping>


Session Factory Configuration:
<property name="dialect">org.hibernate.dialect.PostgreSQLDialect</property>
<property name="jdbc.batch_size">100</property>
<property name="cache.use_second_level_cache">false</property>

If you application is often executing in IdentityMap.entrySet() , then that means you have a very big persistence context with thousands of objects.
What I suggest is to use:
1. Use Extra lazy on big relation collections.
this avoids that hibernate always load's the entire collection before doing anything on it (for example a insert/put)
2. While inserting the 100.000 object in the relation, you should call intermediate flush() and clear() on your session.
This helps to keep your persistent context small and your application performant.

3. Use the batching feature if your JDBC-driver supports it, while doing large inserts, updates.


---
The first 2 points are more important than the 3rd.

I already tried all of this solutions but non seemed to work.
1. Extra lazy slows down my application which process the paths in different places in the code (Also if I set lazy to extra then I have to move the session object to places in the code which previously didn't need it, to do actions like size() and empty()).
2. I'm doing only one save of the object that contains the set, not saving each of the objects in the set.
3. Tried to add those parameters but it didn't work.

What really bothers me is that the set is large but its not that large.. Only 100,000 int values.
And when I do it in JDBC (And I'm doing it by sending an array with 100,000 PathSegments - so the memory I expect the JVM to use is more or less the same) it works in 6 seconds!!

How can this be ?

Also I found this open bug which I think is related:
http://opensource.atlassian.com/project ... e/HHH-1612

Hi Omer,

it's still not clear to me how you are processing.
Following of your 2 statements are confusing to me:





Is your intention just to add a further element in a collection which already exists on database with a size of 100.000 ?
Is this single action which takes to long time?
Or is your intention rather to insert the 100,000 objects into a brand new PathsGroup instance within one single transaction?
Maybe it would help when you attach your code which does the inserts ...

The third Option: "to insert the 100,000 objects into a brand new PathsGroup instance within one single transaction"
In My code there is only one line:

super.saveOrUpdate(pathsGroup);

The PathGroup Class contains a Set of PathSegments Objects.
And by saving the PathGroup Object hibernate also saves the set of PathSegments Objects. (cascade=all-delete-orphan).

Reviewing the stack I recognized something strange in following section:



It seems that for every element in the collection, hibernate calls cascadeToOne which
calls saveOrUpdate on your pathsGroup object.
Therefore it seems that saveOrUpdate on pathsGroup object will be called 100.001 times instead to one time only.
I'm not completely sure it is really happening this, but
if I will find the time next days I will try debug it with breakpoint at Cascade.java:291 ,
(which version of hibernate are you using)?
We have to find out which object is saveOrUpdated due the cascadeToOne action,
if it is pathsGroup itself or not...

I'm using version 3.3.1 but during the tests I changed the version once to 3.3.2 to see if there will be a change, there wasn't.
I'm standing by for your debug results.

Thank u very much for the help.

I tested now with 3.3.1 and I inserted the 100000 PathSegment in 33 seconds.



Maybe the difference is that I mapped the classes with annotations (jpa),
because I'm uncomfortable with hbm-files (I never used hbm-files before ) and I was not able to make it work.
I you can provide me you hibernate configuration file and the sample how you build your
sessionfactory, then I will try another time with exactly your hmb mappings.

I always get following exception when using the hbm files as you specify:



Can you please attach your complete sourcecode of class PathSegment, thanks.

Hey,
here is the src code:

public class PathSegment implements Persistent, Cloneable
{
private Integer pathId;
private Integer id;

PathSegment()
{
}

public Integer getPathId()
{
return pathId;
}

private void setPathId(Integer pathId)
{
this.pathId = pathId;
}

public boolean equals(Object o)
{
if (this == o)
{
return true;
}
if (o == null || getClass() != o.getClass())
{
return false;
}

PathSegment that = (PathSegment) o;

if (getPathId() != null ? !getPathId().equals(that.getPathId()) : that.getPathId() != null)
{
return false;
}

return true;
}

public int hashCode()
{
return (getPathId() != null ? getPathId().hashCode() : 0);
}
}

I never saw the exception you mentioned.

By the way, in the meanwhile I separated the save of PathsGroup and PathSegment (By removing the cascade "all" from pathsGroup hbm).
So now when i save PathsGroup is doesn't save the set of PathSegments. After saving PathsGroup I iterate the set of PathSegment and save them one by one. Total Time: 15 seconds.