NonUniqueObjectException and lost inserts

darac · **Joined:** Thu Sep 17, 2009 3:54 am **Posts:** 5

I used tables with native (identity) generator on both hsql and MSSQL 2005 Database.
Error occures randomly on Hibernate 3.3.1GA & 3.2.6GA therefore it's difficult to debug

I am experiencing a problem when persisting a large amount of Serialized objects within a single Transaction. (But not all objects of same type/mapping)
Sometimes the savOrUpdate doesn't actually perform an insert (skips an insert) - but assigns the bean in the persistence context the wrong generated Id.
The assigned generated id is actually genereted by an insert that should have been executed later in the batch.
The Insert that followed and got the correctly generated id, is inserted correctly to the database according to the transaction logs.
But the check of uniqueness fails because the bean in the persistencecontext which has NOT bean inserted properly (according to the transaction logs of the database and !!without!! errormessage) has the given id already assigned (in session context).

Maybe an example will make more clear what happens.
4 objects A,B,C,D will get persisted with saveOrUpdate the generated ids should be 0,1,2,3
Iteration 1 (OK) :
A -> saveOrUpdate -> Insert A into xyz (persist in Db) -> DB has Transact logs -> call identity delivers ID 0 -> bean in sessioncontext has id 0
Iteration 2 (something wents wrong) :
B -> saveOrUpdate -> Insert B into xyz (persist in Db) -> DB has NO Transact logs -> somehow bean gets id 2
Iteration 3 (OK) :
C -> saveOrUpdate -> Insert C into xyz (persist in Db) -> DB has Transact logs -> call identity delivers ID 1 because previous insert went wriong without error
-> bean in sessioncontext has id 1
Iteration 4 (Error happens) :
D -> saveOrUpdate -> Insert D into xyz (persist in Db) -> DB has Transact logs -> call identity delivers ID 2 -> bean in sessioncontext has id 2
-> checkOfuniqueness throws exception because a bean with the id is already in the session.

org.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: [de.tts.bd.model.HistoryEntry#4223]
at org.hibernate.engine.StatefulPersistenceContext.checkUniqueness(StatefulPersistenceContext.java:613)
at org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:326)
at org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:204)
at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:130)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:210)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:195)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:117)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:93)
at org.hibernate.impl.SessionImpl.fireSaveOrUpdate(SessionImpl.java:534)
at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:526)
at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:522)
at de.tts.bd.business.exchange.serialize.ImportSerializable.persistEntity(ImportSerializable.java:260)

The mapping is straight forward - just a primary key no foreign keys:

Code:

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
        "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
        "http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
    <class name="de.tts.bd.model.HistoryEntry" table="ttbd_history" mutable="true">
      <id name="id" column="history_id" type="long"><generator class="native"/></id>
      <property name="timestamp" type="timestamp" column="event_time" not-null="true" unique="false"/>
        <property name="event" type="character" column="event_type" length="16" not-null="true" unique="false"/> <!-- "+", "-", "#", ">" -->
        <property name="objectID" type="long" column="object_id" not-null="true" unique="false"/>
        <property name="objectType" type="string" column="object_type" length="16" not-null="true" unique="false"/>
        <property name="objectUUID" type="string" column="object_uuid" length="40" not-null="true" unique="false"/>
        <property name="userName" type="string" column="username" length="64" not-null="false" unique="false"/>
        <property name="userProfileID" type="long" column="user_profile_id" not-null="false"/>
        <property name="changes" type="text" column="changes" not-null="false" unique="false"/>
   </class>
</hibernate-mapping>

Code:

...
        public Long getId() { return this.id; }
   public void setId(Long id) { this.id = id; }

   /**
    * @see java.lang.Object#equals(java.lang.Object)
    */
   @Override
   public boolean equals(Object obj) {
       if(obj instanceof Decorator) {
           obj=((Decorator)obj).getDecorated();
       }
      if(obj == null || !this.getClass().isAssignableFrom(obj.getClass())) {
         return super.equals(obj);
      } else if(this.getId() == null || ((Entity)obj).getId() == null) {
         return super.equals(obj);
      } else {
         return this.getId().equals(((Entity)obj).getId());
      }
   }

   /**
    * @see java.lang.Object#hashCode()
    */
   @Override
   public int hashCode() {
      return this.getId() != null ? this.getId().hashCode() : super.hashCode();
   }

   /**
    * @see java.lang.Object#toString()
    */
   @Override
   public String toString() {
      return new StringBuffer()
         .append(this.getClass())
         .append("(")
         .append(this.getId())
         .append(")").toString();
   }
...

Does someone have an idea.
I already tried to reduce the amount of data per transaction by splitting and persisting data in multiple transactions.
Also I tried to isolate the problem with load tests -> massive saveOrUpdates outside the Webcontext of single Entitytypes but wasn't reproducable.
Any ideas or help appreciated.

Froestel · **Posted:** Thu Sep 17, 2009 7:04 am

Hi darac,

if I read your post right you included the DB generated id in your hashCode() and equals() methods. You should not do that!

Sec 4.3 (http://docs.jboss.org/hibernate/stable/ ... lshashcode) of the Hibernate Core documentations reads

Quote:

The most obvious way is to implement equals()/hashCode() by comparing the identifier value of both objects. If the value is the same, both must be the same database row, because they are equal. If both are added to a Set, you will only have one element in the Set). Unfortunately, you cannot use that approach with generated identifiers. Hibernate will only assign identifier values to objects that are persistent; a newly created instance will not have any identifier value. Furthermore, if an instance is unsaved and currently in a Set, saving it will assign an identifier value to the object. If equals() and hashCode() are based on the identifier value, the hash code would change, breaking the contract of the Set. See the Hibernate website for a full discussion of this problem. This is not a Hibernate issue, but normal Java semantics of object identity and equality.

The referred to discussion on this subject can be found at: https://www.hibernate.org/109.html

CU
Froestel

darac · **Joined:** Thu Sep 17, 2009 3:54 am **Posts:** 5

first .. thanks for your reply.
I considered the possibility that hashcode could be the problem but in the end I don't think it's relevant here.
No Objects are overridden here because two objects with a null id are inserted.
As you can see in that case super.hashcode() would be called.
I am aware of that and in my businesscode I'm using Lists until the objects are persisted.
The only place where this could be a problem is the session persistencecontext after the id of an inserted object is updated -> hashcode changes.
A Possible result would be that the contains() method of the sets used by hibernate would not return true any more.
In fact this could not be the problem because the CheckofUniqueness delivers a right result it's only not right that it is there in the first place.

The questions bothering me are:
Why does it happen after hundred or thousands of successfully insertions - that's the main reason why I don't believe in the hashcode theory?
Why is the row/object not inserted at all? And without any error?!
Why does the bean in the pers.context have a generated id it would have only - if the call identity() was called after a later insertion? Is there a race condition?

Code:

  ->shouldn't it be synchronious?
    EntityPersister persister = source.getEntityPersister( entityName, entity );
    Serializable generatedId = persister.getIdentifierGenerator().generate( source, entity );

If it would be a hashcode problem during the id update of the session bean e.g. the bean's id is updated because the current id is null;why does it not have the id of the next iteration but the second that followed? (well sometimes the second / sometimes the first).
And why does the code work in 50% of all tests properly on the same DB/ same Data?

Froestel · **Posted:** Thu Sep 17, 2009 9:20 am

Hi darac,

you are right - it's a theory on the hashCode() thingy. I haven't checked it. But also once had trouble with equals() and hashCode().

Well, obviously you haven't tried to change your equals() and hashCode() - this should be straight forward since most IDEs support some generation mechanism.

I think, you should give it a try. If your hashCode() method calls the super.hashCode() in case of the ID being null you will sooner or later end up having completely different objects with the same hashcode if they both are transient. Or the other way round, you have equal objects with different IDs. I cannot imagine that this is what you want.

If there are at some point Sets involved then there will be trouble since the Hibernate internal set implementation is documented (http://docs.jboss.org/hibernate/stable/core/reference/en/html/collections.html#collections-persistent) to behave like a Java HashSet which involves calls to its elements' hashCode() method.

If you're code does concurrent DB operations equals() and hashCode() are of importance and should be able to distiguish entities upon their business properties.

A generated identifier is a Hibernte/DB - thing this has nothing to do with your business code and you should not rely on it. It can be treated completely invisible for the business code.

Quote:

Why does it happen after hundred or thousands of successfully insertions - that's the main reason why I don't believe in the hashcode theory?

I don't think Hibernate makes any promises on the ordering it does stuff. Especially in concurrent operations you can easily have collisions due to your hashCode() method. It could happen at the second insert but as well at the 2.000.000th.

Quote:

Why is the row/object not inserted at all? And without any error?!

Could it be that Hibernate found it equal to an already inserted item?

Quote:

(well sometimes the second / sometimes the first).

This sounds very much like race conditions.

CU
Froestel

darac · **Joined:** Thu Sep 17, 2009 3:54 am **Posts:** 5

Well it was worth a try. I gave it a chance and changed the hashcode method to simply return super.hashcode()

Code:

public int hashCode() {
      //return this.getId() != null ? this.getId().hashCode() : super.hashCode();
      return super.hashCode();
   }

So the hashcode of the bean shouldn't change during the persistence process.
But the same error occurs.....

Froestel · **Posted:** Fri Sep 18, 2009 5:59 pm

Hi darac,

darac wrote:

... and changed the hashcode method to simply return super.hashcode()
...
But the same error occurs.....

Well, what did you expect? - If the error actually occurs anytime the hashcode is calculated when the id is null you have the same as before... you basically changed nothing according to the error.

If the objects to be persisted are to be distinguished independently of serialization ID you should calculate a hashcode that is unique for an object - at least when it comes to the values of fields. So base your hashcode calculation on the values of the fields of your objects to be persisted. At least this has a better chance for success...

CU
Froestel

Cameron McKenzie · **Posted:** Fri Sep 18, 2009 7:20 pm

Use the instance variables of the class to calculate a hachcode that will uniquely identify instances. This should help avoid this problem.

darac · **Joined:** Thu Sep 17, 2009 3:54 am **Posts:** 5

OK I experimented a little bit with the hashcode .
Because the hashcode() method is placed in a baseclass I chose a more generic approach.

Code:

@Override
public int hashCode() {
   String[] excludedFields={"id","serialVersionUID"};
   int hashCode=org.apache.commons.lang.builder.HashCodeBuilder.reflectionHashCode(this,excludedFields);
   StringBuffer sb= new StringBuffer(getClass().getName()). append(hashCode);
        return sb.toString().hashCode();
   }

Well in the end nothing worked out - at least the NonUniqueObject exception is thrown less often and the following Exception comes up instead.

Code:

Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1
   at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:85)
   at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:70)
   at org.hibernate.jdbc.BatchingBatcher.checkRowCounts(BatchingBatcher.java:90)
   at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
   at org.hibernate.jdbc.BatchingBatcher.addToBatch(BatchingBatcher.java:56)
   at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2431)
   at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2335)
   at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2635)
   at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:115)
   at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
   at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
   at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)

I guess those are the rows which were not inserted at all (as i described in my first post).

Fazit:
I agree that the hashcode should not change during the persitence process - but it's difficult to write a generic algorithm, secure enough that implements it without adding a UUID column to ALL my tables -> the approach above with the "businesskey" isn't secure enough I know - but is not the cause why it's failing in 50% of the time.
I think there is a bug in Hibernate.

jpkrohling · **Posted:** Tue Sep 22, 2009 6:37 am

Can you check if the exception type of the last problem you were experiencing is a "StaleStateException"?

https://www.hibernate.org/hib_docs/v3/a ... ption.html

If not, are you able to build a test case which reproduces the failure?

Stef_Kuypers · **Joined:** Fri Aug 14, 2009 8:06 am **Posts:** 19

Hi,

I'm having a similar problem but in my case I get a NonUniqueObjectException every time I try to insert a second object in my database.

I'm running with the latest stable build of Hibernate (3.4.0 GA), am using hsqldb 1.9rc4 and use eclipse 3.5 on a mac with a 1.6 jvm under snow leopard.

I reduced my hbm file of the class to an absolute minimum:

Code:

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping package="be.bloso.vetrasoft.model.application">
   <class name="DataItem" table="DATAITEM">
      <id name="id" column="DATAITEM_ID">
         <generator class="org.hibernate.id.enhanced.SequenceStyleGenerator" />
      </id>
   </class>
</hibernate-mapping>

The code of the class (reduced too) is:

Code:

public class DataItem implements Comparable<DataItem>, Serializable, DataChangeListener {
   private static final long serialVersionUID = 1L;

   private Long id;
   private Data rootData;
   private String data;

   protected DataItem() {
      super(); // Used by Hibernate
   }

   DataItem(Data rootData, String data) {
      if (data == null) {
         throw new IllegalArgumentException("The data can not be null");
      }

      this.rootData = rootData;
      this.data = data;
   }

   public Long getId() {
      return id;
   }

   protected void setId(Long id) {
      this.id = id;
   }

   public Data getRootData() {
      return rootData;
   }

   protected void setRootData(Data rootData) {
      this.rootData = rootData;
   }

   public String getData() {
      return data;
   }

   public void setData(String string) {
      this.data = string;
   }

   @Override
   public String toString() {
      return data;
   }

   public int compareTo(DataItem item) {
      if (item.rootData != rootData) {
         throw new IllegalArgumentException("Can not compare data items from different data sets.");
      }

      return toString().compareTo(item.toString());
   }

   @Override
   public int hashCode() {
      return super.hashCode() + 37 * data.hashCode() + 37 * 37 * rootData.hashCode();
   }
}

The test code I execute is the following:

Code:

      DataItem dataItem = Data.ACCENT.add("test");
      DataItem subData1 = dataItem.addSubData("subdata 1");
      DataItem subData2 = dataItem.addSubData("subdata 2");

                session = HibernateUtil.getSessionFactory().getCurrentSession();
      session.beginTransaction();
      session.save(dataItem);
      session.save(subData1); // <- error occurs
      session.save(subData2);
      session.getTransaction().commit();

And every time I execute this code I get the same error:

Code:

org.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: [be.bloso.vetrasoft.model.application.DataItem#1]
   at org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:191)
   at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:144)
   at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:210)
   at org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:56)
   at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:195)
   at org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:50)
   at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:93)
   at org.hibernate.impl.SessionImpl.fireSave(SessionImpl.java:562)
   at org.hibernate.impl.SessionImpl.save(SessionImpl.java:550)
   at org.hibernate.impl.SessionImpl.save(SessionImpl.java:546)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:342)
   at $Proxy7.save(Unknown Source)
   at be.bloso.vetrasoft.model.application.DatabaseTest.testDataItem(DatabaseTest.java:88)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
   at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
   at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
   at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
   at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
   at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
   at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46)
   at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

I changed my hashCode() method, deleted my equals() method so it defaults to object identity and still I keep getting an error that I try to insert non unique objects. It's driving me up the wall!

jpkrohling · **Posted:** Tue Sep 22, 2009 10:46 am

Stef_Kuypers, where is your addSubData method?

Stef_Kuypers · **Joined:** Fri Aug 14, 2009 8:06 am **Posts:** 19

I edited a bit much :)

Here it is:

Code:

   private final UniqueList<DataItem> subData = new UniqueList<DataItem>();

   public DataItem addSubData(String string) {
      DataItem item = getSubData(string);

      if (item == null) {
         item = new DataItem(this, string);
         subData.add(item);
      }

      return item;
   }

   public DataItem getSubData(String string) {
      DataItem item = null;
      int index = 0;

      while (index < subData.size() && item == null) {
         DataItem subItem = subData.get(index);

         if (subItem.toString().equals(string)) {
            item = subItem;
         }

         index++;
      }

      return item;
   }

UniqueList is a wrapper for ArrayList which also implements Set.

jpkrohling · **Posted:** Tue Sep 22, 2009 11:40 am

Stef_Kuypers, and what about Data, referenced as "Data rootData"? But really, it is not an uncommon error. I would suggest you to Google for it, as you can read the causes and compare to your environment :-)

darac · **Joined:** Thu Sep 17, 2009 3:54 am **Posts:** 5

@jpkrohling sorry that I'm answering so late but I had a bit of business right here.
You asked for the Exception that is thrown .. you were right it's a StaleStateException and it's thrown when updateing a row wich was not inserted in Db but is falsely updated in Persitencecontext.
So the update query alters zero rows. -> Exception
The accordingly SQL-statement is like follows:
update ttn.ttbd_history set event_time=?, event_type=?, object_id=?, object_type=?, object_uuid=?, username=?, user_profile_id=?, changes=? where history_id=?

I tried out to change the mappings generator to assigned -> works fine no errors at all.
I see no other option then to refactor our code to make use of a hilo generator -> hoping it will work better.

@Stef_Kuypers maybe you should consider changing the genrator too

P.S. I was not able to isolate a testcase -> even when i tried. When I'm saving just HistoryEntry objects in a testcase even 50000 or more everything goes well.
But when I'm serializing/deserializing the whole DB (all mappings) and save them - the error above occures (randomly on the same data in 50% of the cases).
?:-(

Stef_Kuypers · **Joined:** Fri Aug 14, 2009 8:06 am **Posts:** 19

Changing the generator to hilo did the trick. I went to the logs when using org.hibernate.id.enhanced.SequenceStyleGenerator and apparently it only generates 1's, which explains the error. So there's a bug in the generator I guess.

Thanks for the hint!