-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 18 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Mysterious NoSuchElementException
PostPosted: Wed Dec 14, 2005 11:31 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
We are having problems with a mysterious NoSuchElementException within the Hibernate library which only seems to occur when the system is under heavy load and even then only occasionally (once every 5 hours or so).

We are using:
Hibernate version: 3.0.5
Database: PostgreSQL 8.0.3
Spring framework version: 1.2.3

Please see the stacktrace below. I have also included some snippets of the source code involved to clarify why we deem it so mysterious.

The stacktrace and the code snippet of SequencedHashMap.OrderedIterator below indicate that the iterator has already reached its end and that there is no next element available. However the code snippet of IdentityMap.entryArray() shows that prior to the invocation of method next() it has already invoked method hasNext() to verify the presence of a next element.

It seems that the only reasonable explanation for this behaviour is a concurrent modification of the collection by another thread such that afterwards no elements were left. Our application code does not make use of multi-threading! Tests have shown that only the standard system-group threads are present.

Could it be that somehow the Finalizer thread is causing a concurrent modification? E.g. when cleaning up Hibernate sessions that weren’t closed explicitly.

In any case it is clear that the design of method concurrentEntries() is flawed. It is not threadsafe while invoking method entryArray() although it is seemingly meant to support concurrent modification.

java.util.NoSuchElementException
at org.apache.commons.collections.SequencedHashMap$OrderedIterator.next(Unknown Source)
at org.hibernate.util.IdentityMap.entryArray(IdentityMap.java:195)
at org.hibernate.util.IdentityMap.concurrentEntries(IdentityMap.java:59)
at org.hibernate.event.def.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:104)
at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:59)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:26)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:730)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:324)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:86)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:490)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:495)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:468)
at org.springframework.transaction.interceptor.TransactionAspectSupport.doCommitTransactionAfterReturning(TransactionAspectSupport.java:258)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
at $Proxy3.invokeWithinNewTx(Unknown Source)
..........


IdentityMap (v 1.5 2005/05/14 17:51:16):

Code:
/**
* Return the map entries (as instances of <tt>Map.Entry</tt> in a collection that
* is safe from concurrent modification). ie. we may safely add new instances to
* the underlying <tt>Map</tt> during iteration of the <tt>entries()</tt>.
*
* @param map
* @return Collection
*/
public static Map.Entry[] concurrentEntries(Map map) {
   return ( (IdentityMap) map ).entryArray();
}

public Map.Entry[] entryArray() {
   if (dirty) {
      entryArray = new Map.Entry[ map.size() ];
      Iterator iter = map.entrySet().iterator();
      int i=0;
      while ( iter.hasNext() ) {
         Map.Entry me = (Map.Entry) iter.next();
         entryArray[i++] = new IdentityMapEntry( ( IdentityKey) me.getKey() ).key, me.getValue() );
      }
      dirty = false;
   }
   return entryArray;
}


SequencedHashMap.OrderedIterator (Revision: 1.28 $ $Date: 2004/02/18 01:15:42):

Code:
/**
  *  Returns whether there is any additional elements in the iterator to be
  *  returned.
  *
  *  @return <code>true</code> if there are more elements left to be
  *  returned from the iterator; <code>false</code> otherwise.
  */
  public boolean hasNext() {
      return pos.next != sentinel;
  }

  /**
   *  Returns the next element from the iterator.
   *
   *  @return the next element from the iterator.
   *
   *  @throws NoSuchElementException if there are no more elements in the
   *  iterator.
   *
   *  @throws ConcurrentModificationException if a modification occurs in
   *  the underlying map.
   */
  public Object next() {
      if (modCount != expectedModCount) {
          throw new ConcurrentModificationException();
      }
      if (pos.next == sentinel) {
          throw new NoSuchElementException();
      }
      ...
  }


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 11:19 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
you write your application is not multi threaded, but are you truely sure about that ? Don't you have concurrent users for your system under the heavy load you are referring to ?

I'm quite sure that this is happening because of thread unsafe usage of the session.

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 11:34 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
Hi Max. Thank you for your time.

Yes, we are pretty sure there is only one (application) thread.
We made JVM dumps several times by sending a SIGQUIT signal. Also we inserted some code into the Hibernate source for testing purposes. One of those things was a printout of the active threads at the time we caught a NoSuchElementException.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 11:40 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
I forgot to mention that the heavy load is due to processing batches of 100.000 to one million electronic messages per application (JVM) process.
Each message can lead up to 25 record insertions.

Within the application messages are processed sequentially. We (i.e. in our own code) do not use multiple threads.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 11:41 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
so if there only is one thread then there can't be any concurrent issues...and heavy load should not be the trigger, but more *what* the code does.

And I assume you only have one session open at one time here, right?

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 11:45 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
We use Spring to handle the creation and management of Hibernate sessions. While processing a message we have (at least) two Hibernate sessions present. The second session is used for logging purposes; logged information is committed even when the transaction of the first session is rolled back.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 11:47 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
and a finalizer only runs if noone else is referring to the object so don't think that should interfer.

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 11:55 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
It normally should not, but I conjectured that maybe the Session object that was finalized had a (indirect) reference to the IdentityMap that is having a problem. But of course I don't know that much about the Hibernate implementation.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Dec 20, 2005 2:27 pm 
CGLIB Developer
CGLIB Developer

Joined: Thu Aug 28, 2003 1:44 pm
Posts: 1217
Location: Vilnius, Lithuania
try to replace this map implementation with "java.util.IdentityHashMap" probably it can help to debug.
BTW do you share object instances in two sessions for logging purposes ?


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 6:56 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
No, with regard to persistence we do not share object instances in two sessions.

We do however load objects through one session and persist them through another session. In between we have evicted the objects from the first session to avoid conflicts with the second session.

We are currently trying to determine whether our problem has any relationship with the evict-action. The question still is whether and how this can be a concurrency-related problem.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 7:01 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 12:50 pm
Posts: 5130
Location: Melbourne, Australia
There are really three ways this can happen:

(1) Multiple threads accessing the session
(2) An illegal call back into the Session from, eg. a getter/setter method, Interceptor, etc
(3) Re-using the session after an exception occurs


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 8:18 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
Thanks for these suggestions. Do they apply to Session conflicts in general, or also for this particular problem?

I don't see how the latter two can interfere with the following code:
Code:
      while ( iter.hasNext() ) {
         Map.Entry me = (Map.Entry) iter.next();
         entryArray[i++] = new IdentityMapEntry( ( IdentityKey) me.getKey() ).key, me.getValue() );
      }


[/code]


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 9:52 am 
CGLIB Developer
CGLIB Developer

Joined: Thu Aug 28, 2003 1:44 pm
Posts: 1217
Location: Vilnius, Lithuania
Have you tried to run this stuff in debugger ? Suspend JVM (set "java.util.NoSuchElementException" breakpoint ) and view state.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 10:33 am 
Newbie

Joined: Fri Dec 09, 2005 7:57 am
Posts: 8
Location: The Netherlands
No, we haven't because it takes too long. In all of our (failed) runs the first time an exception occurred wasn't until after the first 50.000 messages or so.

Also we suspect that the rate at which this problem occurs is susceptible to variations in timing; for example one version of our software seems to get more frequent NoSuchElementExceptions than another.

At first we suspected some bug in the code with regard to weak references and garbage collection, so as a possible workaround we added some System.gc() statements to urge the JVM to do the collection at a safer moment. The SequencedHashMap implementation however does not make use of weak references. So this probably does not have any effect. However, we did get the impression that the problem occurred less frequent.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Dec 21, 2005 11:02 am 
CGLIB Developer
CGLIB Developer

Joined: Thu Aug 28, 2003 1:44 pm
Posts: 1217
Location: Vilnius, Lithuania
Try to reimplement this stuff, it must not solve this problems, but probably it will throw something more usefull to debug:

Code:
public Map.Entry[] entryArray() {
       
      return (Map.Entry[] )map.entrySet().toArray(new Map.Entry[ map.size() ]);

}


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 18 posts ]  Go to page 1, 2  Next

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.