-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 5 posts ] 
Author Message
 Post subject: Java stack overflow error with long IN lists
PostPosted: Thu Oct 19, 2006 8:40 pm 
Newbie

Joined: Tue Oct 17, 2006 9:53 pm
Posts: 3
With Hibernate 320ga a long "in" list can result in a stack overflow error during the parsing stage. For example, a query element like

where x in (:x)

or a manually constructed

where x in (1,2,3 .....)

can generate a stack overflow if the number of elements referenced by x exceeds a number dependent upon the amount of available stack space. For many JVMs, the limit is between 9,000 and 10,000 assuming a relatively empty stack at the point of query execution. We have applications which occasionally use lists several times this size.

The stack overflow occurs in org.hibernate.hql.ast.util.NodeTraverser which uses a recursive algorithm to walk a parse tree. Long "in" lists generate a subtree of depth about equal to the number of elements in the list. A sufficiently long list results in a stack overflow when NodeTraverser's internal method visitDepthFirst calls itself too many times.

The solution is to replace the recursive tree walking strategy with an iterative one that does not use up stack space. My suggested replacement code follows. This has fixed the problem for our applications.

Code:
package org.hibernate.hql.ast.util;

import antlr.collections.AST;
import java.util.Map;
import java.util.HashMap;

/**
* A visitor for traversing an AST tree.
*
* @author Steve Ebersole
* @author Philip R. "Pib" Burns.   Replaced recursion in tree traversal
*   with iteration.
*/

public class NodeTraverser {
   public static interface VisitationStrategy {
      public void visit(AST node);
   }

   private final VisitationStrategy strategy;

   public NodeTraverser(VisitationStrategy strategy) {
      this.strategy = strategy;
   }

   /**   Traverse the AST tree depth first.
    *
    *   @param ast Root node of subtree to traverse.
    *
    *   <p>
    *   Note that the AST passed in is not visited itself.  Visitation
    *   starts with its children.
    *   </p>
    *
    *   <p>
    *   This method originally called a recursive method visitDepthFirst
    *   which performed a recursive traversal of the tree rooted at the
    *   node specified by the ast parameter.  The original code looked
    *   like this:
    *   </p>
    *   <code>
    *   <pre>
    *   private void visitDepthFirst(AST ast) {
    *      if ( ast == null ) {
    *         return;
    *      }
    *      strategy.visit( ast );
    *      visitDepthFirst( ast.getFirstChild() );
    *      visitDepthFirst( ast.getNextSibling() );
    *   }
    *   </pre>
    *   </code>
    *   </p>
    *
    *   <p>
    *   The current code for traverseDepthFirst uses iteration to
    *   walk the tree.  This corrects stack overflow problems for
    *   constructs such as "x in (:x)" where ":x" specifies a large number
    *   of items.
    *   </p>
    */

   public void traverseDepthFirst( AST ast )
   {
                        //   Root AST node cannot be null or
                        //   traversal of its subtree is impossible.
      if ( ast == null )
      {
         throw new IllegalArgumentException(
            "node to traverse cannot be null!" );
      }
                        //   Map to hold parents of each
                        //   AST node.  Unfortunately the AST
                        //   interface does not provide a method
                        //   for finding the parent of a node, so
                        //   we use the Map to save them.

      Map parentNodes   = new HashMap();

                        //   Start tree traversal with first child
                        //   of the specified root AST node.

      AST currentNode   = ast.getFirstChild();

                        //   Remember parent of first child.

      parentNodes.put( currentNode , ast );

                        //   Iterate through nodes, simulating
                        //   recursive tree traversal, and add them
                        //   to queue in proper order for later
                        //   linear traversal.  This "flattens" the
                        //   into a linear list of nodes which can
                        //   be visited non-recursively.

      while ( currentNode != null )
      {
                        //   Visit the current node.

         strategy.visit( currentNode );

                        //   Move down to current node's first child
                        //   if it exists.

         AST childNode   = currentNode.getFirstChild();

                        //   If the child is not null, make it
                        //   the current node.

         if ( childNode != null )
         {
                        //   Remember parent of the child.

            parentNodes.put( childNode , currentNode );

                        //   Make child the current node.

            currentNode   = childNode;

            continue;
         }

         while ( currentNode != null )
         {
                        //   Move to next sibling if any.

            AST siblingNode   = currentNode.getNextSibling();

            if ( siblingNode != null )
            {
                        //   Get current node's parent.
                        //   This is also the parent of the
                        //   sibling node.

               AST parentNode   = (AST)parentNodes.get( currentNode );

                        //   Remember parent of sibling.

               parentNodes.put( siblingNode , parentNode );

                        //   Make sibling the current node.

               currentNode      = siblingNode;

               break;
            }
                        //   Move up to parent if no sibling.
                        //   If parent is root node, we're done.

            currentNode   = (AST)parentNodes.get( currentNode );

            if ( currentNode.equals( ast ) )
            {
               currentNode = null;
            }
         }
      }
   }
}

_________________
Philip R. "Pib" Burns
Northwestern University, Evanston, IL. USA


Top
 Profile  
 
 Post subject:
PostPosted: Fri Oct 20, 2006 1:28 am 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
bugfixes should go to our jira (but please check if there is an existing one already)

p.s. long in lists like that sounds craaazy ;)

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Fri Oct 20, 2006 2:54 pm 
Newbie

Joined: Tue Oct 17, 2006 9:53 pm
Posts: 3
The guidelines for bug submission say to post to the user forum first and wait for someone from the Hibernate team to determine if a bug submission is warranted. I assume I now have permission to submit the bug report to jira. I'll do it shortly. Thanks!

_________________
Philip R. "Pib" Burns
Northwestern University, Evanston, IL. USA


Top
 Profile  
 
 Post subject:
PostPosted: Fri Oct 20, 2006 3:02 pm 
Hibernate Team
Hibernate Team

Joined: Tue Aug 26, 2003 6:10 am
Posts: 8615
Location: Neuchatel, Switzerland (Danish)
Remember to read the part where I write: "but please check if there is an existing one already" ;)

_________________
Max
Don't forget to rate


Top
 Profile  
 
 Post subject:
PostPosted: Wed May 16, 2007 10:40 am 
Newbie

Joined: Thu Mar 04, 2004 2:41 pm
Posts: 2
Phillip - did you ever enter the bug? What's the bug ID? We're having the same problem, and I'd like to keep track of when the fix goes out.

Thanks,
Wayne


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 5 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.