-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 6 posts ] 
Author Message
 Post subject: Automatic deadlock recovery
PostPosted: Wed Feb 25, 2004 10:00 pm 
Newbie

Joined: Wed Feb 25, 2004 9:27 pm
Posts: 7
I would like to implement an automatic recovery scheme for deadlock exceptions thrown from JDBC commit. Something like this (incomplete) code:

public void commit() throws HibernateException
{
try
{
aJDBCTransaction.commit();
}
catch( HibernateException e )
{
<if e is determined to be JDBC database commit deadlock>
{
// rollback the connection
aSession.connection().rollback();

// wait a random amount of time
long t = 100+random.nextInt(2000);
try{Thread.sleep(t);}catch(Exception eee){}

// flush and commit again
commit();

// done
return;
}

// otherwise, not a deadlock exception
throw e;
}
}


Question: is it ok to call Hibernate's session.commit() again after rolling back the connection, for this particular case? Will Hibernate re-flush all pending DMLs?

If this is not correct, what approach is recommended to automatically retry a deadlocked commit?

(We've used this pattern with other ORMs+Oracle with success..)


Thanks.

--Jeff


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 26, 2004 3:03 am 
Pro
Pro

Joined: Tue Aug 26, 2003 8:07 pm
Posts: 229
Location: Brisbane, Australia
From the Hibernate 2.1.2 manual:
Quote:
Don't treat exceptions as recoverable.

This is more of a necessary paractice than a "best" practice. When an exception occurs, roll back the Transaction and close the Session. If you don't, Hibernate can't guarantee that in-memory state accurately represents persistent state. As a special case of this, do not use Session.load() to determine if an instance with the given identifier exists on the database; use find() instead.

I'm pretty sure that would apply to retrying transaction commits.
What you might want to think about is a command design pattern.

Instead of munging your transaction handling and domain model manipulation into one big procedural chunk, use the command/decorator patterns to do something like this:

Define an interface the represents actions that can be retried:
Code:
public interface RetryableAction {
  void try(Session session) throws HibernateException;
}


Then have an object responsible for executing retryable actions:
Code:
public class Executor {
  public static void execute(RetryableAction action) throw HibernateException {
    // get new HB Session
    // begin TX
    try {
      action.try(session);
      // commit TX
    }
    catch( HibernateException he ){
       // rollback TX
       // close & discard HB Session
       if( wadDeadlockError(he) ){
         // wait a random amount of time
         // get new HB Session
         action.try(session);
         // commit TX
         // close & discard HB Session
       }
  }


Then you'd just write code like this:
Code:
RetryableAction doSomething = new RetryableAction() {
  public void tryAction(Session session){
    // do some domain model stuff
  }
}

Executor.execute(doSomething);


Wouldn't something like this be heaps better than sprinkling retry code throughout your app?

Of course, I've not tried that approach before and I just made that example up.
If you did try this, I'd put some more thought into it that I have. Also, if you do do this and it works, add a page to the wiki that explains the details so other people can benefit from your experience.

_________________
Cheers,
Shorn.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 26, 2004 8:25 am 
CGLIB Developer
CGLIB Developer

Joined: Thu Aug 28, 2003 1:44 pm
Posts: 1217
Location: Vilnius, Lithuania
It must be better to design shema without deadlock(It is always possible in practice)


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 26, 2004 12:55 pm 
Newbie

Joined: Wed Feb 25, 2004 9:27 pm
Posts: 7
Wow, thanks for the speedy responses.

To Stolley: Actually, we do use your RetryableAction pattern, but only for very simple session/transactions that take place entirely within its own business context -- in order words, for a transaction that is ok to run over and over again until it succeeds without reguard to all other transactions and business logic operating around it.

In most cases, however, even though the transactional part of my logic is constrained to Hibernate, the rest of the logic in my transactions is not isolated to Hibernate alone, and so I must often deal with very expensive, non-trivial error recovery logic. Besides, rerunning an entire transaction just to recover from a simple transient database deadlock condition seems like overkill.

To baliukas: Of course logic should be designed to eliminate potiential deadlocks at the business logic level. However there are two cases where this is either unimportant or extremely difficult:

The unimportant case is where you don't really care that much about data integrity, but that you do care that the transaction succeeds. This happens, for example, when writing test applications to measure basic speed and performance of cpus, networks, disks, etc. of a Java+Hibernate configuration.

The unavoidable case is when working with databases that can encounter non-obvious deadlock conditions. For example, the old Sysbase versions did not have row locking, only page locking. The effect of page locking created conditions where two or more completely unrelated business rows could be locked together, just because they happened to be physiclly stored on the same page. Page level deadlocks were unavoidable unless you forced the business logic to take control over how rows are being physically stored in the database.

In newer databases with automatic row-level locking the problem is greatly reduced, but not eliminated. Even with Oracle it is sometimes possible to encounter systemic deadlocks that don't seem to be related to how your business logic is ordered.


--Jeff


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 26, 2004 1:53 pm 
CGLIB Developer
CGLIB Developer

Joined: Thu Aug 28, 2003 1:44 pm
Posts: 1217
Location: Vilnius, Lithuania
for "unimportant case" rows are deleted by garbage collection (it never runs concurently) and it looks like log, without updates.
All "systemic deadlocks" I have ever saw are user errors and app level recovery doe's not help.
I have used dealock recovery myself, but I faund it is possible to avoid deadlocks in all practical cases (Probably you have some very exotic use case, if you can not avoid deadlock).


Top
 Profile  
 
 Post subject: Re: Automatic deadlock recovery
PostPosted: Sun Nov 13, 2011 4:33 am 
Newbie

Joined: Mon Jul 07, 2008 12:45 pm
Posts: 2
Hello,
My company has developed a large/sophisticated system over a highly normalized data schema using Hibernate.
At any given time we have 20-30 concurrent transactions that select/modify data in primary 10-15 tables (total table count > 200 ) using pure Hibernate.

We get deadlocks. ALL THE TIME.

I don't understand how "system deadlocks" can be avoided:
1. SQL Server (our DB of choice) can throw deadlocks on selects, updates, inserts, and deletes. Close analysis of deadlocks reveals that it can occur while DB is updating (or even just selecting) a row, page, or index. This means, that a simple "employee.getName()" can fail.
2. As Hibernate user, I have very little control (if any) or sequence and timing of SQL statements.
3. Even, if we have a "deadlock-free" system, a new use pattern, or addition of new index can cause a deadlock.

In other words, in my practice, there is no clear fool-proof method to get rid of deadlocks in the code, especially in a very complicated system (like ours). "Deadlock-ness" of the system seems to me a complicated, semi-random, function of
1. Use patterns
2. Leve of concurrent use
3. Use of indexes and triggers in the database

Although I agree that deadlocks are bad, etc, I am convinced that Hibernate MUST SUPPORT deadlock recovery!!!

After spending 3 year developing the application (which passed all unit tests with flying colors), and then realizing that it works 99.99% of the time in production, i had to customize my hibernate version to support post-deadlock recovery.

I would like to find some similiar-minded souls and develop this into Hibernate (main or branch), because, as I said earler, this is a MUST-HAVE feature!!!


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 6 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.