-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 13 posts ] 
Author Message
 Post subject: session and pooled connections problem
PostPosted: Tue Jun 03, 2008 11:02 am 
Beginner
Beginner

Joined: Thu Oct 20, 2005 9:26 am
Posts: 27
Location: Barcelona
Hi,

I've got an issue related to pooled connections and the handling of the nhibernate sessions. I have read loads of documentation about these issues and no solution have helped me out so far, so I'll try and explain the whole problem in detail.

The symptoms are that the website I'm testing gives two different types of errors:
NHibernate.ADOException: could not load an entity: [...][...] ---> System.IndexOutOfRangeException: xx at System.Data.ProviderBase.FieldNameLookup.GetOrdinal(String fieldName) at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name) at NHibernate.Driver.NHybridDataReader.GetOrdinal(String name) at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String name) at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String[] names, ISessionImplementor session, Object owner) at NHibernate.Type.AbstractType.Hydrate(IDataReader rs, String[] names, ISessionImplementor session, Object owner) at NHibernate.Loader.Loader.Hydrate(IDataReader rs, Object id, Object obj, ILoadable persister, ISessionImplementor session, String[][] suffixedPropertyColumns)

and less frequently :

NHibernate.ADOException: could not execute query [... ] ---> System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first. at System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command) at System.Data.SqlClient.SqlConnection.ValidateConnectionForExecute(String method, SqlCommand command) at System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior) at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader() at NHibernate.Impl.BatcherImpl.ExecuteReader(IDbCommand cmd) at NHibernate.Loader.Loader.GetResultSet(IDbCommand st, RowSelection selection, ISessionImplementor session)

The site has been running for quite a while without these issues, but the whole problem started when we moved the site from a one server with both web server and database on it, to a three server load balanced setup, where we have two web servers and the database server behind.

From what I've been able to read it seems that these errors are due to pooled connections being mixed up at some level, not receiving a reset connection at the beginning or taking over a connection that belonged to another thread.

We are using the Open Session in View pattern, where the nhibernate session lives during the lifetime of the request. We found out that storing the session in the callcontext is not threadsafe, so we changed that to httpcontext.current.items, but it didn't help.

What does help is the two following things:
- at the beginning we had the database server identified by it' public ip, and changing that to the LAN ip minimized the errors drastically.
- if we add two worker processes per application pool the errors end completely.

This is however not a long term solution, it might work with the load the site has right now, but for futures sake we need to get to the bottom of this.

I'll be happy to post how we handle the sessions and other concrete code samples, but I wanted to give you a general overview of the whole issue first and see if anybody could see anything principally wrong with our setup or conclusions.

We're using NHibernate 1.2.1, ASP.NET 2.0, MS
MS SQL SERVER 2005 Standard but with the ms2000 dialect.

I hope you can help me out.
Thanks,
David


Top
 Profile  
 
 Post subject: Same problem
PostPosted: Fri Jul 18, 2008 10:04 am 
Newbie

Joined: Mon Mar 12, 2007 5:34 am
Posts: 7
We experienced the same problem, which started after updating a webapplication which had no problems before. The same errors.

We use the 2005 dialect (also tried 2000) dialect. Also had the same experience: changing databaseconnection to local ip reduces problems.

The post is old allready, i am wondering: did you find a solution?


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jul 18, 2008 11:44 am 
Beginner
Beginner

Joined: Thu Oct 20, 2005 9:26 am
Posts: 27
Location: Barcelona
no unfortunately we didnt find a solution yet.
I haven't gotten to try out the nhibernate version 2 to see if that helps, but I tried asking on the nhiberate email list as well and nobody had experienced it or had a solution.

Very strange, but at the moment it happens very infrequently so we're getting by. But I'm not happy about it.

I found out also that if you set the recycling of the workerprocesses in the application pool to be lower than default, there is less chance that it occurs.

Let me know if you come up with a solution?

Regards,
David


Top
 Profile  
 
 Post subject: Possible solution to problem
PostPosted: Mon Jul 21, 2008 2:31 pm 
Newbie

Joined: Wed Apr 16, 2008 3:01 pm
Posts: 1
Location: Rio das Ostras, Brazil
I had the same problem with an Oracle database and I was specifying OracleDialect. I read somewhere about someone solving this by changing the dialect to Oracle9Dialect and it worked for me. Although I know that you're not using Oracle, maybe your problem is caused by the same issue. I hope this helps.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Aug 10, 2008 8:43 am 
Beginner
Beginner

Joined: Thu Oct 20, 2005 9:26 am
Posts: 27
Location: Barcelona
I found an interesting long thread about this issue:

http://forums.microsoft.com/msdn/showpo ... 0&pageid=0

One of the solutions they suggest is this: "Microsoft recommends not passing the SqlDataReader to another method and avoiding any static methods usage, they are not sure this is the root cause to this Exception.".

I will try to look into the nhibernate code to see if this is the case and if it can be avoided.

/David


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 11, 2008 5:47 am 
Beginner
Beginner

Joined: Thu Oct 20, 2005 9:26 am
Posts: 27
Location: Barcelona
I believe I found a solution and would like your opinion about it, to see if it is possible to integrate with the official NHibernate release.

I have dived into the NHibernate 2.0.x code and found three instances of an IDataReader object being passed to a static method:
- identifiergeneratorfactory.cs: public static object Get(IDataReader rs, IType type, ISessionImplementor session)
- loader.cs: private static void ReadCollectionElement(object optionalOwner, object optionalKey, ICollectionPersister persister, ICollectionAliases descriptor, IDataReader rs, ISessionImplementor session)
- loader.cs: internal static void Advance(IDataReader rs, RowSelection selection)

I tried changing all of those to non-static methods. The first two methods are not used by NHibernate itself, but the third "Advance" is used in both multiqueryimpl.cs and multicriteriaimpl.cs.
I changed both instances to use an instantiated object inheriting from the Loader class (can't remember the type right now). As far as I can see that shouldn't change anything, but I would like your opinion on that as well.

To test it before I made the changes I first made a simple stress-test during an hour that did produce the errors a couple of times.
After changing it I made two tests, one during an hour and another during 4 hours and in neither case did the error occur even once. (All these tests were averaging a 5 requests per second varying between 10 different pages on the site).

So I'm pretty confident that this is the fix although I can't exactly explain why changing the method from static to non-static would make a difference in these cases.

I hope this would get accepted into the nhibernate official release, but I'm not sure what the normal procedure for this or how long it takes.

Let me know if you get a chance to try it out.


Top
 Profile  
 
 Post subject: Included in 2.0?
PostPosted: Fri Aug 29, 2008 11:59 am 
Newbie

Joined: Fri Feb 22, 2008 9:34 pm
Posts: 2
Was a JIRA ticket ever opened on this or was it addressed in the 2.0 release? I took a look at the 2.0 source and it doesn't look like it has. I'd like to see this fixed, as I am seeing this problem, too.

Also, did this fix all the problems? I only ask because it looks like you fixed static method usages but not other places where a datareader is passed between methods, though an earlier post in the thread seems to indicate that both are a problem.

How can passing a datareader between methods cause a problem unless they're depending on the stack being the same?

Gowry


Top
 Profile  
 
 Post subject:
PostPosted: Wed Sep 03, 2008 10:58 am 
Newbie

Joined: Wed Sep 03, 2008 10:54 am
Posts: 1
dgivoni, made the repair you suggested to our 1.2.1 code NHibernate code base, was wondering if you were willing to post your unit test-- either the design or code for it.

_________________
Man demonstrates his intelligence not by inventing tools, but in inventing tools to make tasks more difficult


Top
 Profile  
 
 Post subject:
PostPosted: Wed Sep 03, 2008 12:34 pm 
Beginner
Beginner

Joined: Thu Apr 27, 2006 5:49 am
Posts: 31
redshirt wrote:
dgivoni, made the repair you suggested to our 1.2.1 code NHibernate code base, was wondering if you were willing to post your unit test-- either the design or code for it.


It is great news that something has been done with this annoying problem. Waiting impatiently for the time when I will be able to download the fixed NHibernate version.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Sep 24, 2008 7:03 am 
Beginner
Beginner

Joined: Thu Apr 27, 2006 5:49 am
Posts: 31
Does anyone know if this solution was used in NHibernate 2.0.0.0.GA?


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jan 30, 2009 3:27 pm 
Newbie

Joined: Fri Jan 30, 2009 3:07 pm
Posts: 1
Location: California USA
Does anyone have a resolution for this yet? We use nHibernate2.0.1 and see this IndexOutOfRangeException occurring 800 to 1000 times a day on each of our high traffic web servers. Our servers have .NET3.5 and IIS6.0.

I understand that the MS Hotfix for the pooled connections reuse issue has been rolled into .Net3.5, but we are still seeing this issue. Leads me to think this is more related to IDataReader being passed to static methods in nHibernate code.

Has anybody implemented the conversion of static methods to non-static ones?? Has that helped alleviate your problem?? If so, please post some details on what changes you made to nHibernate...

Thanks,

Marzban


Top
 Profile  
 
 Post subject: Re: session and pooled connections problem
PostPosted: Tue May 05, 2009 12:08 pm 
Beginner
Beginner

Joined: Thu Oct 20, 2005 9:26 am
Posts: 27
Location: Barcelona
Sorry for not getting back on this issue before.
Actually the fix with the methods I proposed didn't work. I think it reduced it, but soon after one of our clients got an even higher traffic load, so the issue was/is still recurring.
Did any of you find a solution to this?


Top
 Profile  
 
 Post subject: Re: session and pooled connections problem
PostPosted: Tue May 05, 2009 11:42 pm 
Beginner
Beginner

Joined: Thu Apr 27, 2006 5:49 am
Posts: 31
It seems I have found the solution for my case. The problem was connected with using loaded object in different threads. Imagine, you have loaded an object via session and you have passed the object as a parameter to a different thread. If there are any property that are lazy loaded, then you will have another thread using the same session object. That is where the problem arises. If this session is busy with a long query to database and another thread tries to load the property, then you will have an exception and after that the whole NHIbernate become unstable.

I solved the problem by passing only object Id's to different threads and re-fetching them there. I believe that all other problems are connected with such implicit using of sessions in different threads


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 13 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.