Greetings,

I am experiencing sporadic LazyInitializationException that occur when the application is experiencing high load and result in the following log output:

2007-02-26 17:48:26,468 [TP-Processor7] ERROR org.hibernate.LazyInitializationException: illegal access to loading collection
at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:341)
at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86)
at org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:109)
at org.hibernate.collection.PersistentSet.size(PersistentSet.java:139)
at org.apache.taglibs.standard.functions.Functions.length(Functions.java:193)

As you can see this is occurring when a JSP 2.0 custom tag uses the fn:length method to determine the size of a Hibernate managed Collection. We are using the Spring OpenSessionInViewFilter to facilitate a request scoped Hibernate session, and the above Exception does not indicating a problem with the session. Instead we seem to be running into a situation where one thread is attempting to access a collection while it is still loading.

Under light load everything works properly, but under heavy load it starts to fail, which leads me to think we have some kind of thread/timing issue. I have searched the forum and web for this Exception and can only find mention of session issues, which does not seem to be the case we have here.

One query that failed often was used to get the Roles a User (one-to-many) has access to, this is called at the beginning of each request. Since we are using AJAX this can get called multiple times per page view, and under load can result in a large number of similar queries running at the same time. Initially the Roles Collection in the User was set to lazy-load, I have since changed it to be non-lazy which resolved the issue. Now we see similar Exceptions for other Collections that have high usage. Obviously I cannot make all relationships non-lazy since we would end up loading large Object graphs into memory on Object construction.

Can anybody think of a reason why we would have a large number of "illegal access to loading collection" errors when under heavy traffic? I thought that since the Hibernate session is tied to the request it should be thread safe, but this error seems to indicate that multiple threads are accessing the collection simultaneously. I must be missing something, any help is appreciated.

Thank you.

- Matt


Relevant Technologies:
- Hibernate 3.2
- Spring 1.2.7
- Standard Taglib 1.1.2
- Java 1.4.2_11
- Tomcat 5.5.16 w/ Java 1.4 support

Try this
1. Change Lazy = "false" in your hbm.xml file

or

2. Ensure the collection is returned in your hql by doing
"from table n left join fetch n.<whatever the collection> is where n.Id=?"

Jayasimhan,

Thank you for the reply.

The problem is that I need to use lazy loading in some cases to ensure that I don't load large parts of the Object graph into memory when loading a single Object.

We're seeing a case where lazy loading works properly under light load levels, but under higher load we run into an issue with concurrent reads and loads of Collections which causes a LazyInitializationException.

I'm still confused as to how two threads could be accessing a Hibernate managed collection at the same time.

- Matt

Here is a small update.

We are only experiencing these Exceptions within the view (JSP) part of the request cycle and usually within JSTL tags. Since our Hibernate session is request scoped I do not see why this would make a difference. The Exception message indicates a dual access issue (illegal access to loading collection).

I can't bring myself to belive that Hibernate cannot support lazy loading in a high traffic environment, so something else must be occurring. Does anybody have any input or advice on where to look?

- Matt

I think the problem is not with hibernate session but with the application collection being referenced from multiple requests. By any chance you are using second level cache?

Somu,

Thank you for the reply. I agree that the problem seems to be that multiple threads/requests are accessing the collection at the same time. This occurs when there are multiple hits to the same dynamic page simultaneously and each thread is reading the same data, but I would think Hibernate would handle simultaneous hits to the same collection.

We are using the LocalSessionFactoryBean to manage the Hibernate session and have not enabled the second layer cache. By default that cache should be disabled.

I've added "hibernate.cache.use_second_level_cache=false" to our Hibernate configuration to make sure the second layer cache is disabled.

The error is still occurring under load, what is interesting is that it's always in a JSP page and always in the same statement which checks collection length, here is the line.

<c:if test="${fn:length(userView.region.views)==1}">

As you can see this is essentially calling userView.getRegion().getViews().size(). Where userView has a many-to-one relationship with Region and Region has a many-to-many relationship with Views.

Since I only need the size of the Views Collection for the Region I'm going to add a service method that performs a query to find out how many Views there are without actually loading the View Objects. This query will not use the first-level cache that comes with the Hibernate managed Collection, but since it is a very small query that shouldn't be much of an issue.

- Matt

If your different HTTP requests are using different hibernate sessions, then I think getList() for a collection shouldnot return the same reference to the collection. What is happenning in your case is multiple HTTP requests are accessing an identical reference of a collection. I think some debugging might be required to find out if different sessions are used for different HTTP requests and if at all the collections are identical.

I agree, since we are using request based Hibernate sessions each session should be accessing its own copy of the Collection. We are not starting any new threads during the request handling process, so each request should be a single thread.

I'm going to put in some code to ensure that each request has it's own session and that they are not accessing the same Collection.

- Matt

I've found references to people who experienced the "illegal access to loading collection" error when using reflection to calculate persistable Object's hashCodes. We are using a combination of the base Object hashCode and unique id for hashCode, so we should not fall into this bucket. I'm aware of the discussion of using natural, immutable properties vs ie for hashCode generation, and we've taken them into consideration in our design. Since the page that I'm testing with is "read only" the downsides of using an Id based hash should not come into play. Is this a safe assumption?

So, here is the big question... how are two threads apparently calling AbstractPersistentCollection.initialize() at the same time? Here is the method from the Hibernate source code.



This is the only place I can find where a "illegal access to loading collection" Exception is thrown, and unless the session.initializeCollection() method spawns a new thread then two threads must be calling this method simultaneously for the error to occur.

If the same Object is retreived using get() in two separate sessions will they both be instances of exactly the same Object. Ie, pointing to the same place in memory?

The strange occurrence is that this only seems to happen with collections or Objects related to the User we store in http session. Here is the general workflow for session User management.

1. User logs into the application and a Hibernate-attached User Object is put in http session

2. On subsequent page views, our AuthenticationFilter takes the now Hibernate-unattached User Object out of the http session, gets a copy of the User from the database using Hibernate's session.get(), and puts this Hibernate-attached User back into the http session. In this manner we always have a Hibernate-attached User in the http session while the request is being processed.

All of the errors I'm seeing are from this User Object or Objects it is related to (ex. user.getUserViews()).

We're making sure that during a request the User in http session is always attached to the Hibernate session to prevent LazyInitializationExceptions due to session. Is it possible that in doing so we have opened up issues with LazyInitializationExceptions because of "illegal access to loading collection" when under load?

1. Think about the scenario when multiple requests from the same user are getting processed simultaneously (possible under heavy load). Since http session is same, are you probably overwriting the original user object associated with the first hibernate session with the new user object associated with the second hibernate session?

2. I dont know your logic about overriding equals and hashcode (this might not be an issue in your case at all) but recheck your algorithm since this could be a source of error.

Coincidentally we just solved this issue last night and your #1 hypothesis is dead on. Here are some details for future readers who experience this issue.

We were loading the current User into http session at the beginning of each request, meaning that the User in the http session is always attached to the Hibernate session while a request is being processed. In cases where a large number of requests were rapidly made from the same client we started experiencing sporadic "illegal access to loading collection" errors.

This is because each request from the same client shares an http session, and each subsequent request would overwrite the User in session with a User attached to that requests Hibernate session. Because of this multiple request threads would end up using a User attached to a single Hibernate session. If each thread had time to fully initialize the User then we would not see any problems since the issue only occurs when two or more threads are in the same critical section at the same time. I have intentionally set up a worse-case scenario with a very slow database connection which increases the time required to load a Collection from the database, this increases the possibility of another request thread attempting to initialize the Collection while it is already being initialized by the original thread. This all lead to two causes of errors.

1. Since two multiple threads can wind up using the same Hibernate-attached User from the http session it is possible for two to call the AbstractCollection.initialize() method at the same time.

2. The User in http session is always the one put there by the last request. If that request completes before previous requests the previous requests can see "session is closed" errors from Hibernate.

Our solution was to put the User into request scope instead of session scope, this removes the possibility of Hibernate-attached Objects being used by multiple request threads.

The moral of the story is that if you put Hibernate-attached Objects in session then those Objects can be accessed by multiple request threads if they access the same data simultaneously.

Thank you again to somu for the assistance and for confirming this solution.

- Matt

That is why Seam (http://www.jboss.com/products/seam/) synchronizes concurrent access to conversation state that is held in the HttpSession. Any web framework needs to do that.