Hibernate version: 3.0.1 & 3.0.2

Hi,
I've posted regarding this before but what is the feasibility of adding fuctionality to the flush event infrastructure to allow users to tell the session exactly what is dirty (i.e. which properties on which entities are dirty including collections and taking into account referential integrity)?

The reason I ask is that we have a computation that has very many items in the cache at one time (20,000+) and runs many HQL finders in iterations that will alter items that affect the finder therefore causing an auto-flush to occur before each finder. This auto-flush processes each and every one of our 20,000 entities even though only a couple had been dirtied in that iteration.

Unsurprisingly we had terrible performance with this operation and it was taking up to 30 minutes to run. I've spent some time digging around inside version 3.0.1 and implemented some additions to the AbstractFlushingEventListener and associated code that improved our performance hugely by supplying hibernate with which entities/collections actually needed processing in each flush rather than looking at them all. This improved performance down to 3 minutes for that operation!

Unfortunately something has changed in version 3.0.2 and this code no longer works reliably, which is a pity as with some further improvements on our session management we had this operation down to 20 seconds!

Due to current time constraints we've decided to return to using the default Hibernate implementation of flushing again because with the session management changes we put in this operation is running at about 3 minutes which is acceptable for the time being - certainly better than the original 30mins!

However we would in the future like to return to fix this as 20 seconds for this operation is far better. Is this possible to have a custom interceptor of some sort that will provide hibernate with the dirtied entities to process in these flushes rather than processing them all?

regards,
Mike Hurd

FlushMode

Interceptor.findDirty()

Or is it more that you want to track which are dirty yourself somehow and avoid the looping here during flush?

Really I'd think that applying better control over the amount of entities in your session cache would be the correct solution.

That's exactly what we were thinking, we admittedly are using Hibernate in a slightly different manner to most of the web-apps and app server based implementations I've seen mentioned here. We've a swing fat client for complex network design and provisioning, our domain model is huge... Hibernate actually replaced a bespoke persistence mechanism reasonable painlessly as we had a wrapper layer that sits around the persistent object (now a hibernate pojo) and holds the business logic, this layer also manages the referential integrity of the model changes and has been modified to record which entities and accessors have been modified. This modified cache is currently tied to have a lifespan controlled by the Interceptor's postFlush() method. We have implemented the findDirty() method to use this information but it was far less of a performance improvement than removing the loop over the entire session cache.

Having profiled our app we were surprised to see that 90% if the processing was being taken up iterating in the flush mechanism. We figured that if there was some way to pass you the information on what actually has changed since the last flush that would massively improve performance (and it did with our hack which no longer works in 3.0.2).



Unfortunately we've gone as far as we can with minimising the number of objects in the cache, we've implemented a better session management system to restrict the number of extra objects in the cache and that helped (session per transaction) and this particular performance area can't be broken down any more, it is particularly complex.

Well, whats causing the session to flush in the first place? If this is a flush due to auto-flushing prior to running a query, then try using FlushMode as Christian mentioned.

If these are explicit flushes, why so many per transaction?

These are the auto-flushes that are running before a HQL finder, one of the pojos has changed so Hibernate determines that the SQL must be flushed to the db so that the changes show up in the finder when its run.

If we change the flush mode then won't the finders fail to pick any new objects (or newly associated) objects that have been created within the transaction?[/quote]

Let me explain this scenario a little more. This particular operation is very complex, involving the mapping of many possible entities (the main iteration) into a newly created entity. The act of mapping creates many associated objects that can then themselves be used in the subsiquent iterations and their mappings.

The decision of what to map to what is reasonably complicated and had previously been implemented by iterating all over the model interogating various properties of the pojos. This was slow and pulled in a huge amount of the model. We thought that the best solution would be to offload this work to the db in the form of a HQL query, and it should have been. But as I've mentioned each iteration affects the results of the finder to be run in the next iteration hence flush mode AUTO is required, I didn't really expect this to be even slower until I realised hibernate was processing every single pojo in the cache on every flush even though I knew only a handfull of objects had actually changed on each iteration.

I understand that hibernate was written to be light touch so it doesn't trap pojo modifications as they happen (except for collection changes?) but having to process every cached object to flush seems to be a real problem under certain circumstances - i.e. mine :) Incidently having the infrastructure in our framework to hand hibernate this information seemed like a blessing until I realised I couldn't find a way to provide hibernate with this information to incorporate into its flush processing.

I feel that this functionality would be very useful to have, this (I think) reasonably simple extension would allow our software to massively scale.

The other alternative of course is that if there is any functionality to flush the SQL for particular pojos we could turn off the the AUTO flush-mode and simply force a flush ourselves of those pojos we know to be dirty. Does/could hibernate support anything like this? Actually this doesn't sound much different to my previous suggestion, much of the infrastructure required for hibernate would be similar... Any more thoughts from the hibernate team on this thread?

any further thoughts on this?

Please don't bump up your postings. Your questions have been answered - manage your Session and FlushMode.

I'm sorry I appreciate the answers you've given but as I have explained these don't fully answer my questions. I've explained that we have managed the session and improved performance but we can't change the flush mode becaue of our finders.

You still haven't anwered my point about hibernate inspecting every element in the cache prior to each flush. Is this not a valid point considering that in our circumstances we know what is dirty hence can inform hibernate. When I was playing around with your flush implementation this made a massive difference to performance, I'm sure other people could use functionality like this.

And I'm sorry christian firing haughty one word answers after not reading a posting properly does not constitute an answer.

Manage the cache if you don't want Hibernate to check your objects. If your application design doesn't support this - change it.

You could actually acheive this through registering a custom auto-flush event listener.

Hi Steve,
Yes, that's what we did in version 3.0.1. We had a few problems with the visibility of the methods in the AbstractFlushingEventListener class we extended (some of the methods you might wish to call and types that are referenced are private so we had to copy them and related classes into our own package) and it seemed to work until version 3.0.2. It seems that there is some other processing of state happening in those loops and skipping over CollectionEntries and Entities (that we knew hadn't changed) caused some instabilities and unusual behaviour, maybe there's some initialisation that has to happen regardless of whether the processed entity is actually flushed or not.

All in all its a little confusing to work out all the things that are happening in the flush code. It doesn't seem restricted to simply flushing dirty entities.

We will probably wait for a while for this code to settle down and for the api's visibility in the abstract implementations of the event listeners to improve for those extending these classes. We'll then go back and implement this functionality, the performance gain is so substantial.

Well thats what everyone says.

Truth is, this code won't settle down until users start telling us what it is that they need from it. From the Hibernate-perspective, it does what it needs to do ;)

Like I tell everyone who says this: add enhancement requests to JIRA for what you'd like to see from the event stuff. Concrete examples, not "it should be better" ;)