Your test is absolutely worthless then. Please learn about benchmarking Java before blaming Hibernate. Read up on how the hotspot engine works. Really.....






Correct, we don't recommend ORM for this usecase. We recommend stored procedures.






Oh gawd, so you were benching Statement vs PreparedStatement for a single execution of the query??? Statement is always faster in that case. PreparedStatement dominates when the query is executed many times. Really, you shouldn't be doing silly benchmarks if you don't understand basic JDBC. Hibernate is not optimized for such trivial benchmarks, it is optimized for real highly concurrent server-side applications.





Conclusion: Hibernate is not optimized for use in applications that run queries once.

duh.

I don't think it was worthless - at least not for my app. The 200 % difference - after I did the changes - was when Statement was being measured against PreparedStatement.

When PreparedStatements in JDBC are measured against Hibernate ( which always issues PreparedStatements) the difference is 60 %. In this scenario I ran around 20 iterations of the query - varying one of the parameters in the prepared statement.

Perhaps if I run many, many iterations - at least the minimal required for PreparedStatments to catch up with Statements and then more for Hibernate optimization to kick in - Hibernate may reach the JDBC realm.

I just want to convince myself it does.



Seems to me If I can prove to myself that the overhead is in the region of 10-15 % based on all the tests that we do there is no reason why Hibernate should not be an option. As I said - in our case StoredProcedures cannot be considered. The choice is between JDBC and anything else.




Of course. The question is how many times must the query be typically executed before Hibernate pays off? That is by no means obvious and I would think any application should evaluate that.

Do you understand how HotSpot works? If you don't, don't do anymore benchmarks until you do.

I don't claim to be an expert on HotSpot but I do understand the basics.
Secondly - I evaluating an existing application - not providing a benchmark. But parts of it may be useful to others.

I'm quoting below an interview you gave javaperformnacetuning - http://www.javaperformancetuning.com/ne ... w041.shtml



It appears that my test cases are similiar to this. I cannot help it - I have to compare with existing code in our application.

Quoting you again:



I ran these same tests - and that was my first post. These show JDBC to be 5 times faster in some cases.

The keyword you use is "unhelpful in practice" - you do not state the results are wrong. Can you blame someone for doing the tests in a similiar way - especially after reading this interview?

I would suggest you stop putting down everybody who shows unfavorable results.

Welcome to Gavin's world :)

Well listen, I am trying not to put you down, but so far you have taken up more than an hour of my time with a silly benchmark that

(1) showed no awareness of the impact of HotSpot (namely that Java code runs faster the more times you execute it)
(2) showed no awareness of the difference between Statement and PreparedStatement
(3) compared JDBC code that read some data and then did nothing with it, against some Hibernate code, that not only returned the results into a list, but also added it to the session-level cache (I showed you how to avoid that overhead using "select new Foo(...) ...")
(4) probably has various other problems (you have not shown any information about session handling, connection pooling, etc)

I mean, it would be difficult to design a more flawed benchmark if I tried.

What you are forgetting here is that I actually know what the true overhead is, since I have spent many hours doing benchmarking work. Don't you get frustrated when someone comes and tries to argue with you about something when you know they are wrong. What about if you had to have this same argument approximately once a month every month??

Lets add up the cumulative improvements already:

(1) use of select new: 100% difference
(2) fair test of PreparedStatement: 50%ish difference on top of that
(3) actually do something with the data in the JDBC test 20% difference
(4) do stuff in a long enough loop to take advantage of HotSpot opimizations: x% improvement (20 may or may not be enough)

If I'm not mistaken, we are now down in about the 10%-20% range. I have no idea what other "buried treasure" waits to be discovered.

Oh, and

(5) take advantage of query caching: y% improvement

Now Hibernate is actually faster than JDBC.

To be brutally honest, would rather just ignore your flawed benchmark and get on with more serious work. Unfortunately, I have to respond to your benchmark since otherwise otherwise other people will read this thread and think it is correct.

That is incorrect.
I applied the same environmental conditions to JDBC and Hibernate. I didn't see the need to prepare favorable conditions for one or the other.



Wrong again. I do have a good understanding of this. I expected that in some situations Hibernate is clever enough to use Statements.



You already have. Check out your Junit test for performance.



Indeed I would get frustrated. I totally sympathise with you. What I would do though is (a) either write a benchmark and point everyone to it or (b) Utilise this opportunity to guide someone to help you write this benchmark

Secondly, nowhere have I been arguing. I have only been attempting to verify things for myself. This does not mean I do not trust you - but I need to understand the inner details.



For the record, at the end of the first three steps I stated that JDBC is 60 % faster that Hibernate in this query.

I have not verified (4) and the query Cache - but I don't see any reason to doubt the expected performance .



I am glad you responded. Because the alternative would have been to go by your flawed Junit performance tests.

I've said this before - I'll say this again. I like Hibernate. I just have to go through my tests to make sure it suits our application. And it is not a typical ORM application - but I still think Hibernate will be useful in it.

Guys - cool down.

May be I shall share my experience about my recent evaluation of Hibernate?

In my recent POC, this is what I've found out:

Hibernate is a fantastic ORM tool. It just depends on how you make usee of it. In our case, we are trying to remove most of our handcoded JDBC code and push things onto Hibernate.

Our situation looks bad enough - hundreds of tables, mostly denormalized, plus high volumes of data movement, with a few layers of DAO and etc.

About performance - I guess this might not be the most important thing that Hibernate is bringing to the table. Nevertheless, we still see very convincing results, because we are able to refactor out some of the hand-coded JDBC code. We achieved significant performance gain.

The most important thing that Hibernate brought us, is reduced system complexity, so the code is simpler, more organized and elegant.

I see that this topic here being focused on is purely about performance. I feel this might not be appropriate. Of course, projects often have a SLA tied to it. However, corporate IT projects is not just about system performance. I think we do have to consider other aspect like system complexity and managibility as well. On this end, I think Hibernate does excel.

If one is looking for an absolute performance king, may be one should consider writing highly optimized C code and does direct database access. I mean, I could hardly see anything that is faster.

Since we are choosing Java, and Hibernate (which is yet another layer again) I guess may be the better question to ask is, "is hibernate fast enough" or "is the simplicity brought to me by this ORM tool enough for me to sacrifice some performance, in terms of overhead?"

I understand why Gavin is upset. I do have to agree with Gavin that the test environment that is being described is not entirely highly concurrent and etc that the Hibernate product is design to excel in.

May be the tester can consider rerunning the benchmark in the context of the entire use-case? (Get the data, do something with it, etc) Then Gavin's concern might not have been so significant and the benchmark can be closer the the real life setting. I feel that, in that case, both you guys get a win-win situation.

Please consider.

A very happy Hibernate user,
Rob

totally agree with rsoo,
instead of thinking only about performances, just really think about what ORM is. Just read chapter 1 of hibernate in action to know when ORM might be used.

I think it is the best way for an app to let people think about business more than facade, ejb, base layers, or jdbc layer....
What i see is that one year ago Hibernate was already an excellent ORM.

One year later, i think we can say that it is the best, Toplink is also a great product but i love hibernate (and the team ;)) why?
Because they are able to waste 1 hour to teach you why and how things are without using commercial lies.
They are best pratices teachers!

I think Hibernate is mature, everybody is talking about Hibernate (read about ejb 3.0 for example).
So knowing that it is one the best ORM tools, you shouldn't talk about performance. If you need ORM, Hibernate is the most performant tool you need!

And last word about concurrency, add a security layer, a versionning layer, a transaction management layer upon your JDBC tests, and give us YOUR overhead.

I agree in principle with rsoo and Anthony.

I stated right at the beginning that performance is only one of the issues in an ORM tool.

Again, the application I am evaluating Hibernate for is not a classical fit for ORM. It has a domain model, it does a fair amount of business logic, it even has concurrency. And yet it is not a classical ORM application. For instance - it does not benefit much from caching. It does not benefit much from associative fetching. So you see - even if Hibernate fails in my scenario, it may not be Hibernate's fault.

What I'm venturing to prove here is that despite all this Hibernate may fit our application. If it does, this app may prove to be an exciting advertisement for applications that are not classically ORM apps.

I have a few other tests that shows Hibernate in good light - I have not been able to present that yet...

Here's a non-trivial query that I ran in JDBC and Hibernate:

"select claims, cpt4_cpt4group from Claims as claim, Cpt4Cpt4group as cpt4_cpt4group left outer join cpt4_cpt4group where {fn concat(rtrim(claim.cpt4),\'X\')}=cpt4_cpt4group.cpt4RecType and claim.patientid= :patientid' order by claim.patientid, service.fromdate";

Hibernate was around 5% faster than JDBC.

I ran this for 100 patients. In reality, this query will be executed a few million times. Based on my observations I expect that - if anything, Hibernate should be better in that scenario.

I have also not applied any Hibernate optimizations that will help it perform better - because just these results are acceptable to me. Anything more is a gain.

The database was local to my machine. That is how some of our customers run it - for various reasons.

Why do you think this workaround can be usefull for all users ? Implement wrapper for driver and do all workarounds you like, I do it this way too. It will be possible to reuse this code in any framework or to remove workarounds after you will upgrade driver.

In general, PreparedStatements take around 50 iterations and beyond to catch up with Statement objects. I'm sure every application has some SQL which is not executed that many times and they would rather prefer Statements in that scenario and PreparedStatements everywhere also.

This would have been useful in Hibernate for my app. This may not be a big deal for other classical ORM apps.

and PreparedStatements everywhere else.

sorry about the typo.

If your SQL is not executed that many times then you have nothing to optimize, it has no weight .