I'm doing a performance evaluation of Hibernate w.r.t JDBC and I find that
in many cases Hibernate is a lot slower than JDBC - the Hibernate web site claims that the overhead is in the region of 10%.

When I do very simple queries on an object that maps to a table - no joins involved and nothing fancy I find that Hibernate 3-5 times slower than JDBC depending on the number of objects selected. Quite surprised by this I ran the JUNIT test that comes with Hibernate which does a comparison with Hibernate and JDBC and I found the results to be quite similiar to mine. Here are the results:

Objects: 8 - Hibernate: 10ms / Direct JDBC: 10ms = Ratio: 1.0
Objects: 16 - Hibernate: 10ms / Direct JDBC: 0ms = Ratio: Infinity
Objects: 64 - Hibernate: 20ms / Direct JDBC: 10ms = Ratio: 2.0
Objects: 256 - Hibernate: 150ms / Direct JDBC: 30ms = Ratio: 5.0
Objects: 512 - Hibernate: 210ms / Direct JDBC: 40ms = Ratio: 5.25
Objects: 1024 - Hibernate: 410ms / Direct JDBC: 70ms = Ratio: 5.857143
Objects: 2048 - Hibernate: 681ms / Direct JDBC: 180ms = Ratio: 3.7833333


I ran a few other queries in my evaluation - some of them complex and they were around 40 - 50 % slower than JDBC.

Insert statements on simple objects were around 30 - 60% slower than JDBC.

Hibernate seemed to perform best for updates - it was around 10% slower than JDBC. But then mass updates are not supported as documented.

So my question is - what is the scenario under which Hibernate is just ~10% slower than JDBC?

My problem is I like Hibernate - but our application may not be able to afford a performance degradation of 60 - 500 % in moving from JDBC to Hibernate. I'd like to know if my findings are far off the mark or if that is the general experience.

Of course, I haven't shown my code/mapping/db/drivers etc...I'm curious about other's experience in general.

Thanks,
Neel

I've read this article before. This does not address my observations.

Is it reasonable to say that in trivial selects, inserts, updates - but on large amounts of data - JDBC is significantly faster than Hibernate?

With complex queries - Hibernate seems to catch up - but is still 50-60 % slower than JDBC.

The nearest thing to a reprdouceable benchmark I found is the JUnit tests I mentioned - and that has unfavorable results for Hibernate.

I would really love to prove myself that Hibernate is in the JDBC performance region (10-25 % overhead is acceptable) - but that's not exactly what I'm finding....

Neel

Well, I'm not Hibernate Exp. yet, but, for the small time I spent with it, and the projects that I working on now using Hibernate, I'm wouldn't mind the Hiber/JDBC = 4 times.

My good reason is, when I code it take me less than an hour to make everything ready and linked with the database, while it take me ages with JDBC, usually the client won't wait for you to code or make a chnage, but he wouldn't mind few more sec. to see the result, at least my clients don't.

Hibernate is a development neccessity not performance neccessity.

I hope I did not upset you with my reply, anyway, there are allot of steps that you should do to gain more performance, specially if you suspect that you can get allot of records (remmeber you can always get the resultSet row count without actually loading them).

Post your code, let us see what we can do to make it faster.

I understand that there are many reasons one would consider before moving to an ORM tool - performance is just one of them. However, when one reads a statement like "Generally the overhead is much less than 10% of the JDBC calls" one gets very excited and begins to think of using an ORM tool even if the application does not SCREAM for an ORM tool.

This mail focusses on just the performance for Selects I noticed. I will share information on inserts, updates and complex queries. My tests were done on MS Sql Server using the DataDirect driver. Both databases were on the same machine but I also tested when they were across the network. I cleaned the DB cache for every run.

a) Simple selects are 3 - 5 times slower than JDBC on a simple Address object. You can try this on any other object - don't have to use mine.

Here's the Hibernate code:

public void testHibLoadAddress() throws Exception {

log.info(">>> testHibLoadAddress()");
Transaction tx = session.beginTransaction();
String queryString = "from Address";
start = System.currentTimeMillis();

//session.setFlushMode(FlushMode.NEVER);
Query query = session.createQuery(queryString);
//session.find("from Address add");
//query.setMaxResults(10);

List list = query.list();
end = System.currentTimeMillis();
System.out.println("Hibernate : testHibLoadAddress : Total time is "+(end - start));

tx.commit();
log.info("<<< testHibLoadAddress()");
}


Here's the equivalent JDBC code:

public void testJdbcLoadAddressStatement() throws Exception {
log.info(">>> testJdbcLoadAddressStatement()");
String queryString = null;
Statement stat = null;
ResultSet rs = null;
int numOfRows = 0;

try {

queryString = "select * from Address";
start = System.currentTimeMillis();
st = conn.createStatement();
boolean success = st.execute(queryString);
ResultSet rs1 = st.getResultSet();
numOfRows = 0;
while (rs1.next()) {
Address add = new Address();
add.setLastName(rs1.getString(1));
add.setFirstName(rs1.getString(2));
add.setStreet(rs1.getString(3));
add.setCity(rs1.getString(4));
add.setState(rs1.getString(5));
add.setZip(rs1.getInt(6));
}

rs1.close();
end = System.currentTimeMillis();

System.out.println("JDBC : testJdbcLoadAddressStatement : Total time is "+(end - start));

} catch (Exception e) {

e.printStackTrace();

}
}

I made sure I was not using the secondary cache and had not enabled logging.

Neel

I just had a look, Remove transaction cause there is no need for it here, I think this may help a little.

I will try to contribute more when I get back home :)

and why? if you're doing your bench as well as you're isolating your transaction, your bench must be wrong; ;)

I'm getting pretty bored of this topic.

As I have repeatedly pointed out in many threads: this kind of test is completely unrealistic, since you are accessing a dedicated local database, where all data is cached in memory, by the database, from a single thread, where there are no concurrent writers (or readers)!

Sure, for this kind of environment, I sometimes measure Hibernate is 2 times slower (not 5 times, you are doing something wrong to get that). But that is because, in this pathologically bad test environment, there is no disk access happening anywhere! Nor is there any concurrency.

A real data access environment is

(1) a loaded database,
(2) with only indexes and some data cached in memory
(3) and many concurrent readers
(4) and some concurrent writers
(5) probably remote

Furthermore, the overhead you are measuring is due to the session cache. For a fair test against JDBC, where you will measure about 10 - 15 % overhead, change the query like this:



And the overhead vanishes.

Can we stop writing the stupid tests already?

Not for the existing application I am evaluating this test for. As a general case - perhaps.



I think you've made a few assumptions here . I tried the test on local and remote database, the db cache was cleared everytime and I could see the disk reads.

There is no concurreny - but that is one of my use cases. There are other use cases where there is concurrency. Hopefully, Hibernate will fly there.

quote]String queryString = "select new Address(a.street, a.city, a.zip) from Address a";[/quote]

This is all that I was looking for - tell me how to improve my Hibernate code. With this change, Hibernate doubled it's performance. So in the use case where it was 4 times slow, it is now 2 times slow. The overhead still does not vanish.




Not quite. I'm sure other's will benefit from knowing the scenarios in which Hibernate does not do as well as JDBC. If that's acceptable that's fine - but always good to know.

Thanks for your response, Gavin. As I said, I have a few tests where Hibernate does do well.

Neel

You cleared the db cache between each iteration??? I doubt that. Remember, the first iteration will populate the cache.

If you are measuring a performance difference between Hibernate and JDBC accessing a remote database, your test is broken, simple as that. Access to a remote db is usually at least an order of magnitude slower than local. So 200% overhead => 20% overhead



For a system with no concurrency, performance of simple queries is irrelevant. The difference between 10ms per query and 20ms is certainly not detectable to the user. In general, we only care about performance in the case of highly concurrent systems, or for very complex queries (in the case of complex queries, you will certainly not measure any difference between Hibernate and JDBC).



Your test is broken, then. On my box, the overhead of this query is within 10-15%.

It seems that since you have two incorrect results, you are doing something wrong.

P.S. Note that for this test case, Hibernate is pretty much guaranteed to actually beat direct JDBC, since you can use the query cache. Call Query.setCacheable(true), and set hibernate.cache.use_query_cache.

Question about the query cache. I've noticed a lot of JDBC drivers for SQLServer ( jTDS, DataDirect ) use they're own method of query caching by creating a stored proc for each different query ( excepting parameters ) and then running the proc when the same query is run in the future. Does hibernate's query cache complement this behaviour or are they both doing the same thing?

I would imagine hibernate internally maps String -> PreparedStatement in it's cache and the driver does String -> Stored Proc which should complement each other, though I could be wrong about all of that.

It is not equivalent, test results must be very different if you will add objects to list (depends on list size).

That is not a query cache. That is a statement cache. Quite different things.

There are no multiple iterations here. The query is run just once. What I mean't to say is that I ran the test multiple times and before each run I would clear the DB cache to get realistic measures.



I hear this in every thread where same one dares measure simple selects and inserts. In my application, and in many others, it is very relevant. When we need to process 250 - 400 million rows for multiple queries this adds up to hours ! Stored procedures is not an option for various reasons.

Perhaps you find it strange that such applications even consider Hibernate as an option :)



Possibly - but I can't figure out where. My code is simple enough - as posted. The final measurement I have come up with is a difference of ~60% with JDBC being faster.

I also noticed that the execution of the statement on the database has a lot to do with the difference in the speed. Hibernate generates a PreparedStatement whereas in this case JDBC is issuing a regular statement. The execution time on DB difference is around 60 % - this accounts for a significant difference in the time measured.

So I did a test in JDBC itself between issuing regular Statements and PreparedStatements and the difference is ~50%. It seems to take a farim amount of 40 iterations for a Statement object to catch up with PreparedStatement i.e if a query is not likely to be used multiple times use a Statement object instead of PreparedStatement. I do not know if I can get Hibernate to use a StatementObject instead of PreparedStatement. I possibly can with a CreateSQLQuery but then I would lose the optimization Gavin suggested for session cache.




This didn't benefit me in this use case because I run the query just once.



I made this change in the latest measurements. This had a 20% effect.

I mean't to say:

So I did a test in JDBC itself between issuing regular Statements and PreparedStatements and the difference is ~50%. It seems to take multiple iterations for a PreparedStatement to catch up with Statement i.e if a query is not likely to be used multiple times use a Statement object instead of PreparedStatement.