Hi everybody. I have a failry simple question. I have included hibernate in an application where we have to save millions of elements in our database.

A first benchmark showed me that a simple casll to the session.save() method on a very simple bean (like the event of the hibernate tutorial) takes like 30 ms on my dev machine.

Is that normal ? If yes I am afraid that I will have to find another solution as it is really too much for a database writing... ( we would need weeks to save our objects !!)

Thank you for your kind help !


Karim, Tokyo, Japan

PS.: I am using this in a Java 1.6 environment, with Eclipse, a MySQL Database and EHCache.

Hi, this is partially normal, as measuring time for a simple operation is not very reliable.

The save operation is actually faster than that, but not too much as there is some latency involved to open the connection, the communication with your database, etc..

What you really want to do is use batching, you're not going to save just one entity at a time, you could save 1000 at the same time using a single insert and transaction; basically you still use the save() semantics but you're going to tell hibernate to flush the data to the database in big efficient batches (not too big! careful with your memory).

Look for batch operations and manual flushing on the reference, finetuning some parameters you can get extreme high throughput and keep the code quite simple.

http://www.hibernate.org/hib_docs/core/reference/en/html_single/#batch-inserts

hope it helps.

Since you have to update Millions of rows, you should seriously consider updating the data base using pure JDBC instead of using a ORM such as Hibernate. Hibernate is not very well suited for such mass updates.

in continuation with your advise

is there any other orm which is efficient in batch porcesses; ofcourse i know that there are problems in using an orm but there can be a case

Explore IBatis, though I am not an expert on that.

It really depends on the source and lifecycle of this "millions of elements", if you get them from another DB you should use some stored procedure, but if you have to modify or just look at them with Java then hibernate is really good, using batching you can get as fast as with your own JDBC code; all hints you could think of are applicable, and additionally you get him to do the mapping work; also it is well tested and probably Hibernate knows some more tricks to improve performance.

I don't want to start a flame, just express my personal experience: very high throughput
when well configured, and the bottleneck is my DBMS... you just have to enable some settings.

regards,

Thanks for your answers guys. Yes I have only to store objects and then I will make requests on those. Since I will not update the objects, at least not during the persistence procedure, I think that Hibernate will be a great solution as soon as I can get the batches work efficiently...

Remember, Hibernate calls just get translated into JDBC and SQL calls in the end. If the code is written well, and you capture those batch updates and perform intelligent flushing, you'll find Hibernate will often outperform your own JDBC apps, while being significantly more manageable and easy to maintain.

By the way does someone know how can I use Spring to force Hibernate to do batch insert ? As my application is extremely large I have to use Spring for many reasons.

In my DAO class I have:




But I am not sure at all how Spring manages HibernateTemplate and HibernateCallBack.

In my hibernate conf file I have set a batch size but I have no clue about how to force Spring to use it. I cant find any good documentation about it then if someone has an idea, I would appreciate!

Thanks,


Karim