Hello, i already posted this but did not realize i posted it in the Dutch forum. Sorry for the Cross post!!!!

I am in a situation where the most critical part of our app is when a text file is loaded and millions of rows are inserted from the file. concurrency is not so much an issue since only one thread will do the work.

My requirements are to handle several million of transactions, mySql database already held millions of rows before these tests

My setup is basically a page that calls a session bean with the following code:
http://rafb.net/paste/results/M5s1CJ45.html

My object has 50+ fields and all are VARCHAR 255

i dont yet care about RAW performance, i care about understanding RELATIVE performance between hibernate vs jdbc


Also i understand that performance in hibernate is less because of all the wonderful features hibernate has.

My results are these:
http://picasaweb.google.com/fedevela

In raw insert times (without commit) hibernate is outperformed by a factor of 10

in raw commit times, hibernate is outperformed by a factor of 3

Finally we get to my questions:

What issues may arise if I switch to a JDBC approach in the parts of my application that require it?

If i switch to JDBC in certain parts of my code ... what impact may that have in hibernate's cache or other objects?

How hard is it to program a hibernate dialect (or some other artifact) to bypass some of hibernate's functionality in exchange of performance?

Thank you for reading and maybe answering, i hope my data is useful to you.

Any pointers or suggestions greatly appreciated.

Best regards,

Federico Vela

did you read the docs and wiki about batch operations?

and did you apply those advice ?

I have found that you can comfortably use JDBC alongside Hibernate.

Also see the docs on Session's createSQLQuery() method and the Query class. It is probably better to use those than 'raw' JDBC.

Our unit tests have sometimes shown that updates can conflict with items that are already loaded into the session cache. To overcome this you can clear or evict the objects from the session (See the Session docs).

Grant, thank you for your reply,

Your words are full of wisdom, thank you for them, I will try this and post the new hibernate results.

Max, thank you, i found the link for the batch docs:
http://www.hibernate.org/hib_docs/v3/re ... batch.html

I actually developed on my own part of this document, starting from the naive part ;-)

sorry for the newbiness but how do i award ratings to a post?

Best regards,
FV

That makes a lot of sense!

I hope i will only have to come down away from hibernate in those areas where the performance is of utmost importance.

Right now that only involves insert operations.

I also think that to be able to de-synchronize this process could help a lot, say split the text file into a few threads and have each of them insert.

Am i correct if i say that jdbc inserts have no concurrency issues on the db end?

As long as i'm not pulling that specific file's info while inserting i should be safe from "phantom" rows.

Thanks again to all.

Sounds like what i'm looking for.

Thanks all.