I am connecting hibernate with as400 database. There are so many records. I have used getNamedQuery() and in 1 go, it will convert each tuples of the database in object(s).. but, wat if there are 1 million tuples.. then this wont work. I have gone through various books, and I foung setMaxRecords() and know how to scroll through the records. But, this is also not the solution. Can u plz provide me with the solution as soon as possible.

Need help with Hibernate? Read this first:
http://www.hibernate.org/ForumMailingli ... AskForHelp

Hibernate version:

Mapping documents:

Code between sessionFactory.openSession() and session.close():

Full stack trace of any exception that occurs:

Name and version of the database you are using:

The generated SQL (show_sql=true):

Debug level Hibernate log excerpt:

What are you trying to do ? Please explain in more detail so we can understand your use case a little better.

i have written a select query in hbm.xml and teh statement that will call that query is

generic_get_query = session.getNamedQuery(classname+"Query_Get_Generic");

generic_get_query.setProperties(entity);
result = generic_get_query.list();

now, in this result(which is a list), I will get the collection of objects, but if there are 1 million tuples in database, then each of them will be first converted into an object.. which will affect performance and may be it will come as out of memory exception.

Yes an out of memory exception is likely with 1 mil objects in the session cache. What is important here is to determine why you want to read upto 1 mil records in a single select statement? Is it because you have some processing to do on each one? If so what do you want to do?

Not knowing the above - options include:
1) Bulk update features.
2) Read only session (but then again it might not be the best option).
3) Batch processing and managing the session cache.
4) direct JDBC access (get the connection from the Session).

The manual has details for the above approaches in various locations.

I have a similar problem and actually I am still reading the docs and thinking of a strategy to optimise my application.
My software has to read thousands of records from a DB, process them and then store the results as thousands of records into another DB.
As I needed immediate results I tried a homemade optimisation strategy by caching IDs of the source objects into an array made up by cloned ID values and then loading set of source objects through session.load(class,id) method, having them stored into an array in local memory.
Moreover I clear and flush the session after each insert into the target DB, but memory allocation does not seem to decrese, maybe I have to tune GC better and avoid to use iterator and use arrays instead; I'll investgate better on this.

However here is my question:
I need to cache more than only the ID of the source objects and I would like to load only few fields of these objects into memory in order to keep them cloned inside a local memory cached made up always by cloned fields values.
Is this possible?

Many thanks.

I can see you have a handle on the memory issues etc. In your case you have two sessions to manage during the processing. You could read multiple fields as you requiire using projection (into a new class) which does not load into the cache. In reality Hibernate is not best suited to bulk work like this which is why we have provided way to move outside of the API such as direct JDBC, native Queries etc.

Many thanks,
I'll look for the keywords you suggested.

I succeded, one way or another :-P, to have it work and with good performance.

Please check hibernate docs chapter 13 for more accurate infos.

I added the following to the main configuration file


This improved a lot the speed while bulk inserting thousands of records into one of the two opened connections. I flush and clear the session that takes care of writing into the target db every 100 records.
I am using saveOrUpdate to insert newly created objects instances owning sets of children objects.

About the "bulk reading" I eliminated all the iterators and used arrays instead and created a home made caching system in which I load all the IDs I need, then loading blocks of 100 records and then writing them into the target DB.
I hope this can be helpful.