Hi all,

I am a big fan of Hibernate and I have suggested in my project that we use hibernate for a batch process. I wasnt sure, but i was keen that hibernate would ease life. And it did for a while.

However, problems started with increase in data volume. Allow me to explin in detail.

The app is a Batch process invoked on Unix box as a java app (main class)
The app uses a Single long lived session used by the single instance of dao.
We have three classes mapped as shown below.

we get an xml from different sources that we process to make that tree.
now, initially (total object count ~600), we used to create the tree and save the Volume object. and everything was fine.
but when the tree size grew (total object count ~2500), we had to save the children first and then the parents because the process was taking too long to know if anything was happening at all.
Then came a BIG XML file with a total object count of 50k+. Now the session was going too fat and was take close to 3 secs for each insert after 3000 records.

I read somewhere that session.clear will remove all objects from session. make it light.

I wrote a dao interface method to clear session, but the application has to call it after every 1000 records manually. this is not elegant.

There is another problem too.. When I try to get something like



it takes more than 90 secs to load the iterator. I did give the batch size it seems to be ignored.

I have removed cascade altogether, but the effect may not be acceptable.

Here are my questions:

I know my implementation needs to be fine tuned.
1. for the save /update part, There should be an elegant way to limit the size of the session and remove the old objects automatically. or some pattern i could do to help solve it more elegantly.
2. loading the iterator may be improved by some method?

Hibernate version:3

Mapping documents:

Code:

<class name="Volume" table="VOL">
      <id name="id" column="ID" type="long">
         <generator class="sequence" >
            <param name="sequence">VOL_SEQ</param>
         </generator>
      </id>
      
      <bag name="FOL" cascade="all" inverse="true">
         <key column="VOL_ID" />
         <one-to-many class="Folder"/>
      </bag>
      ...
      ...
      ...
</class>

<class name="Folder" table="FOL">
      <id name="id" column="ID" type="long">
         <generator class="sequence" >
            <param name="sequence">FOL_SEQ</param>
         </generator>
      </id>
      
      <bag name="P3" cascade="all" inverse="true"  batch-size="200">
         <key column="FOL_ID" />
         <one-to-many class="File"/>
      </bag>
      ...
      ...
      ...
</class>

<class name="File" table="FIL">
      <id name="id" column="ID" type="long">
         <generator class="sequence" >
            <param name="sequence">FIL_SEQ</param>
         </generator>
      </id>
      
      ...
      ...
      ...
</class>

   

Name and version of the database you are using:Oracle 10G

150 views and no reply? :(

Hi sarath_it,

Give us time taken by each layer ,SQL Execution Time ,SQL and Table .We will see what the best we could do .

But generally I don’t think that 50K record size in problem.

Here is a sample test code.




in the first for loop I am trying to insert 50000 records.

the loop outputs something like

0 126
1 64
2 69
3 61
...
...
...
4500 2106
4501 2213
4502 2249
...
...


The second loop out puts.
0 6933
Clearing Session cache
1 47
2 45
3 49
4 42
..
...

So the getFiles() is returning 50,000 records?

Here's a few things you can try:

-Set FlushMode.MANUAL if you're doing any lookups (i.e. I don't know what updateStatus does)
-Rather than call the getFiles() on the object. Execute a query using a scrollable result set to get the files. That way you won't immediately load 50,000 objects into your session.
-As you're scrolling through the result set do your data manipulation and every x records flush and CLEAR your session. I don't know what the flushDBResources() is doing but if you are simply flushing your session you're going to run into issues.

What is most likely happening is that you're loading a bunch of data into your session and then every time hibernate decides that it needs to flush it goes through every loaded entity and checks to see if the entity is dirty. That tends to chew a load of time as your session grows.

Alternatively you can try using a StatelessSession that doesn't do automatic dirty checking but you lose some features (Interceptors, event model, etc) I believe.