Hello,

I am trying to save a huge number of Objets using Hibernate.

My program relies on one transaction. I retrieved from database all the references I need to
attach to the future objects to persist.

I defined the property hibernate.jdbc.batch_size in my config file.

Here is my current method to insert transient objects :

public void persistBusinessObj(List<BusinessObj> boList) throws DaoException {

int i = 0;
for (BusinessObj current : boList) {

i++;
save(current);
if (i % 1000 == 0) { // 1000, same as the JDBC batch size property value

// flush a batch of inserts and release memory
entityManager.flush();
entityManager.clear();
}
}
}

The boList collection has references to previously retrieved objects from database and after the first call at entityManager.clear(); I logically lose all my references since the persistent context
has been cleared. I get Could not synchronize database state with session.

So I don't know how to proceed. I cannot again get from database all the referential data needed
to persist the next batch of objects.

I could retrieve the id of the newly attached objects and call evict on them specifically instead of calling entityManager.clear() but I think this operation has a heavy cost in time.

I need some opinion from expert.

Thank you.

Kriss

You probably do not want to do that.

A huge transaction places a considerable burden on any database engine.
Also, if you're worried about working with outdated references, any process that may invalidate a reference will have to wait for your transaction to complete.
Also, if the transaction is running for more than a few minutes, you risk that it is aborted by factors outside your control, such as hardware faults, power outages, or the database admin shooting down your transaction because it's blocking another process. You don't want your work rolled back, you want the productive work committed and another run restarted that finishes the remaining records.

I tend to design the data model so that you can directly read off each record whether it needs to be processed or not. That way, I can do single-record transactions: get the record, do whatever the process needs to do with it, update the status, finish the transaction.
Hibernate will support you there in that it will collect all updates and write them out in a single batch at the end of the transaction.

The status can be anything. You talked about creating dependent records, so work only on those master table records where the dependent records don't exist yet.
In other situations, I had records that needed to be upgraded once per month, so the nightly batch run will touch only those records that have a "last processed month" field that's older than the current month.
Filtering out the already-done records can be made efficient by having the proper index in place and using a WHERE condition that will filter them out (via NOT EXISTS if it's "needs dependent records added", or via "last_modified is too old" if it's "needs to be processed once per day").

Thank you for your answer. I will study a new way to process.

Kriss.