Hi all,

I have a requirement like this.

I need to process a large set of jobs (millions) at the end of the month/year. I can batch these jobs into some fixed size, say 1000, and process each batch separately. Each job is independent of other jobs. Each job requires a complex set of activities to complete its processing. If one of the job fails, I should be able to either,

1. Rollback the transactions within the failed job, mark it as failed and proceed with the rest of the jobs in the batch and Commit. I shall work on the failed job later.

2. Commit whatever jobs has been processed succesfully till then, Rollback the failed job and mark it as failed. I shall re-process the rest of the jobs in that batch later.

I have a few options to do this.

1. Do the whole operation in one Transactional context. This is not desireable considering that there are chances of a few jobs failing in each batch, which will result in rollback of all previously successful jobs in that batch.

2. Commit the transaction after each job. (Not sure how to do this when managing transactions through Spring config with Hibernate) Also, I feel it may have a negative impact on performance, considering the volumes I need to process and complexity of each job.

3. Commit after processing some fixed number of jobs, say 100, in each batch. This can minimize the reprocessing to some extend. Again I am not sure how to do it while using declarative transactions.

4. Use Savepoints and Rollback to to the last Savepoint when a job fails. This might give better performance than Option#3. Not sure if this works in my environment.

5. Use Nested Transactions. I can independently manage transactions within each job and decide to Rollback if it fails, but still continue with the rest of the jobs. Not sure if this can be implemented at all.

Please let me know which is the best approach and how to do this given that we have Spring 1.2.7, Hibernate 3.1, Oracle 10g and Weblogic 8.15. We use Spring for managing transactions declaratively.

Thanks in Advance.

Best Regards,
Appu.

Any opinions on this?