I've been working on a system with a large and complex entity model (using 3 sessionfactories/databases etc.) that is used with a web application. I have now reached a point where I need to implement a number of "long running" batch processes, and am wondering what the best approach is. These batch processes involve iterating over a lot of records and traversing many entity associations while producing an output - one instance is producing letters to be printed, the other is carrying out calculations to produce invoices and invoice lines, and there are others as well

Each individual action as part of the batch (i.e. each iteration) is not too long, but the entire batch process could be - up to a number of minutes. I'm concerned about locking of tables impacting on web users.

The entire process doesn't need to be transactional (although in the case of invoicing, this would be nice).

Is the best approach to create a session at the start of the batch process, and then put each iteration of the loop into it's own transaction, beginning and committing the transaction at the start and end of the loop, using the same session for the whole batch? Or should I also open and close the session at the start and end of each iteration? (based on what I've read in the NHibernate manual on page 154, below)

Any advice would be appreciated.

2 of the 3 batch processes I'm writing could be contained to overnight processing, where locking is not such an issue, but the third runs numerous times throughout the day, and could impact web users.

Would another option be detached objects? This would not be preferable, gievn the potential size of the data needed to process the entire batch

anyone? any tips?

Your post is very open ended, and doesn't seem to lend itself to a quick answer.

I would not suggest using a single session for the whole batch. Session efficiency tends to degrade with size, because of dirty-object checking and God-knows-what-else that goes on inside there.

I use NH for some batch processes. First, I'll generate a single session and then "pilfer" its connection for the rest of the sessions (there is probably a way to do this without generating the session, but it works). From that connection I'll build a Session for the initial load (might be 1000s of graphed objects, so that query needs tuning for production). Then I close the session.

In the processing loop, I generate a session per iteration (using the same connection!). This may require locking the objects of interest for that iteration to the new session first. This usually only involves 10s (or fewer) objects, so the load on these sessions is much less than if one session were used for the whole process. This also fits well with our transactional needs.

YMMV. This also may be a monumentally stupid way of going about this, but at least I'm starting a conversation :)

Thanks very much for the reply. I'm glad you decided to start the conversation, as I was starting to lose hope of a reply. Sorry the initial post was a bit open ended, but I wasn't too sure what to ask. I've created the whole site using a particular architecture, and then wrote the batch processes using some similar concepts, and realised that it may be totally inappropriate. When I started looking around for recommendations for batch processing, apart from the couple of short points in the pdf doc, I was unable to find much. I don't have a great deal of time on this project at this stage to allow for a lot of testing different ideas, or for performance testing, so thought I'd see how others were doing it.

My main concern is how the running of the batch may affect other live users. The batch process won't be very long running (< 5 mins), and only runs 5 times throughout the day. the process itself actually runs on a grunty box, so the overall performance of the batch job itself is not a huge issue (although I'd like to optimise it as much as is easily possible). I guess I'm more concerned about locks etc. being caused in the database while the process is running, preventing web users from working.

I see what you mean about the session growing during the execution of the batch process, and using the same connection throughout the process.

I'll be looking at the batch jobs tomorrow, and shall take your suggested approach into account while reworking the job I've already written.

When executing the processing loop, I'm assuming if any of your objects get updated/created, you just use SaveOrUpdate with whatever session you're using at the time to persist the changes?

For the initial load of your "1000s of graphed objects", do you have separate NHibernate queries that retrieve objects using joins? i.e. retrieve the object hierarchy all at once (i.e. non lazily)?

It sounds like generally speaking the batch process I'm working on is quite similar to yours, so if it works for you, hopefully it will work for me :)

Thanks again.

Not sure how good idea it is, why dont u try to simulate ur batch with dummy data....basically a proof of concept, that shall give you more confidence.....think @ it.

Sudhir

I guess it depends on how much you need to shield the records in question from changes while the batch is running. It sounds like in your case, you may want to just load the identifiers for the records you need to act on. Then in the "iteration" section of your batch, load the data with each new transaction. That should minimize collisions.



Yes.



Exactly.