I've tried to find this in the NHibernate documentation but i haven't. The Wilson ORM has a feature where you can get something called an ObjectReader. This is analogous to a SqlDataReader in ADO.NET, except that it gives you your buz object one at a time instead of a record. So if you are processing 20,000 records, rather than getting a collection of 20,000 objects, it lets you get one object at a time, process it and then ask for the next one. When you get the next one, the previous object is destroyed (goes out of scope.) It reads the records from the database one by one to populate the business objects using a (Sql)DataReader with an open cursor to the database.

This is useful in certain situations where having the entire recordset or object collection in memory is not performant.

Does NHibernate have such a feature?

Hi fregas,

I'm not sure if it does have this ability but i would certainly welcome it. It would definitely be useful to be able to stream objects through memory so that you can process large batches of objects without having to load them all into memory first.

cheers,
Paul.

instead of calling the List or List<T> method of a query, you can call the Enumerable or Enumerable<T> method which should do exactly what you want

Honestly, I think you guys may be on the wrong track. If your goal is performance, I think you may run in to some big issues with the given approach. Making 20,000 database calls to load one object at a time will certainly hurt performance.

First I would say that 20,000 records really isn't that many. If you need to load all 20,000, lets me pessimistic and say that your objects are 1K in size, you are talking about 20 megs worth of data. This should not be a problem for most systems. Your access to the memory would be far easier than going to the database for every object. The performance should be far faster loading 20,000 in to memory.

One pitfall to this argument is that NHibernate does not seem to scale linearly when it comes to flushing dirty objects. Dirty checks seem to take longer than a linear time scale. That being said the Enumerable has the same issue. The objects are not going out of memory when you loop to the next one. The object will still be in scope of the session, which means you would still have a reference to the object, not allowing garbage collection to free the memory and still requires the dirty check on flush. You wouldn't be saving any memory anyway. So in both cases when you are finished with objects you would need to evict them from the session to save on dirty checking and memory cost.

Also, take a look at some other searching functions in NHibernate such as SetFirstResult() and SetMaxResults() which let you specify which range of the query result you want returned. This would let you run fewer queries but still batch your inputs. There are some things to watch out for, but this would perform far better than the enumerable method.

Guys,

I thought that calling Enumerable will just make a collection, in which case 20k records will still be in memory, like DavyBrion mentioned. If it doesn't but the object is still in session, is there a way to remove it from session?

Also, when using a SqlDataReader, or Wilson's ObjectReader, its NOT 20,000 queries. Its one query, but the result set is streamed in one record at a time, using the database's cursor. This is more performant in certain situations.

In the case of the ObjectReader, i think he actually forces each object out of memory by removing all instances or calling garbage collection or something. I am not using NHibernate right now but i just wanted to see if it had a similar feature.

Thanks,
Craig

Enumerable is close, but not quite. THe problem is that even if you return the results in a stream, only advancing through the datareader one object at a time, the session still maintains hard references to each object. Even if it falls out of scope, the session still keeps a reference to not only it, but it's data.

For such a scheme to work, the session would probably need to be updated to only keep a weak reference to the object, thus allowing the GC to clean it up when it falls out of scope.

This is an interesting idea, because using weak references actually seems like it might in some fashion be a useful feature all around.

I guess the basic idea would be that objects returned using Enumerable() would be added to the session using a weak reference link instead. The finalizer on the weakreference'd link could add the object's identity to a list on the session (because this happens on the GC thread). The session would then need to examine that list and clean up it's internal structures that relate to the discarded object. This could be done in the enumerator attached to Enumerable() and maybe Flush().

Sounds hard.