Fast add to large collection

ghay · **Posted:** Mon Jun 04, 2007 4:19 pm

I am new (of course) to NHibernate and have been able to get it working with some basic satisfaction. However, when I move to the next level of performance testing I'm having some frustration.

I use the session per unit of work pattern (method scoping). The problem occurs with this type of code

Code:

        public override void Save(IMessageContext message)
        {
            try
            {
                if (log.IsDebugEnabled)
                    log.Debug("Starting Save");
                using (ISession s = _sessionFactory.OpenSession())
                {
                    using (ITransaction t = s.BeginTransaction())
                    {
                        try
                        {
                            t.Begin();
                            Partition p = s.Get<Partition>(_partitionId);
                            p.Messages.Add(message);
                            t.Commit();
                        }
                        catch
                        {
                            t.Rollback();
                            throw;
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                if (log.IsErrorEnabled)
                    log.Error("Error encountered in Save", ex);
                throw;
            }
            finally
            {
                if (log.IsDebugEnabled)
                    log.Debug("Finished Save");
            }
        }

This is fine if there are only a few messages in the "Partition" object, but as this is a very frequent operation (thousands of messages an hour) it very quickly bogs down as it seems to load the entire collection into memory when I call p.Messages.Add(...)

I don't want to keep p in memory (with a long lived session) as there are potentially hundreds of thousands of messages in the collection, and each message is potentially very heavy (image data, etc contained within it).

This is not an ASP.NET application. It is a Windows Service application.

What is the appropriate pattern for such a situation?

Thanks

ghay · **Posted:** Tue Jun 19, 2007 8:59 am

I have had to abandon NHibernate for my initial application due to performance issues. It turns out that every time I use s.Get<Partition>(id); it requeries the database for all of the records that I've stored. When the volume is > 500 records, the performance is > 20ms per message added to the collection. Compared with < 1 ms when used against a ADO.Net sql stored procedure.

Its too bad as I liked what I saw, but I couldn't get the performance I needed. I'm sure I missed something, but deadlines loom :)

baggins · **Joined:** Mon May 16, 2005 1:35 am **Posts:** 67

You need to make sure you mapped your Messages collection on the Partition class as lazy="true". You should be able to add to the collection without causing all items in the collection to be loaded so long as you use the <bag> collection mapping.

rolandz · **Posted:** Mon Jun 25, 2007 1:58 am

ghay wrote:

[...] This is fine if there are only a few messages in the "Partition" object, but as this is a very frequent operation (thousands of messages an hour) it very quickly bogs down as it seems to load the entire collection into memory when I call p.Messages.Add(...)

I don't want to keep p in memory (with a long lived session) as there are potentially hundreds of thousands of messages in the collection, and each message is potentially very heavy (image data, etc contained within it).

Maybe you should change the data domain handling so the property of type Partition would be a property of a Message and you simply assign a message to the instance of partition. No large collections are being hendled in such scenario.

I am often using this aproach whenever I see that I wil have a potentially large collection with parent/child relationship.

ghay · **Posted:** Tue Jun 26, 2007 10:18 am

Thanks, I'll most definitely give that a try