Memory Usage Keeps Increasing

Budric · **Joined:** Fri Jun 16, 2006 3:41 pm **Posts:** 18

Hi, I'm importing a lot of information into the database and after I import a few objects Java runs out of heap space and the program quits. I can't see where this memory leak is comming from (I'm most likely not using the session correctly).

Here's pseudocode of what I have:

Code:

class DataExtractor
{
    private Object machineData;
    
    public static void main(...)
    {
        for (int i = 0; i < numFiles; i++)
        {
            DataExtractor record = new DataExtractor();
            record.initialize();
            record.save();
        }
    }
    public void save()
    {
        Session session = HibernateSessionFactory.getSessionFactory().openSession();
        try
        {
            Transaction transaction = session.beginTransaction();
            session.save(machineData);
            transaction.commit();
        }
        catch(Exception e)
        {
        }
        finally
        {
            session.close();
        }
    }
}

The initialize() function initializes the objects by parsing some files. There are many children and sets of stuff within the actual machineData. But after processing about 20 of these objects and adding them to the database I get an out of memory error. I don't see what is still holding the references to the objects.

By the way HibernateSessionFactory is a class that's a copy of HibernateUtil implementation in section 1.2.5 of Hibernate v3 documentation.

Thanks for any input.

tenwit · **Joined:** Thu Dec 23, 2004 9:08 pm **Posts:** 2008

It's unlikely to be hibernate, as there are much, much larger apps out there that aren't running out of heap space.

Try not ignoring the exception, there might be an issue there.

Also try not creating a new record each time. Create one outside of the loop, then inside the loop, save it, evict it, and clear its ID. Next time around, you'll save a new copy of the record. That loop should run fine; if it does, then the problem is in new DataExtractor or record.initialize(). If not, then post again, or even better, get a profiler and examine memory usage. You could even use the JDK's built-in one, it's good enough for basic analysis.

Budric · **Joined:** Fri Jun 16, 2006 3:41 pm **Posts:** 18

I wasn't calling evict() on the objects. Now I am but I'm still having the problems.

The reason I didn't suspect my code is because I expected everything to be deleted by the GC because I create a new DataExtractor at every iteration of the loop, and this is the only object that points to machineData. So at next iteration the old stuff should be deleted.

Also another question I have is how long does Hibernate usually take to update the data? The initialize() function parses roughly 8.5 megs of data into various objects that contain other objects and sets of other objects etc. This initialization takes roughly 9 seconds. Then the save operation takes about 2 minutes and all it has is the calls to hibernate. So does that sound resonable? Or do I have another mistake somewhere? I notice that the program isn't using the CPU much (13% according to task manager). Oh and I'm using SQL Server Express 2005 (which may be a problem..but maybe not).

tenwit · **Joined:** Thu Dec 23, 2004 9:08 pm **Posts:** 2008

There is no way to comment on the performance. I have no idea how much memory is available to the app, how fast your disc storage is, how busy the network between your application machine and your database machine is, etc. Assuming that nothing you've got is especially powerful or fast, getting 8Mb from the DB, converting the data into JDBC result sets, converting those into your classes, then doing the reverse process, 2 minutes still sounds a bit slow to me.

SQLServer Express is limited by design (it's free, after all), but afaik its not limited in speed, only in the number of simulatoneous connections. That's probably not the factor. For a Microsoft product, it's pretty good.

With regards to when the memory is freed: note that java makes no promises about when deallocation occurs. Once an object goes out of scope it is elligible for garbage collection, but it will almost certainly hang around for a bit. If java doesn't think a gc will help, then it will save the CPU cycles and postpone deallocation for a while. If your loop is run 10 times, that's maybe 190Mb of memory in use by the app (8.5Mb, 2 copies (result sets and your classes), 10 times) plus the JVM and library overheads.. if you've got a 1Gig machine with no other big apps running, then java might not bother running a gc at all.

Budric · **Joined:** Fri Jun 16, 2006 3:41 pm **Posts:** 18

I'm developing all of this on a desktop 3.2 Ghz with HT, lots of ram, and the database is installed localy. I'm using hibernate.cfg.xml that's identical to the one in the documentation section 1.2.3 (except for connection string, and turned off show_sql because then it takes like 6 minutes). I notice there's no second level cache...would that slow things down?

Another thing that's memory related is suppose I have two tables with one-to-many relationship. When I query the table on the "one" side of the relationship, will hibernate fill out every entry in the set? That would take up a lot of memory. This is my actual code. There's a 1:N relationship between MachineClass and MachineDataInstance (which then contains all of the info). I check if this record already exists, so I'm interested in some data but not all of it. How much data will Hibernate load when it has "lazy select fetching" turned on by default? Is there a better way of doing this?

Code:

        try
        {
            Object temp = session.createQuery("from MachineClass where Name = ?").setString(0, machineClass.Name).uniqueResult();
            if (temp == null)  //this class doesn't exist in the database so create everything
            {
                machineClass.MachineDataInstanceSet.add(machineDataInstance);
                session.save(machineClass);
            }
            else  //the info about the machine already exists, so import only the new instance 
            {
                machineClass = (MachineClass) temp;
                if (!machineClass.MachineDataInstanceSet.contains(machineDataInstance))
                {
                    machineClass.MachineDataInstanceSet.add(machineDataInstance);
                    session.save(machineClass);
                    resultID = machineClass.ID;
                }
            }            
            transaction.commit();
        }
        catch (Exception e)
        {
           System.out.println(e);
           e.printStackTrace();
           transaction.rollback();
        }
        finally
        {
            session.clear();
            session.close();
        }

tenwit · **Joined:** Thu Dec 23, 2004 9:08 pm **Posts:** 2008

The second level cache is most useful for a rarely-write situation. Seeing as you write to everything you load (in this app, anyway), the second level cache is not going to help. If anything, it'll slow things down and use up more memory.

Assuming default proxying, a set will be populated only when it is accessed. You are accessing the set (set.contains() is being called), so yes, it will be loaded. If it wasn't loaded, you couldn't tell if it contains your new instance.

That code (probably) has a bug. You're using the set member field directly. Assuming that you're using the default access mode of property, this is a bug. Use the property: machineClass.getMachineDataInstanceSet(). Do not refer to member fields of persistent objects, ever. The only methods that are allowed to refered a member field are the two property methods, get<field> and set<field>. If you're using access="field", then it'll work.

Budric · **Joined:** Fri Jun 16, 2006 3:41 pm **Posts:** 18

Hi,
my default-access="field". I got help on this forum on how to make hibernate access fields directly. So far the code works. It's just slow and takes up a lot of memory...I have to set -Xmx 500m for the program to finish and not crash (although 500 may be too much, I just set it to high).