Hibernate Community • View topic - Loading large arrays

View unanswered posts | View active topics

Board index » Hibernate & Java Persistence » Hibernate Users

All times are UTC - 5 hours [ DST ]

Loading large arrays

Page 1 of 1

[ 3 posts ]

Previous topic | Next topic

Author

Message

greg_barton

Post subject: Loading large arrays

Posted: Wed Apr 07, 2004 6:56 pm

Pro

Joined: Tue Aug 26, 2003 1:24 pm
Posts: 213
Location: Richardson, TX

I've got an application where I need to load large (1/2 million elements) arrays of bytes. I'm currently using the primitive-array mapping, and that works just fine, but for some reason it's taking huge amounts of memory to load the array. (A 1/2 million element array in an object (1/2 MB in size, since they're all bytes) is requiring 96MB of memory to load. Yes. 96.)

Is there any better way to map this? The array elements are stored in a seperate from the main object table, and I can't get around that. Also, I usually have to load the entire array, repeatedly.

Here's the mapping:

Code:

<class name="org.zographos.tilepile.data.MuralData" table="murals">
         
    <cache usage="read-only"/>
     
    <id name="id" type="int">
        <generator class="native"/>
    </id>
         
    <version name="version" unsaved-value="null"/>
         
    <property name="name" type="string" not-null="true" unique="true">
        <meta attribute="use-in-tostring">true</meta>
    </property>

    <primitive-array name="tiles" batch-size="100" outer-join="false">
        <cache usage="read-only"/>
        <key column="mural"/>
        <index column="idx"/>
        <element column="color" type="byte" not-null="true"/>
    </primitive-array>

</class>

It takes 96MB to do the following query:

Code:

session.createCriteria(MuralData.class)
    .add(Expression.eq("name", muralName)).uniqueResult();

Any suggestions for reducing memory usage?

Top

gavin

Post subject:

Posted: Wed Apr 07, 2004 7:50 pm

Hibernate Team

Joined: Tue Aug 26, 2003 12:50 pm
Posts: 5130
Location: Melbourne, Australia

Hibernate should not require so much memory usage. You sure it is not your JDBC driver creating a lot of garbage?

Top

greg_barton

Post subject:

Posted: Thu Apr 08, 2004 1:23 pm

Pro

Joined: Tue Aug 26, 2003 1:24 pm
Posts: 213
Location: Richardson, TX

Quote:

You sure it is not your JDBC driver creating a lot of garbage?

Yep. The following code runs fine, though it requires 64MB on the largest query I run. With Hibernate the memory requirements are 96MB. (I'm using mysql, and have tested both the 3.0 and 3.1 Connector/J JDBC drivers.)

Code:

java.sql.Connection con =
    org.zographos.tilepile.data.access.HibernateTools.getSession()
                                                     .connection();
java.sql.Statement st = con.createStatement();
java.sql.ResultSet set =
    st.executeQuery("select tiles0_.color as color__, tiles0_.mural as mural__, tiles0_.idx as idx__ from tiles tiles0_ where tiles0_.mural=" +
                    muralId
                   );

int count = 0;
set.first();

while(!set.isLast()) {
    count++;
    set.next();
}
System.out.println("Tile count: " + count);

The query text is the query Hibernate generates.

So it seems that the JDBC driver is using the bulk of the memory, though Hibernate's usage is significant as well. I'd also note that if you eliminate "tiles0_.mural as mural__" from the query the memory requirements go down to 48MB for the JDBC. Is it necessary to retrieve that field for this mapping? (The only necessary fields are those involved in the index and element. Why retrieve the key column?)

Top

Page 1 of 1

[ 3 posts ]

Board index » Hibernate & Java Persistence » Hibernate Users

All times are UTC - 5 hours [ DST ]

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum