-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 8 posts ] 
Author Message
 Post subject: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Fri Mar 04, 2016 2:06 am 
Newbie

Joined: Fri Mar 04, 2016 1:59 am
Posts: 4
Code:
    public static void main(String[] args) {
        // memory close to -Xmx value
        List<EntityManagerFactory> emfs = IntStream.range(0, 50).mapToObj(
                n -> Persistence.createEntityManagerFactory("store",
                        ImmutableMap.of(
                                "javax.persistence.jdbc.driver", "net.sourceforge.jtds.jdbc.Driver",
                                "javax.persistence.jdbc.url", "jdbc:jtds:sqlserver://xxx:1433/xxx",
                                "javax.persistence.jdbc.user", "xxx",
                                "javax.persistence.jdbc.password", "xxx"
                        ))
        ).collect(toList());
        // ~1gib of memory.
        emfs.forEach(EntityManagerFactory::close);   
        try { Thread.sleep(60000); } catch (InterruptedException e) {}
        // *STILL* ~1gib of memory.
}


Some notes:
- persistence.xml is stored in a jar that is rather large (20mib), I have artificially increased size to make the problem surface more easy.
- if run exploded (no jar) then issue does not exist
- the memory usage quoted above is the process memory.

Current theory is that hibernate is unpacking the jar each time it creates an EntityManagerFactory and holding onto the contents to create a memory leak.

At this stage only option is to test if other ORM has same issue.


Last edited by bengeorge on Fri Mar 04, 2016 7:49 am, edited 3 times in total.

Top
 Profile  
 
 Post subject: Re: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Fri Mar 04, 2016 2:51 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
Why do you want 50 EntityManagerFactory in a single application node?
An EntityManagerFactory is a singleton that's responsible for interacting with one DataSource.

In your example, you assume that you application will interact with 50 databases at once.
Most apps require using one or two databases, even when using replication, so you need one or two EntityManagerFactory, which can spawn multiple short-lived EntityManager objects.

You need to force a GC too. What if, the GC hasn't run because it can hold 1GB of memory?


Top
 Profile  
 
 Post subject: Re: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Fri Mar 04, 2016 7:44 am 
Newbie

Joined: Fri Mar 04, 2016 1:59 am
Posts: 4
'Why' is a moot point, but fwiw we need to connect to +150 databases. Even with a SINGLE EntityManagerFactory the memory leak most probably exists, just that most applications would not notice it cause they have only 1.

Will try manually calling GC as you suggested, however jstat -gcold indicates that there are enough GC's to clean up if it could (note the sleep for 1 minute after closing all EMF's). Also heap is fine, its the memory out of the heap that is being consumed.


Top
 Profile  
 
 Post subject: Re: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Fri Mar 04, 2016 10:45 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
But Hibernate doesn't allocate memory outside of the heap. I suspect you are talking about Metaspace.

So quoting that SO answer:

Quote:
By default, the Metaspace memory is collected if it reaches the MaxMetaspaceSize. This parameter is initially unlimited. The limit is the memory in your machine. But the memory is automatically freed when a class and class loader is no longer needed. You only need to tune this parameter if you suspect that the ClassLoader has a memory leak.


Try adjusting the "MaxMetaspaceSize" property and see if the VM size is lowered after closing the EMF.


Top
 Profile  
 
 Post subject: Re: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Sun Mar 06, 2016 11:38 pm 
Newbie

Joined: Fri Mar 04, 2016 1:59 am
Posts: 4
Thanks for your advice, though even with the following settings: -Xms256m -Xmx256m -XX:MaxMetaspaceSize=50m, I can see via top or ps that the process is taking around over 1G of memory. Via jvisualvm or jstat I can see usage is as expected - heap is around 270m and meta space is like 14m.

Time to try mybatis.


Top
 Profile  
 
 Post subject: Re: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Mon Mar 07, 2016 1:27 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
I tried to replicate your test case as follows:

Code:
public static void main(String[] args) {
    // memory close to -Xmx value
    Map<String, String> properties = new HashMap<>();
    List<EntityManagerFactory> emfs = IntStream.range(0, 50).mapToObj(
            n -> Persistence.createEntityManagerFactory("templatePU", properties)
    ).collect(toList());
    // ~1gib of memory.
    emfs.forEach(EntityManagerFactory::close);
    System.gc();
    try { Thread.sleep(60000); } catch (InterruptedException e) {}
    System.gc();
    System.out.println("Done");
}


Using the following persistence.xml file:

Code:
<persistence xmlns="http://xmlns.jcp.org/xml/ns/persistence"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/persistence
             http://xmlns.jcp.org/xml/ns/persistence/persistence_2_1.xsd"
             version="2.1">

    <persistence-unit name="templatePU" transaction-type="RESOURCE_LOCAL">

        <description>Hibernate test case template Persistence Unit</description>
        <provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>

        <exclude-unlisted-classes>false</exclude-unlisted-classes>

        <properties>
            <property name="hibernate.archive.autodetection" value="class, hbm"/>

            <property name="hibernate.dialect" value="org.hibernate.dialect.SQLServer2012Dialect"/>
<!--            <property name="hibernate.connection.driver_class" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
            <property name="hibernate.connection.url" value="jdbc:sqlserver://localhost;instance=SQLEXPRESS;databaseName=high_performance_java_persistence"/>-->

            <property name="hibernate.connection.driver_class" value="net.sourceforge.jtds.jdbc.Driver"/>
            <property name="hibernate.connection.url" value="jdbc:jtds:sqlserver://127.0.0.1:1433/high_performance_java_persistence;instance=SQLEXPRESS"/>

            <property name="hibernate.connection.username" value="sa"/>
            <property name="hibernate.connection.password" value="adm1n"/>

            <property name="hibernate.connection.pool_size" value="5"/>

            <property name="hibernate.show_sql" value="true"/>
            <property name="hibernate.format_sql" value="true"/>
            <property name="hibernate.hbm2ddl.auto" value="create-drop"/>

            <property name="hibernate.max_fetch_depth" value="5"/>

            <property name="hibernate.cache.region_prefix" value="hibernate.test"/>
            <property name="hibernate.cache.region.factory_class"
                      value="org.hibernate.testing.cache.CachingRegionFactory"/>

            <!--NOTE: hibernate.jdbc.batch_versioned_data should be set to false when testing with Oracle-->
            <property name="hibernate.jdbc.batch_versioned_data" value="true"/>

            <property name="javax.persistence.validation.mode" value="NONE"/>
            <property name="hibernate.service.allow_crawling" value="false"/>
            <property name="hibernate.session.events.log" value="true"/>
        </properties>

    </persistence-unit>
</persistence>


And this is the memory profile:

Image

So, the used Heap is 30 Mb.

I'm not sure what Java env did you use or what Hibernate version, but with Hibernate 5.0.7.Final it looks fine to me.


Top
 Profile  
 
 Post subject: Re: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Tue Mar 08, 2016 1:46 am 
Newbie

Joined: Fri Mar 04, 2016 1:59 am
Posts: 4
Thanks for trying to replicate. I can replicate on multiple machines, though they all run ubuntu, 1.8.0_74 and running hibernate 5.0.7.Final.

Remember:
- The persistence xml has to be in a jar file, and that jar file is a dependency of the running application. If run exploded (ie: from within ide) it will work just fine.
- The jar file has to be large (15mb should be enough, but larger the quicker you will see the increase)
- The heap and meta space are perfectly fine. It is total memory consumed by the java process that it ridiculously large. ie: at the OS level.

How I solved the problem - used mybatis and memory is Xmx + a few extra mb (as expected).


Top
 Profile  
 
 Post subject: Re: Not sure if bug or not... *massive* memory leak issue.
PostPosted: Tue Mar 08, 2016 3:49 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
If you already have a test acse for this issue, would you mind sharing it? This way, we can also open a JIRA issue for this.


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 8 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.