-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 3 posts ] 
Author Message
 Post subject: OutOfMemory error in loading data to MySql database
PostPosted: Wed Mar 16, 2016 2:29 am 
Newbie

Joined: Tue Mar 15, 2016 8:07 pm
Posts: 2
Hi, I am running a Web application in JBoss + JPA + Hibernate. I configured the application to automatically load a data file to MaraiDB by Hibernate:
LOAD DATA LOCAL INFILE '/home/marlon/ubuntu/src/main/resources/Context.txt' INTO TABLE Context FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' (id,term_id,source,context,URL,publish_time);
The plain text file is around 180MB, and I have been able to load it this way when the web application is started. But yesterday I suddenly received this memory error. I can load the data using the command at MariaDB terminal. The issue may be coming from Hibernate, JBoss or my application, but hope I can get some comments or insights here.

My Java heap size is set as 4GB which should be more than enough to handle this data. Please pay attention to this:

Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.mariadb.jdbc.internal.stream.PacketOutputStream.increase(PacketOutputStream.java:45)
at org.mariadb.jdbc.internal.stream.PacketOutputStream.write(PacketOutputStream.java:223)
-------------------------------
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /home/marlon/ubuntu/dump/java_pid12245.hprof ...
Heap dump file created [227481716 bytes in 1.047 secs]
15:42:00,971 WARN [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (ServerService Thread Pool – 31) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@2f63260c[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@182f8ae2 connection handles=1 lastReturned=1458081711109 lastValidated=1458081711098 lastCheckedOut=1458081711252 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@32bb873d mcp=SemaphoreArrayListManagedConnectionPool@50935882[pool=BombwordDS] xaResource=LocalXAResourceImpl@34d31a8a[connectionListener=2f63260c connectionManager=504ffe68 warned=false currentXid=null productName=MySQL productVersion=10.1.12-MariaDB-1~trusty jndiName=java:/jboss/datasources/BombwordDS] txSync=null]: javax.resource.spi.ResourceAdapterInternalException: Unexpected error
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:688)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.connectionError(BaseWrapperManagedConnection.java:654)
at org.jboss.jca.adapters.jdbc.WrappedConnection.checkException(WrappedConnection.java:1934)
at org.jboss.jca.adapters.jdbc.WrappedStatement.checkException(WrappedStatement.java:1437)
at org.jboss.jca.adapters.jdbc.WrappedStatement.executeUpdate(WrappedStatement.java:418)
at org.hibernate.tool.hbm2ddl.DatabaseExporter.export(DatabaseExporter.java:64)
at org.hibernate.tool.hbm2ddl.SchemaExport.importScript(SchemaExport.java:447)
at org.hibernate.tool.hbm2ddl.SchemaExport.execute(SchemaExport.java:379)
at org.hibernate.tool.hbm2ddl.SchemaExport.create(SchemaExport.java:305)
at org.hibernate.tool.hbm2ddl.SchemaExport.create(SchemaExport.java:294)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:517)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1859)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:857)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:850)
at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.withTccl(ClassLoaderServiceImpl.java:425)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:849)
at org.jboss.as.jpa.hibernate4.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:154)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:117)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:665)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:182)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.mariadb.jdbc.internal.stream.PacketOutputStream.increase(PacketOutputStream.java:45)
at org.mariadb.jdbc.internal.stream.PacketOutputStream.write(PacketOutputStream.java:223)
at org.mariadb.jdbc.internal.stream.PacketOutputStream.sendFile(PacketOutputStream.java:108)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:453)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.result(AbstractQueryProtocol.java:399)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:381)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:337)
at org.mariadb.jdbc.MariaDbStatement.execute(MariaDbStatement.java:280)
at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:474)
at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:490)
at org.jboss.jca.adapters.jdbc.WrappedStatement.executeUpdate(WrappedStatement.java:414)
at org.hibernate.tool.hbm2ddl.DatabaseExporter.export(DatabaseExporter.java:64)
at org.hibernate.tool.hbm2ddl.SchemaExport.importScript(SchemaExport.java:447)
at org.hibernate.tool.hbm2ddl.SchemaExport.execute(SchemaExport.java:379)
at org.hibernate.tool.hbm2ddl.SchemaExport.create(SchemaExport.java:305)
at org.hibernate.tool.hbm2ddl.SchemaExport.create(SchemaExport.java:294)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:517)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1859)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:857)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:850)
at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.withTccl(ClassLoaderServiceImpl.java:425)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:849)
at org.jboss.as.jpa.hibernate4.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:154)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:117)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:665)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:182)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15:42:00,975 ERROR [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool – 31) HHH000388: Unsuccessful: LOAD DATA LOCAL INFILE '/home/marlon/ubuntu/src/main/resources/Context.txt' INTO TABLE Context FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' (id,term_id,source,context,URL,publish_time)
15:42:00,975 ERROR [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool – 31) Error
15:42:00,976 INFO [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool – 31) HHH000230: Schema export complete


Top
 Profile  
 
 Post subject: Re: OutOfMemory error in loading data to MySql database
PostPosted: Wed Mar 16, 2016 6:03 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
HBMDDL is fine for testing but I would not use that in a production environment.
For that, a tool like FlyewayDB is a much better alternative because it allows you to capture every DB change in git as well and you can reconstruct the database at any point in time.

From the stacktrace, I see that the exception is thrown here:

at org.jboss.jca.adapters.jdbc.WrappedStatement.executeUpdate(WrappedStatement.java:418)

So, it all happens in the MySQL driver. You could try to change the driver to see if there's a bug related to such an issue.


Top
 Profile  
 
 Post subject: Re: OutOfMemory error in loading data to MySql database
PostPosted: Wed Mar 16, 2016 11:20 am 
Newbie

Joined: Tue Mar 15, 2016 8:07 pm
Posts: 2
Yes. It's used for development. It seems somehow it is creating a big byte[] array to insert into the database. I suspect it's reading every line from the file into the bytebuffer and attempt tos insert at one time.


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 3 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.