I'm running out memory trying to batch 120,000 records from one database to another. I have a JDBC result set that I pull data from an oracle database and then I have Hibernate DAO methods that save the data into a MySQL database. The problem I'm having is that hibernate keeps eating memory until it exhausts the java heap and eventually generates a out of memory error. Flush and clear do not seem to have any affect on clearing memory. Below are my mappings and config.
I simply loop through the results and after every 16 saves I flush and clear the current session. If i run the code without insert and updates the memory problem is fine. As soon as I starting using hibernate the memory usage starts growing and is never cleared.
<bean id="jotm" class="org.springframework.transaction.jta.JotmFactoryBean">
<property name="defaultTimeout" value="7200" />
</bean>
<bean id="dataSource" class="org.enhydra.jdbc.pool.StandardXAPoolDataSource" destroy-method="shutdown" autowire="constructor">
<property name="dataSource">
<bean class="org.enhydra.jdbc.standard.StandardXADataSource" destroy-method="shutdown">
<property name="transactionManager" ref="jotm" />
<property name="driverName" value="${jdbc.driverClassName}" />
<property name="url" value="${jdbc.url}" />
<property name="user" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
</bean>
</property>
<property name="user" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<property name="maxSize" value="90"/>
<property name="minSize" value="1"/>
</bean>
<bean id="oracleDataSource" class="org.enhydra.jdbc.pool.StandardXAPoolDataSource" destroy-method="shutdown" autowire="constructor">
<property name="dataSource">
<bean class="org.enhydra.jdbc.standard.StandardXADataSource" destroy-method="shutdown">
<property name="transactionManager" ref="jotm" />
<property name="driverName" value="${oracle.jdbc.driverClassName}" />
<property name="url" value="${oracle.jdbc.url}" />
<property name="user" value="${oracle.jdbc.username}" />
<property name="password" value="${oracle.jdbc.password}" />
</bean>
</property>
<property name="user" value="${oracle.jdbc.username}" />
<property name="password" value="${oracle.jdbc.password}" />
<property name="maxSize" value="2"/>
<property name="minSize" value="1"/>
</bean>
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="jtaTransactionManager" ref="jotm" />
<property name="configurationClass" value="org.hibernate.cfg.AnnotationConfiguration"/>
<property name="configLocation" value="classpath:hibernate.cfg.xml"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">${hibernate.dialect}</prop>
<prop key="hibernate.cache.use_query_cache">false</prop>
<prop key="hibernate.show_sql">faalse</prop>
<prop key="hibernate.cache.use_second_level_cache">false</prop>
<prop key="hibernate.cache.provider_class">org.hibernate.cache.EhCacheProvider</prop>
<prop key="hibernate.jdbc.batch_size">16</prop>
<prop key="hibernate.transaction.manager_lookup_class">org.hibernate.transaction.JOTMTransactionManagerLookup</prop>
<prop key="hibernate.transaction.factory_class">org.hibernate.transaction.JTATransactionFactory</prop>
<!-- <prop key="hibernate.transaction.factory_class">org.hibernate.transaction.CMTTransactionFactory</prop> -->
<!-- Test Settings -->
<prop key="hibernate.current_session_context_class">jta</prop>
<prop key="hibernate.connection.release_mode">after_statement</prop>
<prop key="hibernate.transaction.flush_before_completion">true</prop>
<prop key="hibernate.transaction.auto_close_session">true</prop>
<prop key="hibernate.order_updates">true</prop>
<prop key="hibernate.order_inserts">true</prop>
</props>
</property>
</bean>
<bean id="txManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="userTransaction" ref="jotm" />
<property name="transactionManager" ref="jotm" />
</bean>
<tx:annotation-driven transaction-manager="txManager" proxy-target-class="true" />
Here is the standard out of the config settings.
- JOTM started with a local transaction factory which is not bound.
- CAROL initialization
- JOTM 2.0.10
- Using JTA UserTransaction: org.objectweb.jotm.Current@7cca20
- Using JTA TransactionManager: org.objectweb.jotm.Current@7cca20
- Hibernate Annotations 3.3.0.GA
- Hibernate 3.2.5
- loaded properties from resource hibernate.properties: {hibernate.bytecode.use_reflection_optimizer=true, hibernate.bytecode.provider=cglib}
- using bytecode reflection optimizer
- Bytecode provider name : cglib
- using JDK 1.4 java.sql.Timestamp handling
- configuring from url: jar:file:/usr/depot/apache-tomcat/current/webapps/finance/WEB-INF/lib/app.jar!/hibernate.cfg.xml
- Configured SessionFactory: null
- Binding entity from annotated class: ...
...
...
- Hibernate Validator not found: ignoring
- Building new Hibernate SessionFactory
- Hibernate Validator not found: ignoring
- Initializing connection provider: org.springframework.orm.hibernate3.LocalJtaDataSourceConnectionProvider
- RDBMS: MySQL, version: 5.0.22-standard-log
- JDBC driver: MySQL-AB JDBC Driver, version: mysql-connector-java-5.0.8 ( Revision: ${svn.Revision} )
- Using dialect: org.hibernate.dialect.MySQL5InnoDBDialect
- Transaction strategy: org.hibernate.transaction.JTATransactionFactory
- JNDI InitialContext properties:{}
- instantiating TransactionManagerLookup: org.hibernate.transaction.JOTMTransactionManagerLookup
- instantiated TransactionManagerLookup
- instantiating TransactionManagerLookup: org.hibernate.transaction.JOTMTransactionManagerLookup
- instantiated TransactionManagerLookup
- Automatic flush during beforeCompletion(): enabled
- Automatic session close at end of transaction: enabled
- JDBC batch size: 16
- JDBC batch updates for versioned data: disabled
- Scrollable result sets: enabled
- JDBC3 getGeneratedKeys(): enabled
- Connection release mode: after_statement
- Maximum outer join fetch depth: 2
- Default batch fetch size: 1
- Generate SQL with comments: disabled
- Order SQL updates by primary key: enabled
- Order SQL inserts for batching: enabled
- Query translator: org.hibernate.hql.ast.ASTQueryTranslatorFactory
- Using ASTQueryTranslatorFactory
- Query language substitutions: {}
- JPA-QL strict compliance: disabled
- Second-level cache: disabled
- Query cache: disabled
- Optimize cache for minimal puts: disabled
- Structured second-level cache entries: disabled
- Statistics: disabled
- Deleted entity synthetic identifier rollback: disabled
- Default entity-mode: pojo
- Named query checking : enabled
- building session factory
- Not binding factory to JNDI, no JNDI name configured
Login
FAQ
Search