Problem Summary:
When running a multiple user stress test of a Web app which had multiple users submitting a transaction (the same type thus exercising the same code base) at the same time (i.e., click on a Submit button), 6 out of 8 failed.

Env Details:
WebSphere Platform 6.1 [ND 6.1.0.23 cf230910.10]
Oracle DB
Using WebSphere conn pools

Config Details JPA/JTA in dataSourceContext.xml:
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceUnitName" value="punit"/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="true"/>
<property name="database" value="ORACLE"/>
</bean>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.cache.provider_class">org.hibernate.cache.EhCacheProvider</prop>
<prop key="hibernate.transaction.factory_class">org.hibernate.transaction.CMTTransactionFactory</prop>
<prop key="hibernate.transaction.manager_lookup_class">org.hibernate.transaction.WebSphereExtendedJTATransactionLookup</prop>
<prop key="hibernate.ejb.naming_strategy">org.hibernate.cfg.ImprovedNamingStrategy</prop>
<prop key="hibernate.dialect">org.hibernate.dialect.Oracle10gDialect</prop>
<prop key="hibernate.jdbc.batch_size">30</prop>
</props>
</property>
</bean>

<bean id="wsJtaTm" class="org.springframework.transaction.jta.WebSphereTransactionManagerFactoryBean"/>

<bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="wsJtaTm"/>
</bean>
<bean id="ALM_DATASOURCE"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiEnvironment">
<props>
<prop key="java.naming.factory.initial">
com.ibm.websphere.naming.WsnInitialContextFactory
</prop>
<prop key="java.naming.provder.url">
iiop://localhost/
</prop>
</props>
</property>
<property name="jndiName">
<value>jdbc/ALMDB</value>
</property>
</bean>
------------------------------
Partial details from persistence.xml:

<persistence-unit name="punit" transaction-type="JTA">
<jta-data-source>jdbc/ALMDB</jta-data-source>
<class>com.fidsec.common.workflow.domain.WorkFlowInstance</class>
<class>com.fidsec.common.workflow.domain.WorkflowStatus</class>
<class>com.fidsec.common.workflow.domain.WorkflowStatusTypes</class>
----------------------------------------

I have looked at both the JTA and JPA config settings and it looks fine. Everything seems fine when there is a single user however as soon as there is multiple users doing the same thing it starts to croak. The app does plenty of reading and some writing but it seems like this write operation is problematic. Not sure how to proceed at this point. We can reproduce and it does error out consistently so I guess that is a good thing.
-----------------------------------------
Entries from the error log as follows:

Seen plenty of these but they are WARNings so not sure those matter:
[4/14/10 14:39:20:920 EDT] 00000028 SystemErr R 1348557 [WebContainer : 1] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class
[4/14/10 14:41:16:517 EDT] 00000028 SystemErr R 1464154 [WebContainer : 1] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class
[4/14/10 14:41:16:798 EDT] 00000028 SystemErr R 1464435 [WebContainer : 1] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class
[4/14/10 14:41:16:892 EDT] 0000002d SystemErr R 1464529 [WebContainer : 5] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class
[4/14/10 14:41:16:943 EDT] 00000026 SystemErr R 1464580 [WebContainer : 0] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class
[4/14/10 14:41:17:022 EDT] 0000002b SystemErr R 1464659 [WebContainer : 4] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class
[4/14/10 14:41:17:090 EDT] 00000030 SystemErr R 1464727 [WebContainer : 6] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class
[4/14/10 14:41:17:183 EDT] 00000031 SystemErr R 1464820 [WebContainer : 7] WARN org.hibernate.ejb.AbstractEntityManagerImpl - Cannot join transaction: do not override hibernate.transaction.factory_class

These are exceptions that are being thrown when the test is done:

4/14/10 14:41:17:223 EDT] 00000026 SystemErr R 1464860 [WebContainer : 0] INFO org.hibernate.jdbc.ConnectionManager - forcing batcher resource cleanup on transaction completion; forgot to close ScrollableResults/Iterator?
[4/14/10 14:41:17:223 EDT] 00000028 SystemErr R 1464860 [WebContainer : 1] INFO org.hibernate.jdbc.ConnectionManager - forcing batcher resource cleanup on transaction completion; forgot to close ScrollableResults/Iterator?
[4/14/10 14:41:17:228 EDT] 0000002b SystemErr R 1464865 [WebContainer : 4] WARN org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
[4/14/10 14:41:17:228 EDT] 0000002b SystemErr R 1464865 [WebContainer : 4] ERROR org.hibernate.util.JDBCExceptionReporter - DSRA9110E: Connection is closed.
[4/14/10 14:41:17:229 EDT] 0000002b SystemErr R 1464866 [WebContainer : 4] ERROR org.hibernate.ejb.AbstractEntityManagerImpl - Unable to mark for rollback on PersistenceException:
[4/14/10 14:41:17:229 EDT] 0000002b SystemErr R java.lang.UnsupportedOperationException
[4/14/10 14:41:17:230 EDT] 0000002b SystemErr R at org.hibernate.transaction.WebSphereExtendedJTATransactionLookup$TransactionManagerAdapter.setRollbackOnly(WebSphereExtendedJTATransactionLookup.java:161)
[4/14/10 14:41:17:230 EDT] 0000002b SystemErr R at org.hibernate.ejb.AbstractEntityManagerImpl.markAsRollback(AbstractEntityManagerImpl.java:421)
[4/14/10 14:41:17:230 EDT] 0000002b SystemErr R at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:576)
[4/14/10 14:41:17:230 EDT] 0000002b SystemErr R at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
[4/14/10 14:41:17:230 EDT] 0000002b SystemErr R at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:76)
[4/14/10 14:41:17:230 EDT] 0000002b SystemErr R at com.fidsec.alm.dao.AccessRequestDAOImpl.findAccessRequestDetailByCriteria(AccessRequestDAOImpl.java:208)
---------------
[4/14/10 14:41:17:235 EDT] 0000002b SystemErr R javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not execute query
[4/14/10 14:41:17:235 EDT] 0000002b SystemErr R at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
[4/14/10 14:41:17:235 EDT] 0000002b SystemErr R at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:76)
[4/14/10 14:41:17:235 EDT] 0000002b SystemErr R at com.fidsec.alm.dao.AccessRequestDAOImpl.findAccessRequestDetailByCriteria(AccessRequestDAOImpl.java:208)
----------------
[4/14/10 14:41:17:240 EDT] 0000002b SystemErr R Caused by: org.hibernate.exception.GenericJDBCException: could not execute query
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2231)

Caused by: com.ibm.websphere.ce.cm.ObjectClosedException: DSRA9110E: Connection is closed.
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.runtimeXIfNotClosed(WSJdbcConnection.java:2657)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.prepareStatement(WSJdbcConnection.java:2151)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.prepareStatement(WSJdbcConnection.java:2038)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1573)
at org.hibernate.loader.Loader.doQuery(Loader.java:696)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)

at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
at com.fidsec.alm.dao.AccessRequestDAOImpl.findAccessRequestDetailByCriteria(AccessRequestDAOImpl.java:208)
-------------------------------------------------
The above is a good representation of the kind of errors thrown. Here is kind of a generic summary of what the business logic is doing. Note that there is a @Transactional around each sub step and around the wrapper/parent which seems odd to me. This is a contactor's code whom has since left.

@Transactional(propagation=Propagation.REQUIRED, rollbackFor=Exception.class)
- create (wrapper/parent around activity)-> calls method to startWorkflow, calls method to saveNewAccessRequest

@Transactional(propagation=Propagation.REQUIRED, rollbackFor=Exception.class)
WorkFlowInstance startWorkFlow -> calls class libary to startWorkflow

@Transactional(propagation=Propagation.REQUIRED, rollbackFor=Exception.class)
public WorkFlowInstance startWorkFlow(..)

@Transactional(propagation=Propagation.REQUIRED, rollbackFor=Exception.class)
void saveNewAccessRequest(ALMAccessRequest request)
--------------
I am sure there is more detail that can be provided. Any thoughts/suggestions would be greatly appreciated.
Thanks.

a typo?


I'd try to follow this suggestion, it looks like the configuration is being forced - could also be related with typo above.

Thanks for pointing out the typo. I did make the change but still having persistence exceptions.

As I drill down into the code the following seems the most consistent error message though it seems to vary somewhat on each test. It is repeatable however:

4/15/10 13:00:56:008 EDT] 00000024 SystemErr R
xxx.alm.common.PersistenceException: Error saving workflow instance.
[4/15/10 13:00:56:101 EDT] 00000024 SystemErr R at xxx.alm.domain.AccessRequestServiceImpl.createAccessRequest(AccessRequestServiceImpl.java:146)
[4/15/10 13:00:56:101 EDT] 00000024 SystemErr R at xxx.alm.domain.AccessRequestServiceImpl$$FastClassByCGLIB$$2c5732f7.invoke(<generated>)

[4/15/10 13:00:56:117 EDT] 00000024 SystemErr R Caused by: javax.persistence.PersistenceException: Error saving workflow instance.
at com.xxx.common.workflow.dao.WorkflowDAOImpl.saveNewWorkFlowInstance(WorkflowDAOImpl.java:171)
at com.xxx.common.workflow.dao.WorkflowDAOImpl$$FastClassByCGLIB$$7cdbf25a.invoke(<generated>)
Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: Found two representations of same collection: com.xxx.common.workflow.domain.WorkFlowInstance.tasks
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)


-----------------------------------
In terms of the domain objects involved here there is an entity called a Workflow that utizes a sequence generator defined in Oracle for its ID. Embedded in this object there is a <List> of tasks. A task obj also has a sequence generator defined in Oracle. The Workflow object is created and populated with a Task <List>. I use the following to write out the Workflow and Tasks which seems pretty straight forward:


The following is a part of the Workflow:

public class WorkFlowInstance {


The following is the Task object:

I am facing the same issue while trying to save. Where you able to find out the root cause of the issue.


The following errors are showing while trying to Save:

1588154 [WebContainer : 3] INFO org.hibernate.jdbc.ConnectionManager - forcing batcher resource cleanup on transaction completion; forgot to close ScrollableResults/Iterator?

exception clearing maxRows/queryTimeout com.ibm.websphere.ce.cm.ObjectClosedException: DSRA9110E: Statement is closed.