Error in JGroupsBackendQueueProcessor when updating index

dungleonhart · **Joined:** Sat Jul 23, 2011 12:28 am **Posts:** 8

Hi guys,

I've configured Infinispan with JGroups backend for Hibernate Search in my project. During testing, I got stuck at an error that I could not overcome after a couple of days with Google.

Here is the log information for the exception that were thrown when I updated the value of an indexed object in a Slave node:

Code:

2011-08-05 23:31:47,738 [http-8080-7] ERROR org.hibernate.transaction.JDBCTransaction - JDBC rollback failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Can't call rollback when autocommit=true
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
   at com.mysql.jdbc.Util.getInstance(Util.java:381)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1012)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:986)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:981)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
   at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4802)
   at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
   at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
   at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:213)
   at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:192)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:676)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.doRollbackOnCommitException(AbstractPlatformTransactionManager.java:895)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:789)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy83.updateRequirement(Unknown Source)
   at tc.web.service.RequirementServiceController.renameRequirement(RequirementServiceController.java:93)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
   at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:427)
   at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:415)
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:788)
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:717)
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
   at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:65)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368)
   at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
   at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.security.CookieBaseSessionFilter.doFilter(CookieBaseSessionFilter.java:99)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:119)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.security.QasUsernamePasswordAuthenticationFilter.doFilter(QasUsernamePasswordAuthenticationFilter.java:142)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.service.ClientDatabaseSelectorFilter.doFilter(ClientDatabaseSelectorFilter.java:46)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:109)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.security.NoSessionFilter.doFilter(NoSessionFilter.java:34)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169)
   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
   at java.lang.Thread.run(Thread.java:662)
2011-08-05 23:31:47,741 [http-8080-7] ERROR org.springframework.orm.hibernate3.HibernateTransactionManager - Commit exception overridden by rollback exception
java.lang.NoSuchMethodError: org.jgroups.Message.<init>(Lorg/jgroups/Address;Lorg/jgroups/Address;Ljava/io/Serializable;)V
   at org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueProcessor.run(JGroupsBackendQueueProcessor.java:88)
   at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:143)
   at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.afterCompletion(PostTransactionWorkQueueSynchronization.java:102)
   at org.hibernate.search.backend.impl.EventSourceTransactionContext$DelegateToSynchronizationOnAfterTx.doAfterTransactionCompletion(EventSourceTransactionContext.java:189)
   at org.hibernate.engine.ActionQueue$AfterTransactionCompletionProcessQueue.afterTransactionCompletion(ActionQueue.java:591)
   at org.hibernate.engine.ActionQueue.afterTransactionCompletion(ActionQueue.java:209)
   at org.hibernate.impl.SessionImpl.afterTransactionCompletion(SessionImpl.java:602)
   at org.hibernate.jdbc.JDBCContext.afterTransactionCompletion(JDBCContext.java:273)
   at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:146)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy83.updateRequirement(Unknown Source)
   at tc.web.service.RequirementServiceController.renameRequirement(RequirementServiceController.java:93)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
   at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:427)
   at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:415)
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:788)
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:717)
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
   at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:65)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368)
   at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
   at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.security.CookieBaseSessionFilter.doFilter(CookieBaseSessionFilter.java:99)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:119)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.security.QasUsernamePasswordAuthenticationFilter.doFilter(QasUsernamePasswordAuthenticationFilter.java:142)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.service.ClientDatabaseSelectorFilter.doFilter(ClientDatabaseSelectorFilter.java:46)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:109)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at tc.security.NoSessionFilter.doFilter(NoSessionFilter.java:34)
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
   at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169)
   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
   at java.lang.Thread.run(Thread.java:662)

Hibernate Search configurations in my master node:

Code:

<prop key="hibernate.search.default.directory_provider">infinispan</prop>
<prop key="hibernate.search.worker.backend">jgroupsMaster</prop>

in my slave node:

Code:

<prop key="hibernate.search.default.directory_provider">infinispan</prop>
<prop key="hibernate.search.worker.backend">jgroupsSlave</prop>

and Hibernate Search libraries that I'm using:

It's really great for me to have your help.

Thanks a lot and Best regards,
Leonhart

sanne.grinovero · **Posted:** Fri Aug 05, 2011 2:00 pm

This is the cause of your error, quoting your stack:

Quote:

java.lang.NoSuchMethodError: org.jgroups.Message.<init>(Lorg/jgroups/Address;Lorg/jgroups/Address;Ljava/io/Serializable;)V

But this constructor does exist in JGroups jgroups-2.12.0.Final.jar, so I guess you have another copy of the JGroups jar around containing an incompatible class definition.

I'd suggest you to use Tattletale, it's a very simple command line tool able to scan the classpath and find duplicates no matter where they are hiding.

dungleonhart · **Joined:** Sat Jul 23, 2011 12:28 am **Posts:** 8

Hi s.grinovero,

First of all, thank you a lot for response.
I've used Tattletale to scan my WEB-INF/lib, but the org.jgroups.Message class is only found in the jgroups-2.12.0.Final.jar library.

For more information, I debugged and saw that the exception was thrown at the following line of code in the org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueProcessor class:

Code:

      try {
         Message message = new Message( null, factory.getAddress(), ( Serializable ) filteredQueue ); // the error is thrown from here
         factory.getChannel().send( message );
         if ( trace ) {
            log.trace( "Lucene works have been sent from slave {} to master node.", factory.getAddress() );
         }
      }

Moreover, I also tried to simulate the above codes by adding then executing the below line:

Code:

Message message = new Message(null, null, (Serializable)(new ArrayList<LuceneWork>()));

and it can run through this line without any exception.

Honestly, I don't know what to do next but ask for your help.

Thanks a lot and Best regards,
Leonhart.

sanne.grinovero · **Posted:** Sat Aug 06, 2011 11:25 am

Hi dungleonhart,

this NoSuchMethodError obviously means you have an incompatible version of JGroups somewhere, maybe it's not in your "WEB-INF/lib" but in other places.

It looks like you're deploying on Tomcat, correct? In that case you'll have to check for duplicates in all other Tomcat libraries, and in all other deployed applications & their extensions as well, since Tomcat doesn't isolate classloaders.

please double-check your classes, I'm confident you'll find a duplicate.

pbratton · **Joined:** Fri Nov 04, 2011 11:32 am **Posts:** 2

I think we ran into the same issue you did. The org.jgroups.Message constructor in jGroups used to compile Hibernate Search 3.4.1.Final expects a Serializable, but the jar distributed with HS-Infinispan expects an Object. The signature is bound to the calling library at compile time (see this issue on Stack Overflow: http://stackoverflow.com/questions/8081749/explicit-cast-causing-nosuchmethoderror/8081834#8081834), hence the NoSuchMethodError. The documentation even mentions that the HS jar must be upgraded from the default, but it's not clear precisely what version will work.

The following combination of jars worked for us:
Hibernate Search 3.4.1.Final
Infinispan 4.2.1.Final
jGroups 2.11.1.Final

For more information, here's our dialog with Sanne and the HS-Infinispan team:
http://community.jboss.org/message/635103