Upgrade of HS and related infinispan issues

declaw · **Joined:** Mon Aug 22, 2011 6:14 am **Posts:** 4

I have upgraded to HS 3.4 and plugged in Infinispan. I have two nodes, first node is responsible for writing to the index, the second exclusively handles search requests, a read-only node.

I am seeing lots of exceptions when the app starts. As a result the app does not successfully start.

On the second node I see the following :-

Code:

2011-08-22 06:00:16,754 [Timer-5,tc6-5758] ERROR util.TimeScheduler2 : failed running task org.jgroups.protocols.FD_ALL$TimeoutChecker@7d441a6c
java.lang.NullPointerException
        at org.jgroups.util.UUID.toString(UUID.java:193)
        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at org.jgroups.protocols.pbcast.GmsImpl$Request.toString(GmsImpl.java:126)
        at org.jgroups.protocols.pbcast.GMS$ViewHandler.add(GMS.java:1238)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:878)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:267)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:309)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:141)
        at org.jgroups.protocols.FD_ALL.suspect(FD_ALL.java:320)
        at org.jgroups.protocols.FD_ALL$TimeoutChecker.run(FD_ALL.java:373)
        at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:592)
        at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:524)
        at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:429)
        at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:295)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 06:00:17,661 [Timer-5,tc6-5758] ERROR protocols.PING : failed sending discovery request
java.lang.NullPointerException
        at org.jgroups.util.UUID.get(UUID.java:113)
        at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:48)
        at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:525)
        at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:592)
        at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:524)
        at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:429)
        at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:295)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

The first node complains with the following errors:-

Code:

2011-08-22 05:56:03,375 [Timer-3,tc1-61819] ERROR util.TimeScheduler2 : failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@31955dd4
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$1
        at org.jgroups.protocols.pbcast.STABLE.sendStableMessage(STABLE.java:663)
        at org.jgroups.protocols.pbcast.STABLE.access$1100(STABLE.java:43)
        at org.jgroups.protocols.pbcast.STABLE$StableTask.run(STABLE.java:808)
        at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:592)
        at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:524)
        at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:429)
        at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:295)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.jgroups.protocols.pbcast.STABLE$1
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1484)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)

The first node exceptions would indicate I am missing a jgroups dependency, however, jgroups-2.12.0.CR5.jar is on the classpath. Here is my POM just to confirm versions:-

Code:

   <dependency>
               <groupId>org.hibernate</groupId>
               <artifactId>hibernate-core</artifactId>
               <version>3.6.6.Final</version>
        </dependency>
        <dependency>
               <groupId>org.hibernate</groupId>
               <artifactId>hibernate-search</artifactId>
               <version>3.4.0.Final</version>
        </dependency>
        <dependency>
               <groupId>org.hibernate</groupId>
               <artifactId>hibernate-search-analyzers</artifactId>
               <version>3.4.0.Final</version>
        </dependency>
        <dependency>
               <groupId>org.hibernate</groupId>
               <artifactId>hibernate-search-infinispan</artifactId>
               <version>3.4.0.Final</version>
        </dependency>
        <dependency>
                <groupId>org.infinispan</groupId>
                <artifactId>infinispan-cachestore-jdbc</artifactId>
                <version>4.2.1.FINAL</version>
        </dependency>
        <dependency>
               <groupId>org.hibernate</groupId>
               <artifactId>hibernate-ehcache</artifactId>
               <version>3.6.6.Final</version>
        </dependency>

Here is my configuration (on both nodes) :-

hibernate.search.default.directory_provider=infinispan
hibernate.search.infinispan.configuration_resourcename=/default-hibernatesearch-infinispan.xml
hibernate.search.default.locking_strategy=native

Firewalls disabled on both nodes, using java.net.preferIPv4Stack=true. Is there something here that I am missing??

Here is my infinispan config:-

Code:

  <global>

        <!-- Duplicate domains are allowed so that multiple deployments with default configuration 
            of Hibernate Search applications work - if possible it would be better to use JNDI to share 
            the CacheManager across applications -->
        <globalJmxStatistics
            enabled="true"
            cacheManagerName="HibernateSearch"
            allowDuplicateDomains="true" />

        <!-- If the transport is omitted, there is no way to create distributed or clustered 
            caches. There is no added cost to defining a transport but not creating a cache that uses one, 
            since the transport is created and initialized lazily. -->
        <transport
            clusterName="HibernateSearch-Infinispan-cluster"
            distributedSyncTimeout="50000">
            <!-- Note that the JGroups transport uses sensible defaults if no configuration 
                property is defined. See the JGroupsTransport javadocs for more flags -->
            <!--
            <properties>
                <property name="configurationFile" value="/jgroups.xml"/>
            </properties>
            -->
        </transport>

        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER. 
            Hibernate Search takes care to stop the CacheManager so registering is not needed -->
        <shutdown
            hookBehavior="DONT_REGISTER" />

    </global>

  <default>

        <locking
            lockAcquisitionTimeout="20000"
            writeSkewCheck="false"
            concurrencyLevel="500"
            useLockStriping="false" />

        <!-- Invocation batching is required for use with the Lucene Directory -->
        <invocationBatching
            enabled="true" />

        <!-- This element specifies that the cache is clustered. modes supported: distribution 
            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as 
            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of 
            Lucene indexes, but make sure you have enough memory to store the index in your heap.
            Also distribution scales much better than replication on high number of nodes in the cluster. -->
        <clustering
            mode="replication">

            <!-- Prefer loading all data at startup than later -->
            <stateRetrieval
                timeout="20000"
                logFlushTimeout="30000"
                fetchInMemoryState="true"
                alwaysProvideInMemoryState="true" />

            <!-- Network calls are synchronous by default -->
            <sync
                replTimeout="20000" />
        </clustering>

        <jmxStatistics
            enabled="false" />

        <eviction
            maxEntries="-1"
            strategy="NONE" />

        <expiration
            maxIdle="-1" />

    </default>



   <namedCache name="LuceneIndexesMetadata">
      <clustering mode="replication">
         <stateRetrieval fetchInMemoryState="true" />
         <async useReplQueue="true" replQueueInterval="300" asyncMarshalling="false" />
      </clustering>
      <invocationBatching enabled="true" />
      <jmxStatistics enabled="false" />
      <loaders passivation="false" shared="true" preload="true">
         <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" 
                fetchPersistentState="true" 
                ignoreModifications="false" 
                purgeOnStartup="false">

            <!-- See the documentation for more configuration examples and flags. -->
            <properties>
               <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
               <property name="createTableOnStart" value="true" />

               <!-- Settings for MySQL: -->
               <property name="datasourceJndiLocation" value="java:comp/env/jdbc/Lucene" />
               <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory" />
               <property name="dataColumnType" value="bytea" />

               <property name="idColumnType" value="VARCHAR(256)" />
               <property name="idColumnName" value="idCol" />
               <property name="dataColumnName" value="dataCol" />
               <property name="stringsTableNamePrefix" value="LueneIndexesMetadata" />

               <property name="timestampColumnName" value="timestampCol" />
               <property name="timestampColumnType" value="BIGINT" />
            </properties>
            <async enabled="true" flushLockTimeout="25000" shutdownTimeout="7200" threadPoolSize="5" />
         </loader>
      </loaders>
      <eviction maxEntries="-1" strategy="NONE" />
      <expiration maxIdle="-1" />
   </namedCache>


   <namedCache name="LuceneIndexesData">
      <clustering mode="replication">
         <stateRetrieval fetchInMemoryState="true" />
         <async useReplQueue="true" replQueueInterval="300" asyncMarshalling="false" />
      </clustering>
      <invocationBatching enabled="true" />
      <jmxStatistics enabled="false" />
      <loaders passivation="false" shared="true" preload="true">
         <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" 
                fetchPersistentState="true" 
                ignoreModifications="false" 
                purgeOnStartup="false">

            <!-- See the documentation for more configuration examples and flags. -->
            <properties>
               <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
               <property name="createTableOnStart" value="true" />

               <!-- Settings for MySQL: -->
               <property name="datasourceJndiLocation" value="java:comp/env/jdbc/Lucene" />
               <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory" />
               <property name="dataColumnType" value="bytea" />

               <property name="idColumnType" value="VARCHAR(256)" />
               <property name="idColumnName" value="idCol" />
               <property name="dataColumnName" value="dataCol" />
               <property name="stringsTableNamePrefix" value="LuceneIndexesData" />

               <property name="timestampColumnName" value="timestampCol" />
               <property name="timestampColumnType" value="BIGINT" />
            </properties>
            <async enabled="true" flushLockTimeout="25000" shutdownTimeout="7200" threadPoolSize="5" />
         </loader>
      </loaders>
      <eviction maxEntries="-1" strategy="NONE" />
      <expiration maxIdle="-1" />
   </namedCache>

   <namedCache name="LuceneIndexesLocking">
        <clustering mode="replication">
                <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>
                <sync replTimeout="60000"/>
         </clustering>
   </namedCache>

</infinispan>

sanne.grinovero · **Posted:** Mon Aug 22, 2011 7:42 am

Hi,
the missing class is in that version of JGroups, so for some reason it seems it's not on the classpath.

Could you post the output of

Code:

mvn dependency:tree

And also try again and post the output of the same command again after adding the following dependency to your project?

Code:

org.jgroups:jgroups:jar:2.12.1.3.Final

declaw · **Joined:** Mon Aug 22, 2011 6:14 am **Posts:** 4

Code:

[INFO] +- log4j:log4j:jar:1.2.16:compile
[INFO] +- junit:junit:jar:3.8.1:test
[INFO] +- commons-lang:commons-lang:jar:2.4:compile
[INFO] +- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.0.3:compile
[INFO] +- org.hibernate:hibernate-core:jar:3.6.6.Final:compile
[INFO] |  +- antlr:antlr:jar:2.7.6:compile
[INFO] |  +- commons-collections:commons-collections:jar:3.1:compile
[INFO] |  +- dom4j:dom4j:jar:1.6.1:compile
[INFO] |  +- org.hibernate:hibernate-commons-annotations:jar:3.2.0.Final:compile
[INFO] |  +- org.hibernate.javax.persistence:hibernate-jpa-2.0-api:jar:1.0.1.Final:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.6.1:compile
[INFO] +- org.hibernate:hibernate-search:jar:3.4.0.Final:compile
[INFO] |  \- org.apache.lucene:lucene-core:jar:3.1.0:compile
[INFO] +- org.hibernate:hibernate-search-analyzers:jar:3.4.0.Final:compile
[INFO] |  +- org.apache.lucene:lucene-analyzers:jar:3.1.0:compile
[INFO] |  \- org.apache.solr:solr-analysis-extras:jar:3.1.0:compile
[INFO] |     +- org.apache.solr:solr-core:jar:3.1.0:compile
[INFO] |     |  +- org.apache.solr:solr-solrj:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-highlighter:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-memory:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-misc:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-spatial:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-spellchecker:jar:3.1.0:compile
[INFO] |     |  +- org.apache.solr:solr-commons-csv:jar:3.1.0:compile
[INFO] |     |  \- commons-io:commons-io:jar:1.4:compile
[INFO] |     +- org.apache.lucene:lucene-smartcn:jar:3.1.0:compile
[INFO] |     \- org.apache.lucene:lucene-stempel:jar:3.1.0:compile
[INFO] +- org.hibernate:hibernate-search-infinispan:jar:3.4.0.Final:compile
[INFO] |  \- org.infinispan:infinispan-lucene-directory:jar:4.2.1.FINAL:compile
[INFO] +- org.infinispan:infinispan-cachestore-jdbc:jar:4.2.1.FINAL:compile
[INFO] |  +- c3p0:c3p0:jar:0.9.1.2:compile
[INFO] |  \- org.infinispan:infinispan-core:jar:4.2.1.FINAL:compile
[INFO] |     +- org.jgroups:jgroups:jar:2.12.0.CR5:compile
[INFO] |     +- org.jboss.javaee:jboss-transaction-api:jar:1.0.1.GA:compile
[INFO] |     +- org.jboss.marshalling:river:jar:1.2.3.GA:compile
[INFO] |     +- org.jboss.marshalling:marshalling-api:jar:1.2.3.GA:compile
[INFO] |     \- org.rhq.helpers:rhq-pluginAnnotations:jar:3.0.1:compile
[INFO] |        \- i18nlog:i18nlog:jar:1.0.10:compile
[INFO] +- org.hibernate:hibernate-ehcache:jar:3.6.6.Final:compile
[INFO] |  \- net.sf.ehcache:ehcache-core:jar:2.4.3:compile
[INFO] +- javassist:javassist:jar:3.12.0.GA:compile
[INFO] +- commons-codec:commons-codec:jar:1.4:compile
[INFO] \- javax.transaction:jta:jar:1.1:compile

with JGroups 2.12.1.3.Final

Code:

[INFO] [dependency:tree {execution: default-cli}]
[INFO] +- log4j:log4j:jar:1.2.16:compile
[INFO] +- junit:junit:jar:3.8.1:test
[INFO] +- commons-lang:commons-lang:jar:2.4:compile
[INFO] +- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.0.3:compile
[INFO] +- org.hibernate:hibernate-core:jar:3.6.6.Final:compile
[INFO] |  +- antlr:antlr:jar:2.7.6:compile
[INFO] |  +- commons-collections:commons-collections:jar:3.1:compile
[INFO] |  +- dom4j:dom4j:jar:1.6.1:compile
[INFO] |  +- org.hibernate:hibernate-commons-annotations:jar:3.2.0.Final:compile
[INFO] |  +- org.hibernate.javax.persistence:hibernate-jpa-2.0-api:jar:1.0.1.Final:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.6.1:compile
[INFO] +- org.hibernate:hibernate-search:jar:3.4.0.Final:compile
[INFO] |  \- org.apache.lucene:lucene-core:jar:3.1.0:compile
[INFO] +- org.hibernate:hibernate-search-analyzers:jar:3.4.0.Final:compile
[INFO] |  +- org.apache.lucene:lucene-analyzers:jar:3.1.0:compile
[INFO] |  \- org.apache.solr:solr-analysis-extras:jar:3.1.0:compile
[INFO] |     +- org.apache.solr:solr-core:jar:3.1.0:compile
[INFO] |     |  +- org.apache.solr:solr-solrj:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-highlighter:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-memory:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-misc:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-spatial:jar:3.1.0:compile
[INFO] |     |  +- org.apache.lucene:lucene-spellchecker:jar:3.1.0:compile
[INFO] |     |  +- org.apache.solr:solr-commons-csv:jar:3.1.0:compile
[INFO] |     |  \- commons-io:commons-io:jar:1.4:compile
[INFO] |     +- org.apache.lucene:lucene-smartcn:jar:3.1.0:compile
[INFO] |     \- org.apache.lucene:lucene-stempel:jar:3.1.0:compile
[INFO] +- org.hibernate:hibernate-search-infinispan:jar:3.4.0.Final:compile
[INFO] |  \- org.infinispan:infinispan-lucene-directory:jar:4.2.1.FINAL:compile
[INFO] +- org.infinispan:infinispan-cachestore-jdbc:jar:4.2.1.FINAL:compile
[INFO] |  +- c3p0:c3p0:jar:0.9.1.2:compile
[INFO] |  \- org.infinispan:infinispan-core:jar:4.2.1.FINAL:compile
[INFO] |     +- org.jboss.javaee:jboss-transaction-api:jar:1.0.1.GA:compile
[INFO] |     +- org.jboss.marshalling:river:jar:1.2.3.GA:compile
[INFO] |     +- org.jboss.marshalling:marshalling-api:jar:1.2.3.GA:compile
[INFO] |     \- org.rhq.helpers:rhq-pluginAnnotations:jar:3.0.1:compile
[INFO] |        \- i18nlog:i18nlog:jar:1.0.10:compile
[INFO] +- org.hibernate:hibernate-ehcache:jar:3.6.6.Final:compile
[INFO] |  \- net.sf.ehcache:ehcache-core:jar:2.4.3:compile
[INFO] +- javassist:javassist:jar:3.12.0.GA:compile
[INFO] +- commons-codec:commons-codec:jar:1.4:compile
[INFO] +- javax.transaction:jta:jar:1.1:compile
[INFO] \- org.jgroups:jgroups:jar:2.12.1.3.Final:compile

That NoClassDefFoundError may be a red herring. I cannot reproduce. The NPE however happens *most* of the time when the application starts.

sanne.grinovero · **Posted:** Mon Aug 22, 2011 9:59 am

ok, let's focus on the NPE.

It seems you have hit JGRP-1260, which we couldn't reproduce and is waiting for a test case. Do you think you could provide a reproducible testcase to reproduce this? that would be very useful.

declaw · **Joined:** Mon Aug 22, 2011 6:14 am **Posts:** 4

OK, I'll see what I can do. Just another point, I am now testing on a single node and it happening there too. I have set jgroups.bind_addr=127.0.0.1.

It is weird because the behavior is not consistent across restarts of Tomcat. Sometimes there are no exceptions. Here is a full exception trace (that includes the NPE, the NoClassDefFoundError - even with the latest version of JGroups - and a few others)

Code:

2011-08-22 15:27:21,104 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-10612
2011-08-22 15:27:21,106 [Incoming-1,tc7-54685] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:21,107 [main]  WARN rpc.RpcManagerImpl : Could not find available peer for state, backing off and retrying
2011-08-22 15:27:35,937 [STREAMING_STATE_TRANSFER-sender-1,tc7-10612] ERROR jgroups.JGroupsTransport : Caught while responding to state transfer request
org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 30 seconds)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:176)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:217)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:578)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:690)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1484)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:651)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:580)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 30 seconds)
        at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilAcquired(JGroupsDistSync.java:42)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateTransactionLog(StateTransferManagerImpl.java:210)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:166)
        ... 12 more
2011-08-22 15:27:37,382 [main]  INFO factories.TransactionManagerFactory : Using a batchMode transaction manager
2011-08-22 15:27:37,393 [main]  INFO jgroups.JGroupsTransport : Starting JGroups Channel
2011-08-22 15:27:37,393 [main]  INFO jgroups.JGroupsTransport : Unable to use any JGroups configuration mechanisms provided in properties {}.  Using default JGroups configuration!
2011-08-22 15:27:37,396 [main]  INFO jgroups.JChannel : JGroups version: 2.12.1.3.Final
2011-08-22 15:27:37,607 [Incoming-1,tc7-10612]  INFO jgroups.JGroupsTransport : Received new cluster view: [tc7-10612|2] [tc7-10612, tc7-54685, tc7-35590]
2011-08-22 15:27:37,633 [Incoming-1,tc7-54685]  INFO jgroups.JGroupsTransport : Received new cluster view: [tc7-10612|2] [tc7-10612, tc7-54685, tc7-35590]
2011-08-22 15:27:37,639 [main]  INFO jgroups.JGroupsTransport : Received new cluster view: [tc7-10612|2] [tc7-10612, tc7-54685, tc7-35590]
2011-08-22 15:27:37,658 [main]  INFO jgroups.JGroupsTransport : Cache local address is tc7-35590, physical addresses are [127.0.0.1:50302]
2011-08-22 15:27:37,658 [main]  INFO factories.GlobalComponentRegistry : Infinispan version: Infinispan 'Ursus' 4.2.1.FINAL
2011-08-22 15:27:37,666 [main]  INFO decorators.AsyncStore : Async cache loader starting org.infinispan.loaders.decorators.AsyncStore@51b32cda
2011-08-22 15:27:37,675 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-10612
2011-08-22 15:27:37,767 [Incoming-1,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: java.io.EOFException: Read past end of file
        at org.infinispan.statetransfer.StateTransferManagerImpl.assertDelimited(StateTransferManagerImpl.java:395)
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:318)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: Read past end of file
        at org.jboss.marshalling.AbstractUnmarshaller.eofOnRead(AbstractUnmarshaller.java:184)
        at org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByteDirect(AbstractUnmarshaller.java:312)
        at org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByte(AbstractUnmarshaller.java:280)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
        at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
        at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:182)
        at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:186)
        at org.infinispan.statetransfer.StateTransferManagerImpl.assertDelimited(StateTransferManagerImpl.java:393)
        ... 31 more
2011-08-22 15:27:37,768 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-54685
2011-08-22 15:27:37,883 [Incoming-1,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: java.io.EOFException: Read past end of file
        at org.infinispan.statetransfer.StateTransferManagerImpl.assertDelimited(StateTransferManagerImpl.java:395)
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:318)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: Read past end of file
        at org.jboss.marshalling.AbstractUnmarshaller.eofOnRead(AbstractUnmarshaller.java:184)
        at org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByteDirect(AbstractUnmarshaller.java:312)
        at org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByte(AbstractUnmarshaller.java:280)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
        at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
        at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:182)
        at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:186)
        at org.infinispan.statetransfer.StateTransferManagerImpl.assertDelimited(StateTransferManagerImpl.java:393)
        ... 31 more
2011-08-22 15:27:37,884 [main]  WARN rpc.RpcManagerImpl : Could not find available peer for state, backing off and retrying
2011-08-22 15:27:38,884 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-10612
2011-08-22 15:27:38,886 [Incoming-2,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:38,887 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-54685
2011-08-22 15:27:38,889 [Incoming-1,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:38,890 [main]  WARN rpc.RpcManagerImpl : Could not find available peer for state, backing off and retrying
2011-08-22 15:27:40,890 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-10612
2011-08-22 15:27:40,892 [Incoming-2,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:40,893 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-54685
2011-08-22 15:27:40,895 [Incoming-1,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:40,895 [main]  WARN rpc.RpcManagerImpl : Could not find available peer for state, backing off and retrying
2011-08-22 15:27:44,896 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-10612
2011-08-22 15:27:44,901 [Incoming-1,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:44,902 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-54685
2011-08-22 15:27:44,906 [Incoming-2,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:44,907 [main]  WARN rpc.RpcManagerImpl : Could not find available peer for state, backing off and retrying
2011-08-22 15:27:52,907 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-10612
2011-08-22 15:27:52,910 [Incoming-1,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:52,910 [main]  INFO rpc.RpcManagerImpl : Trying to fetch state from tc7-54685
2011-08-22 15:27:52,912 [Incoming-1,tc7-35590] ERROR jgroups.JGroupsTransport : Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
        at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:326)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:199)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:595)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1441)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:99)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2011-08-22 15:27:52,913 [main]  WARN rpc.RpcManagerImpl : Could not find available peer for state, backing off and retrying
2011-08-22 15:28:07,680 [STREAMING_STATE_TRANSFER-sender-1,tc7-10612] ERROR jgroups.JGroupsTransport : Caught while responding to state transfer request
org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 30 seconds)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:176)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:217)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:578)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:690)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1484)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:651)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:580)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 30 seconds)
        at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilAcquired(JGroupsDistSync.java:42)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateTransactionLog(StateTransferManagerImpl.java:210)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:166)
        ... 12 more
2011-08-22 15:28:07,781 [STREAMING_STATE_TRANSFER-sender-1,tc7-54685] ERROR jgroups.JGroupsTransport : Caught while responding to state transfer request
org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 30 seconds)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:176)
        at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:217)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:578)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:690)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
        at org.jgroups.JChannel.up(JChannel.java:1484)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
        at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:477)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:651)
        at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:580)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 30 seconds)
        at org.infinispan.remoting.transport.jgroups.JGroupsDistSync.blockUntilAcquired(JGroupsDistSync.java:42)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateTransactionLog(StateTransferManagerImpl.java:210)
        at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:166)
        ... 12 more
 [OOB-2,tc7-54685] ERROR protocols.UDP : failed handling incoming message
java.lang.NullPointerException
        at org.jgroups.util.UUID.add(UUID.java:103)
        at org.jgroups.protocols.Discovery.up(Discovery.java:326)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
 [OOB-2,tc7-35590] ERROR protocols.UDP : failed handling incoming message

 [OOB-2,tc7-35590] ERROR protocols.UDP : failed handling incoming message
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
        at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:517)
        at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:702)
        at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:588)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:251)
        at org.jgroups.protocols.UNICAST.up(UNICAST.java:309)
        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:805)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:667)
        at org.jgroups.protocols.BARRIER.up(BARRIER.java:123)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
        at org.jgroups.protocols.Discovery.up(Discovery.java:293)
        at org.jgroups.protocols.PING.up(PING.java:69)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1647)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

declaw · **Joined:** Mon Aug 22, 2011 6:14 am **Posts:** 4

One other thing I failed to mention - I do not specify a jgroups configuration file. I am relying on defaults here.

sanne.grinovero · **Posted:** Fri Aug 26, 2011 8:44 am

Hi,
I've been looking into the JGroups code. There is no way possible for this variable to be ever null, besides possibly some strange classloader problem.

In what environment are you running this?

Could you inspect your classpath's health using a tool such as tattletale?