I posted this to the jgroups list and multiple individuals have said that this is a hibernate bug.

Thoughts?

[ERROR] [2012-02-23 09:38:44.185] couldn't deliver message [dst: <null>, src: dex-qa-34386 (2 headers), size=19794 bytes] - pbcast.NAKACK [Incoming-2,EH_CACHE,dex-qa-26129] [67500444 ms]


java.lang.IllegalArgumentException: java.io.InvalidObjectException: Could not find a SessionFactory [uuid=55c2b83f-4af2-4f46-81f7-859ef9dedfdd,name=null]
at org.jgroups.Message.getObject(Message.java:379)
at net.sf.ehcache.distribution.jgroups.dex.JGroupsCacheReceiver.receive(JGroupsCacheReceiver.java:63)
at org.jgroups.JChannel.invokeCallback(JChannel.java:758)
at org.jgroups.JChannel.up(JChannel.java:719)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
at org.jgroups.protocols.pbcast.StreamingStateTransfer.up(StreamingStateTransfer.java:262)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:481)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:730)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140)
at org.jgroups.protocols.Discovery.up(Discovery.java:355)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1699)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.InvalidObjectException: Could not find a SessionFactory [uuid=55c2b83f-4af2-4f46-81f7-859ef9dedfdd,name=null]
at org.hibernate.internal.SessionFactoryImpl.locateSessionFactoryOnDeserialization(SessionFactoryImpl.java:1787)
at org.hibernate.internal.SessionFactoryImpl.readResolve(SessionFactoryImpl.java:1767)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1061)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1761)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:479)
at net.sf.ehcache.distribution.EventMessage.readObject(EventMessage.java:138)
at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at java.util.ArrayList.readObject(ArrayList.java:593)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at org.jgroups.util.Util.objectFromByteBuffer(Util.java:416)
at org.jgroups.Message.getObject(Message.java:376)
 ... 18 more

So.. we were assuming that we'd only have to set the bootstrap factory once in the default cache and the others would pick it up.
Instead you have to set it for every cache, even if it is identical.

<bootstrapCacheLoaderFactory class="net.sf.ehcache.distribution.jgroups.JGroupsBootstrapCacheLoaderFactory" properties="bootstrapAsynchronously=false">

The error has gone away, but now we get this warning. Which has the same caused by.
Anything we should worry about?

[WARN ] [2012.02.23 14:58:14] jgroups.JChannel - failed calling receive() in receiver
java.lang.IllegalArgumentException: java.io.InvalidObjectException: Could not find a SessionFactory [uui d=45010a8a-9481-44c4-af8a-29c887f9e705,name=null]
at org.jgroups.Message.getObject(Message.java:291)
at net.sf.ehcache.distribution.jgroups.JGroupsCacheReceiver.receive(JGroupsCacheReceiver.java:62 )
at org.jgroups.JChannel.up(JChannel.java:1490)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:263)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:484)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:908)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:838)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:667)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:135)
at org.jgroups.protocols.Discovery.up(Discovery.java:294)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1109)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1665)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1642)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.InvalidObjectException: Could not find a SessionFactory [uuid=45010a8a-9481-44c4-af8a -29c887f9e705,name=null]
at org.hibernate.internal.SessionFactoryImpl.locateSessionFactoryOnDeserialization(SessionFactor yImpl.java:1787)
at org.hibernate.internal.SessionFactoryImpl.readResolve(SessionFactoryImpl.java:1767)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1061)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1761)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:479)
at net.sf.ehcache.distribution.EventMessage.readObject(EventMessage.java:138)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at java.util.ArrayList.readObject(ArrayList.java:593)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at org.jgroups.util.Util.objectFromByteBuffer(Util.java:398)
at org.jgroups.Message.getObject(Message.java:288)
... 17 more

we wound up doing two things which seems to have resolved it:

1) updated the jgroups tcp configuration for that cache. jgroups 3.x has some significant changes to their api. So, I grabbed tcp.xml from the jar and modified just the bind port and initial hosts strings. This seemed to clear up the bootstrap warnings.

Note: the example tcp unicast configs here: http://ehcache.org/documentation/replication/jgroups-replicated-caching won't work with jgroups3.

2) to deal with the sessionfactory deserialization messages, I needed to just define the hibernate. session_factory_name which sets up a jndi context and this gets rid of these messages. (which again, didn't seem to be causing a problem anyway)

I figured the latter out thanks to this fine blog post:
http://rstechjournal.blogspot.com/2012/07/hibernate-ehcache-cache-invalidation.html

1) right, JGroups 3.x isn't "drop-in" compatible with 2.x. If the version of EHCache you're using requires an older JGroups, you should either use the older JGroups version or update EHCache's code to use latest JGroups.

2) since you can have multiple sessionFactories in a same machine/JVM, when receiving a message over the network a SessionFactory id is needed to know were to route the messages.