we're facing an issue in a clustered deployment using Hibernate Search with an Infinispan directory provider and JMS master/slave topology.
Our current deployment is the following: 2 slave nodes, sending JMS messages for index updates and one master node responsible of consuming messages from the target JMS queue (The suggested Master/Slave topology by Hibernate with Infinispan as back-end).
After performing initialisation of the indexes, the system operates normally. However, at some undefined time, there are entries in the master node's logs, indicating message timeouts when trying to send messages via Jgroups to the slave nodes. Sometimes, after a number of these errors is logged, exceptions begin to appear indicating failure to obtain locks on some entity.
Infinispan directory provider has been configured to support persistence of the Lucene data & metadata caches, using the SingleFileStore class, in Replication mode.
We are using the default-hibernatesearch-jgroups-udp.xml for JGroups transport configuration, and our infinispan.xml configuration is the following:
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:6.0 http://www.infinispan.org/schemas/infinispan-config-6.0.xsd"
xmlns="urn:infinispan:config:6.0">
<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->
<global>
<!-- Duplicate domains are allowed so that multiple deployments with default
configuration of Hibernate Search applications work - if possible it would
be better to use JNDI to share the CacheManager across applications -->
<globalJmxStatistics enabled="false"
cacheManagerName="HibernateSearch" allowDuplicateDomains="true" />
<!-- If the transport is omitted, there is no way to create distributed
or clustered caches. There is no added cost to defining a transport but not
creating a cache that uses one, since the transport is created and initialized
lazily. -->
<transport clusterName="test-HibernateSearch-Infinispan-cluster">
<!-- Note that the JGroups transport uses sensible defaults if no configuration
property is defined. See the JGroupsTransport javadocs for more flags -->
<properties>
<property name="configurationFile" value="default-hibernatesearch-jgroups-udp.xml" />
</properties>
</transport>
<!-- Note that the JGroups transport uses sensible defaults if no configuration
property is defined. See the Infinispan wiki for more JGroups settings: http://community.jboss.org/wiki/ClusteredConfigurationQuickStart -->
<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER,
DONT_REGISTER. Hibernate Search takes care to stop the CacheManager so registering
is not needed -->
<shutdown hookBehavior="DONT_REGISTER" />
</global>
<!-- *************************** -->
<!-- Default "template" settings -->
<!-- *************************** -->
<default>
<locking lockAcquisitionTimeout="20000" writeSkewCheck="false"
concurrencyLevel="500" useLockStriping="false" />
<invocationBatching enabled="false" />
<!-- This element specifies that the cache is clustered. modes supported:
distribution (d), replication (r) or invalidation (i). Don't use invalidation
to store Lucene indexes (as with Hibernate Search DirectoryProvider). Replication
is recommended for best performance of Lucene indexes, but make sure you
have enough memory to store the index in your heap. Also distribution scales
much better than replication on high number of nodes in the cluster. -->
<clustering mode="replication">
<!-- Prefer loading all data at startup than later -->
<stateTransfer timeout="480000" fetchInMemoryState="true" />
<!-- Network calls are synchronous by default -->
<sync replTimeout="20000" />
</clustering>
<jmxStatistics enabled="true" />
<eviction maxEntries="-1" strategy="NONE" />
<expiration maxIdle="-1" />
</default>
<!-- ******************************************************************************* -->
<!-- Individually configured "named" caches. -->
<!-- -->
<!-- While default configuration happens to be fine with similar settings
across the -->
<!-- three caches, they should generally be different in a production environment. -->
<!-- -->
<!-- Current settings could easily lead to OutOfMemory exception as a CacheStore -->
<!-- should be enabled, and maybe distribution is desired. -->
<!-- ******************************************************************************* -->
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<namedCache name="LuceneIndexesMetadata">
<persistence passivation="false">
<store class="org.infinispan.persistence.file.SingleFileStore" fetchPersistentState="false" preload="true"
shared="false" purgeOnStartup="true" ignoreModifications="false">
<async enabled="true" flushLockTimeout="1" shutdownTimeout="25000" modificationQueueSize="1024" threadPoolSize="1"/>
<properties>
<property name="location" value="${infinispan.filestore.directory}"/>
</properties>
</store>
</persistence>
<clustering mode="replication">
<stateTransfer fetchInMemoryState="true" />
<sync replTimeout="60000" />
</clustering>
</namedCache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<namedCache name="LuceneIndexesData">
<persistence passivation="false">
<store class="org.infinispan.persistence.file.SingleFileStore" fetchPersistentState="false" preload="true"
shared="false" purgeOnStartup="true" ignoreModifications="false">
<async enabled="true" flushLockTimeout="1" shutdownTimeout="25000" modificationQueueSize="1024" threadPoolSize="1"/>
<properties>
<property name="location" value="${infinispan.filestore.directory}"/>
</properties>
</store>
</persistence>
<clustering mode="replication">
<stateTransfer fetchInMemoryState="true" />
<sync replTimeout="60000" />
</clustering>
</namedCache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<namedCache name="LuceneIndexesLocking">
<clustering mode="replication">
<stateTransfer fetchInMemoryState="true" />
<sync replTimeout="60000" />
</clustering>
</namedCache>
</infinispan>
The following Hibernate Search configuration properties are set, common between master & slave:
Login
FAQ
Search
