-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 6 posts ] 
Author Message
 Post subject: Hibernate search with multiple applications
PostPosted: Tue Nov 26, 2013 3:35 am 
Newbie

Joined: Tue Nov 26, 2013 3:14 am
Posts: 3
Hi

I have several applications which use Hibernate Search and we have several problem using them in a clustered environment. I have configured infinispan to use as directory provider. In each application I configured a different hibernate search clusterName with a different infinispan configuration (the main difference is the location of the indexes)

For example in persistence.xml of application 1 I have
Code:
<property name="hibernate.search.services.jgroups.clusterName" value="HibernateSearchDocroom-Infinispan-cluster-PRD" />

and in application 2 I have

Code:
<property name="hibernate.search.services.jgroups.clusterName" value="HibernateSearchPerso-Infinispan-cluster-PRD" />


Starting the cluster works fine, but I see a lot of warning messages. Even when I have only one node up, I see lots of these messages in the log file.

Code:
08:13:38,122 WARN  [org.jgroups.protocols.UDP] (OOB-2,HibernateSearchDocroom-Infinispan-cluster-PRD,jbosssrv02-11306) [JGRP00012] discarded message from different cluster HibernateSearchPerso-Infinispan-cluster-PRD (our cluster is HibernateSearchDocroom-Infinispan-cluster-PRD). Sender was 1adae883-a056-9515-53c7-e37b7c767398 (received 7 identical messages from 1adae883-a056-9515-53c7-e37b7c767398 in the last 65411 ms)


I suspect that all the configured hibernate search clusterNames get all the jgroups messages and have to discard them.
Is there a way to fix this because after several time the jboss 7.1 cluster won't work anymore. (The applications do not respond anymore) And I have to kill all the nodes except one to get the applications working again.
Is it possible that the messages from application1 are only seen by the hibernateSearch cluster of application1? I tried some parameters in a custom jgroups config (bind_port, mcast_port, ...) but without luck.

Does anybody have a similar configuration and can give me some hints ?

My configuration is jboss 7.1.1, Hibernate Search 4.4

Regards

Georges Goebel


Top
 Profile  
 
 Post subject: Re: Hibernate search with multiple applications
PostPosted: Tue Dec 03, 2013 8:33 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hi Georges,
JGroups uses the "cluster name" strategy to make sure that unrelated messages are ignored, but it literally is a "last defense". It's better to actually prevent that from happening by changing the networking options so that you make sure they don't "talk" to each other.

Indeed as you have tried you should change the ports, or alternatively on Linux you can create additional ethernet interfaces using different networks (even on the same physical device / connection).
If you post the JGroups configuration files with the changed ports I can try if I see the mistake myself, but it's probably a better question for the JGroups mailing list.

A suggestion on used versions: use WildFly 8 rather than JBoss 7.1 if you can: lots of improvements in the clustering technology!

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Hibernate search with multiple applications
PostPosted: Wed Nov 05, 2014 7:05 am 
Newbie

Joined: Tue Nov 26, 2013 3:14 am
Posts: 3
Hi, thanks for you reply

In the meanwile I upgraded to wildfly 8.1 and my applications use the hibernate search library (4.5.1) provided by wildfly 8.1 Final . However I still get te same messages because I guess the same issue is still present. I tried several modifications (jgroups.udp.mcast_port) in groups.xml without any success.
If you can point me to the place in groups.xml (or anywhere else) to fix these messages. I think the best solution would be (if possible) to have the same multicast address for all applications with hibernate search, but each application uses a different mcast_port.

Code:

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-2.12.xsd">
    <UDP
            mcast_addr="${jgroups.udp.mcast_addr:228.10.16.9}"
            mcast_port="${jgroups.udp.mcast_port:45799}"
            bind_port="${jboss.jgroups.udp.bind_port:55300}"
            tos="8"
            ucast_recv_buf_size="20000000"
            ucast_send_buf_size="640000"
            mcast_recv_buf_size="25000000"
            mcast_send_buf_size="640000"
            loopback="true"
            discard_incompatible_packets="true"
            max_bundle_size="64000"
            max_bundle_timeout="30"
            ip_ttl="${jgroups.udp.ip_ttl:2}"
            enable_bundling="true"
            enable_diagnostics="false"
            thread_naming_pattern="pl"
            thread_pool.enabled="true"
            thread_pool.min_threads="2"
            thread_pool.max_threads="30"
            thread_pool.keep_alive_time="5000"
            thread_pool.queue_enabled="false"
            thread_pool.queue_max_size="100"
            thread_pool.rejection_policy="Discard"
            oob_thread_pool.enabled="true"
            oob_thread_pool.min_threads="2"
            oob_thread_pool.max_threads="30"
            oob_thread_pool.keep_alive_time="5000"
            oob_thread_pool.queue_enabled="false"
            oob_thread_pool.queue_max_size="100"
            oob_thread_pool.rejection_policy="Discard"
            />

    <PING timeout="3000" num_initial_members="10"/>
    <MERGE2 max_interval="30000" min_interval="10000"/>
    <FD_SOCK/>
    <FD/>
    <BARRIER />
    <pbcast.NAKACK exponential_backoff="0"
                   use_mcast_xmit="true"
                   retransmit_timeout="300,600,1200"
                   discard_delivered_msgs="true"/>
    <UNICAST timeout="300,600,1200"/>
    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="1000000"/>
    <pbcast.GMS print_local_addr="true" join_timeout="3000" view_bundling="true"/>
    <UFC max_credits="500000" min_threshold="0.20"/>
    <MFC max_credits="500000" min_threshold="0.20"/>
    <FRAG2 frag_size="60000"  />
</config>


infinispan config

Code:
<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:6.0 http://www.infinispan.org/schemas/infinispan-config-6.0.xsd"
        xmlns="urn:infinispan:config:6.0">

    <!-- *************************** -->
    <!-- System-wide global settings -->
    <!-- *************************** -->
    <global>
        <!-- Duplicate domains are allowed so that multiple deployments with default configuration
            of Hibernate Search applications work - if possible it would be better to use JNDI to share
            the CacheManager across applications -->
        <globalJmxStatistics enabled="true" cacheManagerName="HibernateSearch" allowDuplicateDomains="true" />
        <!-- If the transport is omitted, there is no way to create distributed or clustered
            caches. There is no added cost to defining a transport but not creating a cache that uses one,
            since the transport is created and initialized lazily. -->
        <transport clusterName="HibernateSearch_${project.name}-Infinispan-cluster-${classifier}" distributedSyncTimeout="50000">
            <!-- Note that the JGroups transport uses sensible defaults if no configuration
                property is defined. See the JGroupsTransport javadocs for more flags -->
           
                        <properties>
                            <property name="configurationFile" value="jgroups.xml"/>
                        </properties>
          </transport>
        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
            Hibernate Search takes care to stop the CacheManager so registering is not needed -->
        <shutdown hookBehavior="DONT_REGISTER"/>
    </global>

    <!-- *************************** -->
    <!-- Default "template" settings -->
    <!-- *************************** -->
    <default>
        <locking lockAcquisitionTimeout="20000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false" />
        <lazyDeserialization enabled="false" />
        <!-- Invocation batching is required for use with the Lucene Directory -->
        <invocationBatching enabled="true" />
        <!-- This element specifies that the cache is clustered. modes supported: distribution
            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
            Lucene indexes, but make sure you have enough memory to store the index in your heap.
            Also distribution scales much better than replication on high number of nodes in the cluster. -->
        <clustering mode="replication">
            <!-- Prefer loading all data at startup than later -->
            <!-- Old: <stateRetrieval timeout="60000" logFlushTimeout="30000" fetchInMemoryState="true" alwaysProvideInMemoryState="true" /> -->
            <stateTransfer timeout="60000" fetchInMemoryState="true" />
            <!-- Network calls are synchronous by default -->
            <sync replTimeout="20000" />
        </clustering>
        <jmxStatistics enabled="true" />
        <eviction maxEntries="-1" strategy="NONE" />
        <expiration maxIdle="-1" />
    </default>

    <!-- ******************************************************************************* -->
    <!-- Individually configured "named" caches.                                         -->
    <!--                                                                                 -->
    <!-- While default configuration happens to be fine with similar settings across the -->
    <!-- three caches, they should generally be different in a production environment.   -->
    <!--                                                                                 -->
    <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->
    <!-- should be enabled, and maybe distribution is desired.                           -->
    <!-- ******************************************************************************* -->

    <!-- *************************************** -->
    <!--  Cache to store Lucene's file metadata  -->
    <!-- *************************************** -->
    <namedCache name="LuceneIndexesMetadata_${project.name}_${classifier}">
        <clustering mode="replication">
            <stateTransfer fetchInMemoryState="true" />
            <sync replTimeout="25000" />
        </clustering>
        <persistence>
            <singleFile preload="true" shared="false" location="/data/indexes/infinispan/fullTextStore/${project.name}_${classifier}" />
        </persistence>
    </namedCache>

    <!-- **************************** -->
    <!--  Cache to store Lucene data  -->
    <!-- **************************** -->
    <namedCache name="LuceneIndexesData_${project.name}_${classifier}">
        <clustering mode="replication">
            <stateTransfer fetchInMemoryState="true" />
            <sync replTimeout="25000"/>
        </clustering>
        <persistence>
            <store class="org.infinispan.persistence.file.SingleFileStore">
                <properties>
                    <property name="location" value="/data/indexes/infinispan/fullTextStore/${project.name}_${classifier}" />
                </properties>
            </store>
        </persistence>
    </namedCache>

    <!-- ***************************** -->
    <!--  Cache to store Lucene locks  -->
    <!-- ***************************** -->
    <namedCache name="LuceneIndexesLocking_${project.name}_${classifier}">
        <clustering mode="replication">
            <stateTransfer fetchInMemoryState="true" />
            <sync replTimeout="25000" />
        </clustering>
    </namedCache>
</infinispan>


Top
 Profile  
 
 Post subject: Re: Hibernate search with multiple applications
PostPosted: Thu Nov 06, 2014 3:18 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Consider that both Infinispan and the JGroups backend of Hibernate Search use JGroups, and by default each component will start an independent JGroups channel.
I'm understanding that you are aiming at isolating each Search-powered application from each other by using different ports; are you also isolating the two channels that each application is going to start from each other? Essentially you'd need to pass a different JGroups configuration file to Infinispan and to the Hibernate Search backend configuration.

I think that would solve the problem you're seeing, but there is yet another solution now which is more elegant: rather than needing to assign two sets of ports for each deployed application (and one for the application server), you could have them all share the same JGroups channel using the FORK protocol.

See also:
- http://planet.jboss.org/post/how_to_hijack_a_jgroups_channel_inside_infinispan_jboss_and_get_away_with_it

How to get the Search backend to use the FORK based channel: you'd need to add a bit of code for that, sorry I didn't have time to make this easier yet. One trick is would be to inject the JGroups channel into the configuration properties, but that implies your application has to boot Hibernate programmatically (not using a persistence.xml configuration file).. for more details see org.hibernate.search.backend.jgroups.impl.DispatchMessageSender (I'm referring to latest version 5.0.0.Beta1). If you're interested into exploring this we can certainly collaborate to patch it and make it easier.

For Infinispan, I don't know the details but there is an integration point as well:
org.infinispan.remoting.transport.jgroups.JGroupsChannelLookup

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Hibernate search with multiple applications
PostPosted: Thu Nov 06, 2014 3:24 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
I've created https://hibernate.atlassian.net/browse/HSEARCH-1714 to make sure we can eventually make this easier.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Hibernate search with multiple applications
PostPosted: Mon Nov 10, 2014 9:30 am 
Newbie

Joined: Tue Nov 26, 2013 3:14 am
Posts: 3
Hi,

Thanks for your information, I will give it a try. However, I am looking forward when you add this feature in the standard lib.

Georges


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 6 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.