Master/Slave JMS configuration

pourmo · **Joined:** Mon Oct 22, 2007 6:04 am **Posts:** 6

Hibernate Search version 3.0.0:
JBoss version 4.2.2:

I need to know how to setup Hibernate search Master/Slave configuration on multiple servers. I have seen many discussion about this on this forum, however no one has mentioned how I would need to setup JMS to point to master server.

I was under impression that I would need to setup something on the slave configuration which whould indicate where the remote server address (IP) is and the remote server needs to know which nodes it needs to broadcast to.

Code:

hibernate.search.default.sourceBase = /mnt/mastervolume/lucenedirs/mastercopy

# local copy location
hibernate.search.default.indexBase = /Users/prod/lucenedirs

the sourceBase refers to a directory on the server, but where is the bit that mentions the remote server url?

Kind regards
Mo

hardy.ferentschik · **Posted:** Tue Jul 08, 2008 12:37 pm

Hi there,

Quote:

I was under impression that I would need to setup something on the slave configuration which would indicate where the remote server address (IP) is and the remote server needs to know which nodes it needs to broadcast to.

The slaves don't have to know where the master is. All they need is access to the JMS queue. They are posting indexing events onto this queues which the master consumes and act upon. The JMS message broker can be completely standalone and decouples master and slaves. if you run within a J2EE container like JBoss you probably want to use the build in messaging capabilities. The example in the online documentation is build on this. Of course you can also use a different message broker, eg ActiveMQ. Have a look at this: http://hibernate.org/421.html.

On top of the JMS messaging there is one other requirement which to a certain degree breaks the loose coupling of master and slaves. The assumption of the current master/slave solution is that master as well as slaves have access to the same file system, eg via NFS or a SAN. The master will write to the master index (hibernate.search.default.sourceBase) at given time intervals (hibernate.search.default.refresh). The slaves need to be able to read this index (also at a given time intervals).

Please have a read through the online documentation http://www.hibernate.org/hib_docs/search/reference/en/html_single/#jms-backend.

Maybe post what is unclear in the documentation since we get the question regarding JMS configuration quite often. Maybe we can make the documentation clearer.

Or does this diagram make it a little clearer: http://www.hibernate.org/hib_docs/search/reference/en/shared/images/jms-backend.png.

Good luck ;-)

--Hardy

pourmo · **Joined:** Mon Oct 22, 2007 6:04 am **Posts:** 6

Hi
Thanks for you feedback.

We have implemented this and it partly works.
Updated an entity on master; we can search on it on both master and after the duration can search on it in slave.

But updating in slave has no effect. Is there something that we may have missed.

Regards
Mo

hardy.ferentschik · **Posted:** Thu Jul 10, 2008 7:55 am

Hi,

hard to tell what your problem is without further information. My guess is that the master is not listening/receiving indexing events from the slaves. How did you register the master as message consumer to the queue?

Maybe this thread helps: http://forum.hibernate.org/viewtopic.php?t=988454.

If this is not the solution to your problem you could maybe post your configuration - hibernate config files, indexing code, ...

--Hardy

pourmo · **Joined:** Mon Oct 22, 2007 6:04 am **Posts:** 6

Thanks you for your reponse;
We have the same SEAM based ear application deployed to two machines running JBoss-4.2.2
the master-slave configurations are as follows; The MDB used is based on the Hibernate docs;
The changes in the master is detected by slave so the dir mappings are ok; however, changes in the slave have no effect;
I have tested this with and without the optional property setting: hibernate.search.worker.jndi.* with no difference.
Many thanks for your help;

Master

Code:

  <!-- ==== master configuration  ===== --> 
  
  <persistence-unit name="MyApp">  
    <provider>org.hibernate.ejb.HibernatePersistence</provider>
    <jta-data-source>java:/MyAppDatasource</jta-data-source>
    <properties>
          <property name="hibernate.dialect" value="org.hibernate.dialect.SQLServerDialect"/>
          <property name="hibernate.hbm2ddl.auto" value="update"/>
          <property name="hibernate.cache.use_query_cache" value="true"/>
          <property name="hibernate.show_sql" value="false"/>
            
          <property name="hibernate.cache.provider_class" value="org.hibernate.cache.HashtableCacheProvider"/>          
                     
          <property name="hibernate.search.default.sourceBase"  value="c://myapp/indexes/mastercopy"/>
          <property name="hibernate.search.default.indexBase"  value="c://myapp/indexes"/>
          <property name="hibernate.search.default.refresh" value="1200" /> 
          <property name="hibernate.search.default.directory_provider"  value="org.hibernate.search.store.FSMasterDirectoryProvider"/>
          
                
        <property name="hibernate.ejb.event.post-insert" value="org.hibernate.search.event.FullTextIndexEventListener"/>
        <property name="hibernate.ejb.event.post-update" value="org.hibernate.search.event.FullTextIndexEventListener"/>
        <property name="hibernate.ejb.event.post-delete" value="org.hibernate.search.event.FullTextIndexEventListener"/>         

        <property name="jboss.entity.manager.factory.jndi.name" value="java:/MyAppEntityManagerFactory"/>          
        <property name="hibernate.search.worker.batch_size" value="100" />  

          
    </properties>
  </persistence-unit>   

Slave

Code:

  <!-- ====== slave configuration  ======== --> 
  
  
  <persistence-unit name="MyApp">  
    <provider>org.hibernate.ejb.HibernatePersistence</provider>
    <jta-data-source>java:/MyAppDatasource</jta-data-source>
    <properties>
          <property name="hibernate.dialect" value="org.hibernate.dialect.SQLServerDialect"/>
          <property name="hibernate.hbm2ddl.auto" value="update"/>
          <property name="hibernate.cache.use_query_cache" value="true"/>
          <property name="hibernate.show_sql" value="false"/>
            
          <property name="hibernate.cache.provider_class" value="org.hibernate.cache.HashtableCacheProvider"/> 
          
          <!--    k is the map drive to the master server     -->          
          <property name="hibernate.search.default.sourceBase"  value="k://myapp/indexes/mastercopy"/> 
          
          <property name="hibernate.search.default.indexBase"  value="c://myapp/indexes/slave"/>
          <property name="hibernate.search.default.refresh" value="1200" />  

          <property name="hibernate.search.default.directory_provider"  value="org.hibernate.search.store.FSSlaveDirectoryProvider"/>
          
          <property name="hibernate.search.worker.backend" value="jms" />  
                    
          <property name="hibernate.search.worker.jndi.java.naming.factory.initial" value="org.jnp.interfaces.NamingContextFactory" />  
          <property name="hibernate.search.worker.jndi.java.naming.factory.url.pkgs" value="org.jboss.naming:org.jnp.interfaces" /> 
          <property name="hibernate.search.worker.jndi.java.naming.provider.url" value="MASTER_SERVER_IP" /> 
          
          <property name="hibernate.search.worker.jms.connection_factory" value="java:/ConnectionFactory" />  
          <property name="hibernate.search.worker.jms.queue" value="queue/hibernatesearch" />  
          
  
                    
        <property name="hibernate.ejb.event.post-insert" value="org.hibernate.search.event.FullTextIndexEventListener"/>
        <property name="hibernate.ejb.event.post-update" value="org.hibernate.search.event.FullTextIndexEventListener"/>
        <property name="hibernate.ejb.event.post-delete" value="org.hibernate.search.event.FullTextIndexEventListener"/>         

        <property name="jboss.entity.manager.factory.jndi.name" value="java:/MyAppEntityManagerFactory"/>          
        <property name="hibernate.search.worker.batch_size" value="100" />  

          
    </properties>
  </persistence-unit>  

The MDB in the master:

Code:

  
  @MessageDriven(messageListenerInterface=MessageListener.class, activationConfig = {
        @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
        @ActivationConfigProperty(propertyName="destination", propertyValue="queue/hibernatesearch"),
        @ActivationConfigProperty(propertyName="DLQMaxResent", propertyValue="1")
        } )
  public class MDBSearchController 
           extends AbstractJMSHibernateSearchController 
           implements MessageListener {
     
     @PersistenceContext EntityManager em;
     //method retrieving the appropriate session
     protected Session getSession() {
        return (Session) em.getDelegate();
     }
     //potentially close the session opened in #getSession(), not needed here
     protected void cleanSessionIfNeeded(Session session){
     }
  
}

The queue-hibernatesearch-service.xml file deployed in the JBoss deploy dir for slave node:

Code:

<server>
   <mbean code="org.jboss.mq.server.jmx.Queue"
      name="jboss.mq.destination:service=Queue,name=MDBSearchController">
      <attribute name="JNDIName">queue/hibernatesearch</attribute>
      <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends>
   </mbean>


</server>

hardy.ferentschik · **Posted:** Thu Jul 10, 2008 2:02 pm

Hi,

the configuration looks correct. I am not sure about the additional JNDI properties and the deployment descriptor for the queue.

Is there anything in the log? Maybe you can turn on debug trace? Given that you are using JBoss you could probably inspect the queue using the JBoss console to see whether it receives any messages.

How does the code look like modifying an entity on a slave?

--Hardy

Htaylor · **Joined:** Tue Jul 22, 2008 8:19 am **Posts:** 8

To the Hibernate Search team,

I have sucessfully implemented Hibernate Search in a Tomcat/Spring/Hibernate environment.

We have decided to upgrade our application server to JBOSS as it supports the full J2EE stack and this is where I came across the JMS/Master Slave config for Hibernate Search

In order to create a working prototype I have ported 3 instances of the webapp from Tomcat to the JBOSS deployer. I have configured 2 instances as slaves and one as the master.

My slave hibernate config is as follows

1. Ignore the second level stuff for Memchached as its disabled
2. The two instances each have their own index directory "slave1" and "slave2". For this example all directories are on the same machine.

Code:

hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect

# Needed by Hibernate3 Maven Plugin defined in pom.xml
hibernate.connection.username=####
hibernate.connection.password=####
hibernate.connection.url=jdbc:mysql://localhost/tantalus?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=utf-8
hibernate.connection.driver_class=com.mysql.jdbc.Driver

hibernate.cache.use_second_level_cache=false
hibernate.cache.provider_class=com.googlecode.hibernate.memcached.MemcachedCacheProvider
hibernate.generate_statistics true
hibernate.cache.use_structured_entries true 
hibernate.memcached.servers=################:11211

hibernate.search.default.sourceBase=C://java//indexes//shared
 # local copy location #
hibernate.search.default.indexBase=C://java//indexes//slave1

 # refresh every 5 minutes
hibernate.search.default.refresh=300 
# appropriate directory provider 
hibernate.search.directory_provider org.hibernate.search.store.FSSlaveDirectoryProvider 
## Backend configuration 
hibernate.search.worker.backend jms
hibernate.search.worker.jms.connection_factory java:/ConnectionFactory
hibernate.search.worker.jms.queue queue/hibernatesearch
##java specific settings
java.naming.provider.url jnp://localhost:1099

My Master configuration is as such. there may be some nefarious entries owing to playing aorund trying to get it to work

Code:

hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect

# Needed by Hibernate3 Maven Plugin defined in pom.xml
hibernate.connection.username=#####
hibernate.connection.password=#####
hibernate.connection.url=jdbc:mysql://localhost/tantalus?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=utf-8
hibernate.connection.driver_class=com.mysql.jdbc.Driver

### master configuration
## DirectoryProvider # (remote) master location where information is copied to
hibernate.search.default.sourceBase=C://java//indexes//shared
# local master location
hibernate.search.default.indexBase=C://java//indexes//master
# refresh every 2 minutes
hibernate.search.default.refresh=120
# appropriate directory provider 
hibernate.search.directory_provider org.hibernate.search.store.FSMasterDirectoryProvider 

hibernate.cache.use_second_level_cache=false
hibernate.cache.provider_class=com.googlecode.hibernate.memcached.MemcachedCacheProvider
hibernate.generate_statistics true
hibernate.cache.use_structured_entries true 
hibernate.memcached.servers=################:11211
##java specific settings
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jnp.interfaces
java.naming.provider.url jnp://localhost:1099

The master also contains the MDBSearchController specified in the examples as well as the events specifed in the config guide for the session factory.

I have specified the queue in JBOSS

Code:

<mbean code="org.jboss.mq.server.jmx.Queue" 
name="jboss.mq.destination:service=Queue,name=hibernatesearch"> 
<attribute name="JNDIName">queue/hibernatesearch</attribute> 
<depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> 
<depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends>
<attribute name="SecurityConf">
<security>
<role name="guest" read="true" write="true" create="true"/>
<role name="publisher" read="true" write="true" create="true"/>
<role name="durpublisher" read="true" write="true" create="true"/>
</security>
</attribute>

</mbean> 

The results I get at this stage are as follows

1) The slaves correctly do not update their own indexes, but send a message to the queue, I can see this happening in the JMX consol so I know the messages are being sent.

2) The Master updates its own index but it does not seem to be fulfilling its role as a message consumer as
a) Its not processing the messages sent to the queue
b) Its not initiating the fie system replication and copy of the index

What is eluding me at this point is what else is needed to initate points a) and b)?

If you need any other config let me know. I am pretty keen to get this as a working prototype so I can do some stress and performance testing.

Would anybody be able to shed any light or provide any avenues of investigation as to why this may not be working.

Bearing in mind I am intermediate Spring/Hibernate and pretty new to JBOSS/JMS etc

Thanks

Htaylor · **Joined:** Tue Jul 22, 2008 8:19 am **Posts:** 8

Well geeting there

As I suspcted it appeared to be an issue with Spring talking to the JMS server.

So I have added the following config into applicationContext of the webapp

Code:

<bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate"> 
         <property name="environment">   
             <props>     
               <prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</prop>     
               <prop key="java.naming.provider.url">jnp://localhost:1099</prop>  
               <prop key="java.naming.factory.url.pkgs"> org.jboss.naming:org.jnp.interfaces</prop> 
            </props> 
         </property>
      </bean>
      <bean id="connectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean"> 
             <property name="jndiTemplate">   
             <ref bean="jndiTemplate" /> 
             </property>
              <property name="jndiName">   
               <value>java:/ConnectionFactory</value> 
              </property>
       </bean>
       <bean id="queue" class="org.springframework.jndi.JndiObjectFactoryBean"> 
       <property name="jndiTemplate">   
          <ref bean="jndiTemplate" /> 
        </property>
         <property name="jndiName">   
         <value>queue/hibernatesearch</value> 
         </property>
       </bean>
    
 <bean id="listenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">    
            <property name="concurrentConsumers" value="5" /> 
             <property name="connectionFactory" ref="connectionFactory" />    
             <property name="destination" ref="queue" />    
             <property name="messageListener" ref="searchController" />  
         </bean>
    
    <bean id="searchController" class="com.tantalus.jms.MDBSearchController">
     <property name="sessionFactory" ref="sessionFactory"/>
    </bean>

This effectively injects the properties neccessary into application context .

The listener is now working, and the master index is being updated when entites are added to the slave instances.

However, the slave indexes are not being updated,

Getting there

Hexain · **Joined:** Mon Jun 29, 2009 6:49 am **Posts:** 1

Your solution is sound. But have you thought through the consequences of what happens if your JMS call in the callback fails?

sanne.grinovero · **Posted:** Tue Jun 30, 2009 11:07 am

with Hibernate Search, the reliable source of data is always the database, so if a DB commit fails the index is not updated,
but if an index change fails we won't rollback the DB transaction: you can always rebuild the index as long as your DB is consistent,
and most (all?) use cases can live with a slightly out of date index, giving time to operations to recover from failures and rebuild the index.
(better explanations in the book)

The same applies for JMS, if it fails you don't want to block your business but will live with a slightly out of date index until you fix the failure. As soon as the master recovers you can rebuild the index and continue.
You might like to configure JMS to be transactional, that's not very useful for this scenario, but the code won't prevent you to do.

There's much work going on to make a truly distributed index, it will probably be released in 3.2.0