Hi,

I ve a few questions with regards to master/slave config. Appreciate your feedback..

1. Could topic be used instead of queue for JMS cause I need durability.
2. Can I have 2 masters in the architecture.
3. Slave is deployed on a J2EE container but not the master. Thus, for the master, I need to manually retrieve from Slave's JMS machine and do a manual indexing?

thanks!

As far as I know durability is not a question of topic vs queue. Both topic and queues can be durable. In fact durable topics are more complicated to handle than durable queues. You need to read up on the documentation of your JMS provider. Also have a look here - how-do-durable-queues-and-topics-work


I guess you could configure your JMS architecture to behave like a distributed queue. In this case messages are getting forwarded to other brokers until they get consumed. Assume you have to independent masters which share the same index directory. If the setup is correct only one of the master should be able to consume a message. If one master server goes down all messages would get consumed by the single remaining master. Remember, this are JMS specific details and has not really anything to do with Hibernate Search per se.
It might be easier though to just have a single master. If you decide for a JMS architecture you are already accepting a certain time delay between index changes and the ability to search for this changes in the slave. If you have some sort of monitoring software for the master it might be enough to say automatically restart the master. The JMS infrastructure will take care about the guaranteed delivery of the message.


Not sure what you mean here.

--Hardy

Q1. Another reason for using topic is because it allows multiple consumers. for my scenario, I ve 2 masters.

Q2. I would like to have 2 set of indexed file for the 2 masters (not shared). I was thinking of having 2 queues (if topic is not possible) to ensure that both master gets the messages. However, I supposed I cannot configure 2 jndi queues paths in the xml file(hibernate.cfg), right?
Thus, I am supposed to manually add to the queue, rite?

Q3. I understand that @MessageDriven only works in a J2EE container. Since my master is a standalone app, I need to code the retrieval of messages from the queue, and then to do a manual indexing, right?

The current JMS backend is written to work for a single queue. If you want to try to use a Topic approach you would have to write your own backend and directory provider. Have a look at the classes in org.hibernate.search.backend.impl.jms and org.hibernate.search.store.
If you are using two independent masters which are not sharing the same index directory you will also have to solve the problem on how slaves get their index copy. Per default slaves get a copy of the index via a file system copy. Currently you specify one single directory. How would that work in your topic approach? And how would you know that one master went down?

On trunk we added a JGroups based master/slave configuration option and there is an experimental Infinispan directory. Maybe you should have a look at these options.

--Hardy

Thanks! What does slave send to the queue for the master to index? Does slave send the pk or the object itself?

I have changed the app to sends the object to the topic for the 2 consumer to index. However, when the consumers attempts to do a manual index after retrieving the object from the topic, it hit on "TransientObjectException: The instance was not associated with this session".

They are obviously from different session at the point creation at slave and index at master. how to resolve this error??

The slave node sends its work list to the JMS queue. This contains the lucene document that needs to be indexed with additional data.

I was wondering why you need to have 2 master nodes? Is this so you can have a back up when the primary goes down? In my previous project we spent some time discussing this issue and how to implement a back up master. Eventually we decided that in the event the master goes down not all will be lost. Firstly the messages will be on the queue so once the master is back up again it will start indexing the payload. Secondly if the queue went down then we had the database so we could reindex overnight or at a time when the application wasn't being used.

As Hardy there is the JGroups or I did a directory provider for GigaSpaces.

Not sure if that helped but it would be useful to work out your requirement.

Cheers
Amin

So objects cannot be used to pass to the master via JMS topic.

My architecture is similar to a multiple single-HS. But i need to separate the web tier with the data tier. Thus, the indexed files needs to be stored on a separate data server.

The data tier is master in my context and i do not have any slaves. The master service the search requests and I also cannot afford much delays in the copying from master to slave node.

Instances of LuceneWork instances have to be placed on the queue. One of this instance contains the Lucene document to be indexed, the class being indexed and the (serializable) id. Since it seems that you switched to topics instead of queues, I am wondering how you've done this? Did you write your own backend and directory provider?

It think you are trying to put your actual instances onto the JMS topic. Since your subscriber is running in a separate Session (persistent context) you cannot just take the instance from the queue and index it. You would have to reattach it to a new Session on the server. So it does not make sense to pass entity instances. You might as well just pass the id and retrieve the to index entity again on the master.


I don't understand. So you seems to have a webapp setup and I assume you want to be able to search within the webapp. To do this you need an index on the webapp machine.


So you have multiple webapps, but search requests you want to pass to a single master? What do you really try to achieve?

My main aim is to reduce the number of database connection as it may affect the system performance. Could you enlighten me on the disadvantages of reattaching to a new Session?

I did not want to amend hibernate search codes to ease future maintenance. Instead I listen to the "on flush entity" event and (webapp) send the objects to JMS topic manually.

The data server is a standalone app that listens to the JMS topic and manually index.. It also service all search requests forwarded from the webapp. Due to some constraints, the architecture is designed like that.

There is no disadvantage as such. However, reattaching a new Session also means a new database access so there is nothing saved here.


Adding new directory and backend implementations are actually envisioned extension points and we encourage people to go down this route. Under the light of future maintenance it is a good way to go.