I am in the process of setting up a master/slave configuration per the recommendation in the book and online docs. I do have a couple of questions.

1) Why do so many messages get put on the shared queue? Is that normal? My setup currently is using ActiveMQ + Tomcat + Hibernate Search. This is a sample of the kind of messages I am getting very frequently. The client node is not even up yet and there are no changes in the database, but yet these messages are constantly getting pushed to the queue.

15:00:41 DEBUG MasterBootstrap.onMessage - Processing message ActiveMQObjectMessage {commandId = 5, responseRequired = true, messageId = ID:Macintosh-2.local-54700-1300473739449-0:180497:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:Macintosh-2.local-54700-1300473739449-0:180497:1:1, destination = queue://HibernateSearchController, transactionId = null, expiration = 0, timestamp = 1300474841107, arrival = 0, brokerInTime = 1300474841107, brokerOutTime = 1300474841108, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@4d5efe45, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}
15:00:41 DEBUG MasterBootstrap.onMessage - Processing message ActiveMQObjectMessage {commandId = 5, responseRequired = true, messageId = ID:Macintosh-2.local-54700-1300473739449-0:180498:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:Macintosh-2.local-54700-1300473739449-0:180498:1:1, destination = queue://HibernateSearchController, transactionId = null, expiration = 0, timestamp = 1300474841111, arrival = 0, brokerInTime = 1300474841111, brokerOutTime = 1300474841111, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@26603074, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}

As I write, 200K+ messages have been written and consumed from the queue and nothing has changed in the database. I also see most of the message sizes are "0" signifying no change I presume.

Aren't messages meant to be pushed to the queue ONLY then there is a change on the slave node(s)?

2) I understand Hibernate Search keeps 2 copies of the index on the master node, and then copies the stable/active version to the shared location for slave nodes to consume. I do see the two directories created on the master node (i.e. <indexBase>/1 and <indexBase>/2) but pointing Luke to any of these locations is giving me "0" document count. I have even pointed Luke against the "shared location" of the client and still the same result persist. Am I missing something? All I want is the location when the final build index is stored so I can see what is built and query against it.

Thanks in advance

Hi, welcome.
1)that's something with the ApacheMQ implementation details, I think it'sconnecting to the other nodes as part of the startup procedure. You're right with your expectations, Hibernate Search doesn't push any change to the queue unless database updates require so, still the MQ might need to ping the other nodes regularly to verify everything is fine.
You might want to ask the ApacheMQ people, or read the MQ manuals, or change JMS implementation (any will do).
Hibernate Search also provide a JGroups implementation as alternative for JMS (doesn't provide the same guarantees as JMS though, but might be interesting if you want to trade that for higher performance)

2)Seems your index is empty.
Did you rebuild the index, or update some entities to trigger indexing? The indexes are initialized empty, if you have an existing database you have to start the MassIndexer on the master to create the initial index.

Thanks Sanne, I'll investigate further.

1) Seems strange ActiveMQ will be connecting to other nodes. I'll try your suggestion of looking further into my ActiveMQ setup or trying a different jms implementation. Strangely enough, I started out with an empty database and when I peeked at the ActiveMQ admin console, I could see there was one consumer (the master) and no messages to consume. Then I started a single client node and added an entity via a web interface. At this point, I expected to see one message get put on the queue and then get consumed by the master node. What I saw was a total of 107K+ messages get put on the queue over the course of 2 days (with the single added entity) and then get consumed by the master. As I said, I'll dig deeper and follow up with a post of my findings. I am aware of the other alternatives and another approach I intend to prototype will be to use Terracotta (or infinispan) to sync changes. I wanted to explore this route first since it was the recommended approach by the book.

2) I did build the initial index with MassIndexer (to prototype, I just have 3 entries in the database) and still the index is turning up empty. That also seems strange, because if I build the index without the master/slave configuration, like I have often done, things turn out OK. It seems something to do with this whole configuration set up. I'll probe further.

OK, I investigated further and the ActiveMQ settings all checked out as I suspected. The problem was with the configuration I had on the master node. It also specified a *jms* backend (which should only be present on the slave node) and thus, the frequency in the messages placed on the queue.

Once that part was taken care of, things worked as advertised and issues #1 and #2 I listed in my previous post have gone away. I am also still yet to try out the in-memory approach with Terracotta/Infinispan. Now that this is behind me, I intend to investigate that really soon, as the latency of the master/slave + jms option might not work for the application in question.

Thanks for your help

great, let me know. I'm the author of the Infinispan Directory as well (yes it was designed for Search, as in our opinion the Terracotta one wasn't a good fit), so you could find help either here or on the Infinispan forums. But even if you try the Terracotta one, I'd like to hear about your experience.

Is there a version of Infinispan that is Java 5 compatible? From reading the docs, it seems Java 6 is a requirement. If that is the case, that is a show stopper because my application runs on an older version of Websphere (v6) which is only compatible with Java 5

no, sorry. it's strongly coupled to the highly efficient concurrency packages. btw Java5 is deprecated and unsupported by everybody I know of since some time already, does your management seriously consider it safe to deploy a new application depending on it in 2011?

I mean, I can understand that there are many apps still running which where developed in a pre 1.4 era, and are likely to continue working fine.. but when developing a new application I'd raise some concerns.

There's no way to make it Java5 compatible, it would need a lot of work and no benefit at all (besides being likely much less performant).

You know how these things go Sanne. Management is almost always slow to adapt to change. We have a mature production environment layered on websphere 6 with quite a lot of applications already on it. I doubt there will be infrastructure change just to accommodate this.

Thanks for your prompt response. Now I know my options.

right, sorry for the rant.

Going back to your other options, yes you can set the file copy period to be very aggressive, it's a sync-based algorithm which will only copy over the small changes each time. you can tune the index parameters to
A) avoid optimizing too often, as that merges the small files in a big one, making it harder to copy over all over network
B) there are options to make sure your index segments are more and smaller, could help as well

both options will impact search performance, you'll have to find a good balance. Anyway usually searching is so fast that you could afford even doubling times (I assume).

Sanne - new requirements came to light today and our client does want realtime search when updates are made to the database. Anything shy of 20 secs latency is unacceptable. So the only viable option now seems to be the in memory solution. Given our environmental constraint at the moment (ie not able to use infinispan due to WAS 6), do you have an option or suggestion for me?

BTW, I spoke to management and I was informed we are moving to WAS 7 later in the year. However until then, we are in construction phase and have to work with what we've got.

Thanks