I'm wondering if it's possible for the to have the master be switched at runtime depending on the fact that the previously selected master has died. how can i do this via configuration???
this is a very advanced topic and outside of the scope of Hibernate Search. There is no option provided in Search to do such a thing.
In order to do that you probably need to look at several things. First, you need to configure your JMS setup to have a persistent setup. This means that messages gets stored so that in case the JMS setup fails or there is a power failure you are not loosing messages. Clustering JMS itself is a topic in itself.
The second part is clustering the master server. Given that the index is available over a shared file system you can have multiple masters. You just have to make sure that only one is active at the time. One way would be pacemaker (http://www.clusterlabs.org/
). The idea is to setup a heartbeat and in case of the failure of the main master move the JMS queue consumer to one of the standby servers. You need to write some scripts/apps which allow you to start/stop message consumptions and configure these as "resources" in pacemaker.
As I said, this is outside of the scope of Search and very much on network/server level.
Depending on your needs it might be just enough to monitor the server (nagios etc). If the server goes down, you fix the problem and bring it back online. Once running the server will consume the indexing work in the queue. Given that the master/slave setup has some delay anyways in regards when new content becomes searchable this might be a workable solution.
I hope this gives you some ideas on what to look at to make your setup more failsafe. Feel free to share your experience and setup.