Hi,

A bit of background first: I'm currently using Hibernate Search 4.2 in my Java web application and I deal with write heavy Lucene Indexes in which I store quite a bit of data. An indexing operation of single object stored in my biggest index takes around 1 second when using the directory-based indexmanager.

To improve performance, I switched the indexmanager to near-real-time and the performance improved leaps and bounds, but now I would like to implement zero downtime deployments using Tomcat Parallel Deployments (which allows me to have 2 different versions of the same application side-by-side in a single Tomcat) and I have found out that I can't use the near-real-time indexmanager anymore because it buffers the modifications in memory and avoids flushing to disk until the buffer ram is full or the application shuts down.

My question is: what alternative solutions do I have in this situation? I would like to keep the indexing process as synchronous as possible. I have taken a look at the code of Hibernate Search and I have seen that the there are different commit policies and a class called IndexWriterHolder that allows me to commit and flush writes, but I'm not sure if or how those APIs are publicly exposed. I've also seen that Lucene 4 implements concurrent flushing and I've discovered the max_thread_states flag in Hibernate Search 5 that allows me to specify the number of concurrent writer threads per IndexWriter, but I've never used it before and I'm not sure if concurrent flushing would help in my situation.

Any help is greatly appreciated. Thank you.

I might help a little.

What do you mean by zero down time with parallel deployments? Do you need a failover server? What is your architecture of main relational database application (UI) and Search application? Is it all embedded within same application?

Hibernate Search has a concept of Master Slave applications to serve on same index. Have a look that might be solution to your problem.

I couldn't get what difference will it make having both applications deployed in same tomcat?

Thanks for the reply. I have a monolith (a single WAR deployed on a Tomcat) connected to a Postgresql database and I use Hibernate Search with the Lucene backend for its fulltext search capabilities.

Right now whenever I deploy a new version of my application I have to undeploy the old WAR and deploy the new one, causing service disruption, even if it's just a few minutes. Tomcat has a mechanism of serving two versions of the same application on the same context path called Par allel Deployment (http://java-monitor.com/forum/showthread.php?t=1288). This would allow me to deploy a new version of the application without stopping Tomcat (new users would see the new version of the webapp, users that already have a session would see the old webapp until their session expired), hence achieving zero downtime.

The problem is that I'm using the near-real-time indexmanager for write performance purposes and having two webapps using the same index under theses conditions is not possible because any writes done by webapp A wouldn't be seen by webapp B (or the other way around) until all the commits are flushed from the in-memory buffer and that can take some time.

I'm trying to keep my indexing process as synchronous as possible, because in several parts of my application, I read the data stored in my fulltext index instead of querying the underlying Postgresql DB.

That's why I asked if upgrading to Hibernate Search 5 (which uses Lucene 4) would help, because the performance problems I originally had stemmed from the slow flushing times of Lucene 3 and I saw that Lucene 4 implements concurrent flushing (http://blog.mikemccandless.com/2011/05/ ... cenes.html)

Hi,

From documentation:


Not sure how to address your specific situation, as NRT index is not shared or clustered s per documentation. Few hints I can give:

Try to solve index write performance issue with different approach such as:

1. Use standard Index Manager but put update yourself in separate thread (transaction) and let your main transaction to succeed. This will keep your indexes on filesystem consistent to maximum.
2. Might be looking at in-memory backend infinispan can be an option. As in memory writes can be fast! and infinispan can off load fully or partially to secondary storage

In above both cases - if needed you can cluster two applications as explained in Hibernate Search documentation.

I solved such issues by having a separate search application that can receive updates (through JMS or SOAP or REST) and serve search requests through SOAP/REST etc. This keeps main application independently maintained and deployed. This might not be best fit for your situation - simply another option to consider!



I am guessing all what I can think of your design, so you need a good R&D and decide what is better for you. OR let's wait for Hibernate Team to help on this.

Hi, interesting question. I replied on SO but let's use the forum to discuss options.

First, let me explain some basics. Apache Lucene will never allow to two IndexWriters to modify the same index: one IndexWriter needs to own the lock on the index, and when it does any other IndexWriter we attempt to open will throw an exception.

Hibernate Search can use the IndexWriters in several different ways, but we need to respect that basic rule.

So in your use case: application A is old and has active users, we should assume some index writes are still necessary, partially caused by previous actions which haven't been completed yet, partially because of current activity, so application A needs to be holding the index lock.

What would you expect application B (the new one being started) to do? We could enhance Hibernate Search to allow completely booting up even if the lock is not available yet, and buffer the writes but this would introduce various problems:
- Obviously the buffer will need to be limited, so what would happen when the queue is full? Would it be ok to block?
- Second, queries on application B will not be able to "see" the changes which have been buffered and not applied yet.
- Most tricky, the two applications would be updating still the same RDBMs in parallel but applying all operations from two independent queues: order of updates would not be guaranteed! We'll end up with an index which is not guaranteed to be representing the database state.

So it looks like such an approach wouldn't be safe. I'd recommend the shared "index master" architecture: you could use a synchronous JMS master queue (since you want things to be synchronous). Then the only missing step would be to find a way to configure the master node to use NRT and get a reference to the NRT IndexWriter to run up to date queries w/o flushing. This would be trivial by running the queries on the "index master application".
I'm familiar with WildFly: on that application server I'm confident I would be able to isolate the application's classpath and still share a service. Do you think we could figure out a way to get such a reference and yet keep the application deployments isolated on Tomcat?

As an alternative, you could try migrating to our latest Hibernate Search 5.6 or 5.7 versions: they can use an Elasticsearch cluster as an optional replacement of having the Lucene index, so both applications could write/query in parallel.

Thank you very much for taking your time to write such a detailed answer. I have just upgraded my Hibernate Search from v4.2 to the v5.3 following the migration guides (I didn't upgrade to the latest version because it would require me to upgrade to Hibernate 5 and Java 8 as well, and I'd rather do as little work as possible to minimize testing and potential breakage).

I ran some simple performance tests against the webpp to test indexing performance. The test were done on my development machine, running 100 requests/second for 20 seconds against an endpoint that indexes different objects.

Here are some of the results I've seen so far:

Hibernate Search 4.2 using the directory-based indexmanager
The red portion of the plot are faling requests (HTTP code != 200).




Hibernate Search 4.2 using the near-real-time indexmanager




Hibernate Search 5.3 using the directory-based indexmanager




2 Tomcat WARs in parallel deployment mode using the same index. Hibernate Search 5.3 using the directory-based indexmanager
hibernate.search.default.exclusive_index_use=false




As you have said @sanne.grinovero, since two IndexWriter can't write concurrently to the same index, the performance takes a nosedive when 2 applications are writing to the same index and Hibernate Search has to release and reacquire locks.



I've just read the documentation about the JMS master-slave setup, but I'm unfamiliar with this approach. What would the lifecycle look like? In my situation, do I have to configure both applications A and B as slaves and create a third daemon application (master) that handles the incoming updates from the queue? How would this approach result in synchronous updates, since I have refresh the slave indexes? I'm a bit confused sorry.

Your understanding of needing "a third daemon application (master)" is correct. But then rather than copying the index periodically back to the slaves, I would see if you can manage to run the queries on this third application as well.
I don't see any alternative, as if you don't run the queries on the same node as the writer we'd at very least need this writer to flush all changes after each commit, which would get you the performance of non-NRT.

I'm not suggesting it as an ideal solution, as it wouldn't be possible to make changes to this "third application" w/o downtime, but it might be a better scenario? One benefit is that you might be able to design your other applications to be able to deal with the "search component" to be unavailable for short times when you have to update that component.

Two other alternatives:
- use the Elasticsearch integration
- switch to the non-NRT configuration and compensate with other tuning options.

For example since Hibernate Search 5.5 the usage of @org.hibernate.search.annotations.SortableField might give you a good performance boost, although on query execution only (I understand you're limited by index writing? Not sure if your graphs show a mixed workload, in that case tuning queries might help reducing IO and hence benefiting writes). Incidentally this version doesn't require Java 8 yet, that requirement is new since Hibernate ORM >=5.2 and Hibernate Search >= 5.7, previous versions should work fine on Java SE 7.

One more idea: since Hibernate Search 5.0 if you enable "async indexing" there also is a configuration property index_flush_interval which allows you to bound the async flush period.

I understand you said you want the service to be "as much synch as possible", but if you can tolerate enabling async with a rather small max delay, then the Hibernate Search flushing engine might be able to speedup the index writes significantly. For the record, many Lucene based services (like Elasticsearch) default to a rather large 5 second flush cycle as that has great improvements on the write throughput. I believe setting just 200ms in your case could help.

The tests I ran were 100% write-only.



Thanks for the clarification, I get what you mean now. You mentioned a way to do this in Wildfly, could you give me some pointers or keywords so I can look and see if something similar is possible in Tomcat? I like this solution even if I might not implement it right now because it sorta decouples the search and indexing service from the main monolith, which is the general direction I'm heading towards with regards to the application's overall architecture.



How stable is the Elasticsearch integration at this time? Is it ready for production? Moving to Elasticsearch or Solr is on the TODO list for me due to the additional search capabilities that they offer, so I'll keep an eye on this.





Actually, any non-NRT configuration suits me well as long as indexing time is reasonable and doesn't hurt the user perception (slowness). I can also tolerate some small delays (a few seconds) that's why I said "as synchronous as possible" instead of "perfectly synchronous". This might be one of the cases where a best-effort approach would be good enough.

The async indexing option with a bounded flush period sounds very attractive to me. I'll run some more tests tomorrow. Thank you!