Hi,

I have a very weird issue with hibernate search (latest version with lucene 2.3.1). I have a RESTful web interface that exposes search capabilities. Some of them are using lucene.

I have performed load testing on this interface and the figures are good when running on a pure Lucene query (50 msec for 1 concurrent user, 150msec for 10 concurrent users)

Now, If I combine a lucene search and a database search (hybrid query), it takes 300msec for 1 user and 3sec for 10 users. When I run the profiler on the server, I see deadlocks on org.apache.lucene.index.SegmentReader.

Switching to "non-shared" index reader reduces those deadlocks but it's still very slow with 10 concurrent users (2.8sec).

The hybrid query does the following:

* Run a query on the lucene index
* Run a query on the database with some criterion
* Intersect the IDs that match
* Return the matches using select a from MyClass a where ID in (2,4,6,7...)

The result is limited to 20 results in my test.

Hi,

Do you mean the combined result, or are you limiting both lucene's index results and database results?


The complexity of such a query could grow without control and is not recommended...
use it only if you are sure that both sub-results are very limited.

I explained a better approach to this type of search here:
http://forum.hibernate.org/viewtopic.php?t=986180

Yes I know, I am the one who asked the question :)

The combine result is limited. The lucene search and the database search are not (and that's most probably the problem)

So I use a ScrollableResultset and I fetch only the primary key from the database. I am wondering though what you mean by "adding them all to a set". This solution may consume a lot of memory no?

Can you maybe explain in more details the algorithm?

Any idea regarding the deadlock on SegmentReader?

Thx,
Stéphane

So I am back with this problem. I have investigated the problem the whole day.

I have written my test in a simple java class that uses only the lucene API. This test has one IndexSearcher instance that 10 threads share. The 10 threads perform the exact same lucene query as the one I'm doing in hibernate search.

Result: 100 msec on average per client with 10 concurrent clients

With Hibernate search:

- Around 2 sec with the default reader strategy (lots of deadlocks on SegmentReader in this case)
- Around 1 sec with reader strategy as not shared

I tried to upgrade my code to use the lucene API directly as described in http://www.hibernate.org/hib_docs/searc ... ative.html

but it's still dead slow.

Does anyone has an idea of what could be wrong? If I don't find the solution, I wll simply rebuild a lucene service and use hibernate search only for indexing.

Hi,
I'm interested: the performance should be similar between Search and Lucene, either you have some configuration error or you found some bug.


Are you sure you are using the Shared strategy? This is default now, but early versions of Hibernate Search did use the No-Shared strategy.

http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-106

Which version of Search are you using?
Could you verify that you really are getting an instance of the Shared Reader?

I am using HS 3.0.1 GA and yes I have a shared reader. If I use a non-shared reader I don't have the deadlocks anymore but I think it's related to the fact that I am browsing a lot of results in the index tree. I can reproduce this with pure lucene calls now. I've asked on the lucene user list the best way to retrieve a lot of results from a query without locking the underlying resources.

The problem has been investigated and mostly explained. check here

http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-194