Hi,

I'm currently using HSearch 3.0.1.GA to implement search on a site. I am looking to use the Filter mechanism to speed up searching, however the filter caching is not working currently. I believe a fix for this was deployed as a response to HSEARCH-136, however I don't think it's worked 100% as expected.

The problem is that ReaderProviderHelper.buildMultiReader(...) creates a new CacheableMultiReader for each query, and this reader is used in CachingWrapperFilter as the key to the WeakHashMap that caches the BitSet.

Using the SharedReaderProvider the underlying readers are always the same (for the same query), but because a new instance of CacheableMultiReader is created for each query ,and CachingWrapperFilter stores its cache in a WeakHashMap, the CacheableMultiReader goes out of scope immediately the search is complete and by the time another query is run it's been garbage collected. As the CacheableMultiReader was the key in the WeakHashMap the entry caching the BitSet is also gone.

It seems that the only solution to this is to cache the instances of CachingMultiReader keyed by their underlying IndexReaders to ensure that instances do not go out of scope on a per query basis.

It looks like the problem is quite deeply rooted in how the SharedReaderProvider performs its Reader caching, and whilst I'd love to fix it the problem appears somewhat non-trivial as the simple solution, of storing hard references to the Readers is going to cause memory leakage in production.

Cheers,

Nick

Arggggg. I did no foresee this one either. I thought they used SoftReferences
OK so let me detail some solutions:
- go back to a specific implementation of CachingWrapperFilter that read the underlying readers and cache by the first element: this was a workaround proposed here http://forum.hibernate.org/viewtopic.php?p=2369054#2369054

The problem with this solution is that caching by the first element is wrong, it should be cached by "all elements" in a way. This is probably possible by storing an HashMap as a value containing all readers having their first elements as key.

- use the Lucene RemoteCachingWrapperFilter: this cache does not use WeakReferences but rather a hard limit and a Thread pool cleaning the map. I am not a big fan of it especially since the FilterManager is statically defined and as any hard limit cache it is subject to OOME
But this should solve your problem in the mean time

- introduce a @FullTextFilterDef.enableCachingWrapperFilter
whose value could be AUTOMATIC, TRUE, FALSE
HSearch could then use it's own implementation of CachingWrapperFilter
AUTOMATIC would use a wrapper when cache=true is used, and not use one when cache=false
This solution is my favorite so far, I think I will introduce it in 3.1

What do you think?

See http://opensource.atlassian.com/project ... SEARCH-174 and http://opensource.atlassian.com/project ... SEARCH-173