I'm investigating an OutOfMemory error in our application and I've found that hibernate-search seems to be at the center of the issue. The JVM was kind enough to dump an hprof file when it ran out of memory and digging around has revealed that we have 152 instances whose field points to a map containing 2-4 byte arrays between 1MB and 4MB in size. The net result is close to 1GB of memory used.

Under normal conditions, however, there nowhere near this level of memory used.

I've told hibernate search to use a of 50 and is 1. The webserver uses a maximum of 90 DB connections at once. The JVM dump only shows 30 instances of Thread.

I haven't been able to reproduce this in a laboratory environment so I can't determine exactly what is causing this pile-up of readers and their associated large arrays. One possibility could be that a long-running operation is holding a lock on the search index and then these other operations are waiting for that one to complete, but the system runs out of memory before they have a chance to do so.

Any idea on why there could be so many of these readers left around? Based on the way things are configured, I wouldn't expect so many readers to be open with so few threads.

Also, any thoughts on how to eliminate all of these arrays altogether? I thought maybe putting NO_NORMS on all my fields might do the trick but I noticed that "_hibernate_class" is still creating a norms array when entities are deleted and I don't see any way to delete the norms for that field - the field setup looks to be hard-coded for both the class and DocumentId.

A bit more digging reveals that the class CachingWrapperFilter is keeping references to the IndexReader in its cache up to the strong ref count. There doesn't appear to be any code to pro-actively clean up the normsCache in the CacheableMultiReader so the normsCache is kept around for the lifetime of the CachingWrapperFilter along with the reader.

I guess I'll have to see whether anything was filed in JIRA about this and get rid of all uses of CachingWrapperFilter in order to avoid this over-use of memory.

I would still like to get rid of all the unnecessary norm arrays, perhaps I should submit to JIRA that the code to create the hibernate built-in fields should omit norms.

DIdn't find it in JIRA so I added it, sorry if its a dupe or already fixed in the next release (I'm working off the last GA release):

http://opensource.atlassian.com/project ... SEARCH-469

thanks, that's a good spot!
I'll also open issues to remove norms from classtype and id, they are definitely not needed.