Hi Guys,

I currently have a OutOfMemoryException occuring when searching is taking place. I have failed to replicate the cause locally but the problem arises after a period of time 1-2 weeks in our test environment.

Stack trace when OOMException occurs looks like so



My index sizes in total are roughly 1.9g, so small in terms of lucenes capabilities. My Vm has 1g allocated to it and is set up to use the FSSlaveDirectoryProvider. I took a closer look at the heap using MAT and I can see who is hogging the memory. The org.apache.lucene.search.FieldCacheImpl is accounting for roughly 850MB of the total VM.

Im just looking for any tips on a way forward with this one, any suggestions of a cause/solution??

Thanks guys,
LL

Hi, the FieldCacheImpl is known to take a lot of memory; depending on your index structure and what you indexed it might grow to a sensible % of the whole index.
Are you sure this is a memory leak, and not a constant requirement? you might want to check the behaviour using a larger heap.

If you really find a memory leak in FieldCacheImpl there's not much we can do, you should write to the lucene mailing list.
BTW which Lucene version are we talking about?

The data isnt exactly what i would call big, there is alot of records but generally the field data is small, mainly strings 15-30 characters long on average.

Can you tell me more about how the FieldCacheImpl works........what gets stored and when does it get released?? What happens for example when the slave directory switches over to the new index, does fieldcacheimpl still store all the old values from the old index or does it release them? My hunch is maybe after switching X amount of times that maybe the FieldCacheImpl still holds the old index values in memory and this builds up over time?? It is just a hunch/theory at the moment and i need to prove it.


Im using lucene 2.9.2 wtih HS 3.1.1. I know there is newer versions and plan to step up but would like to solve this problem first or at least find out the reason for it.

It's used mainly for sorting purposes, so if you do sort it gets used more. Also it depends on the number of unique terms you're sorting on, so it depends on the analyzers, how you split sentences and how many unique terms this project generates (always good to tune the analyzers).


Each FieldCache is local to the currently opened IndexReader, so as long as you don't forget to close indexReaders you shouldn't have trouble. Search is handling IndexReaders lifecycle in sane ways AFAIK, but you might be asking for readers using the low level API and not closing them all?
Which ReaderProvider are you using? the default one should be your best choice.

Hi Sanne,

Yes we do sort, if we do a sort query will the FieldCache load up all the unique items all in one go or does it do it over a period of time?? So if it is the same query but different input to the query, it shouldnt make the Fieldcache grow bigger after the first query because all the unique terms should be loaded from the first query, right???


Can you expand on the tuning side here, any examples or links?? Couldnt find anything on web about it.


Using default.

Thanks Sanne,
LL

partially right, assuming the new Query didn't hit any term, so that the where all loaded by the first query. It might hit new terms, which make the cache grow larger as it initializes them.


sorry my fault, "tuning" is not very appropriate in this case. I meant to carefully choose and configure your analyzers: depending on how you analyze the text you might produce a larger set of unique terms.
So for example if you don't lowercase your text, for each word you'll have the different combinations of cases found in your document for each word, while if you lowercase it's going to contain only the lowercase representation.
If you do lowercasing, remove simbols, and apply tri-grams (chunking words in groups of three characters), you'll have more terms per document but likely a reduced number of unique terms, so consuming less memory for sorting purposes.

correction: I nearly forgot, you won't be able to sort on fields using tri-grams as the field must contain a unique term to make sense. but the example is still fine in the sense that you could apply some analysis to reduce the set of unique terms.

This is more a subject for the Lucene mailing list, there you'll find better insight on these aspects; all I can say is that it's possible that you're not hitting a memory leak but just having an higher requirement in terms of memory in your use case of Lucene; in case you can't try with a larger heap you should try changing the way text is processed, or limit the sorting use cases on some fields you can predict to be of limited cardinality.

Hi Sanne,
I have setup a load test locally and query my index with the same query but with random input values, after each GC my VM size stays roughly around the 40MB mark. No updates to the index were done. I then introduced updates to the index etc etc and continue with the same query I know notice that after a while that my VM after GC is 80MB.

I had a look at the before and after picture over a period of time (12hours), the difference is the size of the search.FieldCacheImpl, looking at the different hashmap buckets and there entries in them. It looks like there are duplicate entries in comparison to how it looked when no switching of the index goes on.

Question for you, when the swap occurs from using "current1" to "current2", how does HS know the swap has occured and close the old readers?? From what I can see all HS has to work off is the interface DirectoryProvider and that has start(),stop(),getDirectory(). I know you said it works off a lifecycle basis so how or when does the underneath index reader get closed for the old directory??

I adapted the FSSlaveDirectory slightly just to see what happens when we do the swap over for the new index. It looks something like this


Ive cleaned out unwanted parts but you should get the idea, when the marker changes we close the current directory and open up the new one. I just wanted to see what would happen underneath the covers. What I would have expected is the heap to grow and have a lot more frequent GC cycles but what happens is it grows and grows and eventually blows up because nothing gets released even though we closed the old directory.

How would you have expected the above to run??

Cheers,
LL

Hi Sannne,
Could you point me in the right direction a bit please, on the switch over if i proactively cleaned up rather than wait for "lifecycle" to finish up. What would I need to do?? I see that the FieldCacheImpl has a purgeAllCaches() which would be good to call on the switch over but dont know what effects it might have if any??

Do you know how i could get hold of it through the HS API??


Thanks,
LL

Hi,
I'm getting to an idea, could you please verify if you still have the problem using the "not-shared" ReaderProvider?
you'll experience lower performance, but it might fix the problem. I'm not sure of it, sorry I'm guessing, to write a proper test for it I'll find some time next week.

My idea is that the new internal caches of Lucene 2.9 don't work properly with the way we reuse buffers in the SharingBufferReaderProvider when using the master/slave directories - if you could confirm this that would be great and I think we could propose a solution.


From the "native" chapter in the reference docs you can find hints about getting access to all Lucene APIs, we don't hide it in case of advancaed need; unfortunately I'm not sure of Lucene's API is exposing this. Also look into ImmutableSearchFactory, you should be able to access more non-public-api services (but no backwards compatibility is attempted)

I will set this reader and let you know what happens.


A quick and dirty way would be to just to adapt the SlaveDirectoryProvider to check if the marker files has changed and if it has then close the current directory and open the new directory. Repeat the cycle and watch what happens. Something similar to whats posted above, would get you started fairly quick.



I am using the master/slave paradigm but without JMS. I have my own custom updater who updates the master index. The rest follows the master/slave paradigm.

Out of curiousity, did you try out the JGroups baased backend?
Also if you plugin an Infinispan Directory you won't need the Directory switch anymore and greatly improve performance and ease of setup - that's the goal we have for next version.

Hi Sanne.
Tried the not shared reader and that is actually worse. I made 5 queries and i get a OOM exception. So it kind of fast forwarded the problem. Please note i am using lucence 2.9.2 with HS 3.1.1. I am going to step up to the latest versions of HS and see if the problem persists.

A quick look at my heap and you can see who is hogging the memory

Class Name | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------
org.apache.catalina.loader.WebappClassLoader @ 0x51110340 | 168 | 867.624.864 | 93,49%
|- class org.apache.lucene.search.FieldCache @ 0x8f30f8b0 | 48 | 861.936.056 | 92,87%
| |- org.apache.lucene.search.FieldCacheImpl @ 0x54d3e3f8 | 16 | 861.935.928 | 92,87%
| | '- java.util.HashMap @ 0x54d3f038 | 40 | 861.935.912 | 92,87%
| | '- java.util.HashMap$Entry[16] @ 0x54d40b00 | 80 | 861.935.872 | 92,87%
| | |- java.util.HashMap$Entry @ 0x54d37f50 | 24 | 861.933.992 | 92,87%
| | | '- org.apache.lucene.search.FieldCacheImpl$StringIndexCache @ 0x54d3e3a8| 16 | 861.933.968 | 92,87%
| | | '- java.util.WeakHashMap @ 0x54d3efb8 | 48 | 861.933.952 | 92,87%
| | | |- java.util.WeakHashMap$Entry[16] @ 0x54d40a80 | 80 | 718.278.240 | 77,39%
| | | |- java.util.WeakHashMap$Entry @ 0x54d37e78 | 40 | 143.655.632 | 15,48%
----------------------------------------------------------------------------------------------------------------------------------

Hopefully you can replicate it on your side.


No I havent, time is against me im afraid so i wont be able to give it a go.

Just another thing, im using it inside tomcat!!

I just seen http://opensource.atlassian.com/project ... SEARCH-314, although im not redeploying or anything like that but could be related.

Are you sure you close the indexReader? Yes please try HS3.2

The fact that your'e hitting OOM sooner using this implementation makes me thing that you just don't have enough memory, not that there's a leak, bringing up the first thoughts that you might need much memory to do sorting.

No, if that is a problem that's really only limited to the redeployment on the same tomcat.