We are currently using Hibernate/JCS in a production system with more than a 1000 users. The large number of users seems to have unearthed a few serious problems in JCS.

JCS IndexedDiskCache Deadlock
The IndexedDiskCache causes a deadlock. We are able to reproduce it every time in 2 to 10 minutes of load testing.

We have implemented a new IndexedDiskCache. It has the following improvements overJCS' IndexedDiskCache:

- clears the cache on startup
- reuses empty space within the indexed file.
- does not deadlock
- removes deleted elements from the in-memory index

It can be used by simply changing the class name of the jcs.auxiliary.DC.attributes attribute in cache.ccf.

LRUMemoryCache memory leak
We have spent 6 weeks trying to find a memory leak in production, which only happens in production. The VM heap should use about 200M but would grow over about 3 days to the maximum of 1600M and then give an OutOfMemory exception. It turns out it is our usage of JCS which is the problem. Now this is a direct use of JCS, rather than through Hibernate, so this issue belongs with the JCS mailing list. However because JCS is not a project in its own right but a subproject of Turbine, and I guess, of Hibernate. So I am posting this here as well.

We have found that setting the caches to clear using expriy in cache.ccf and not calling removeAll() solves the problem. The memory leak stops. However we also lose the ability to clear the caches programmatically.

We invoke the removeAll() method on CompositeCache, which in turn calls removeAll() on LRUMemoryCache. We end up with tens of thousands of objects of type MemoryElementDescriptor. These are not garbage collectable. This is the memory leak. It appears that the LRUMemoryCache is not threadsafe. We think that the doubly linked list is ending up as not linear, but with branches hanging off it. This was beyond the ability of our profiler to determine. We speculate that the issue happens in production but not in testing because there are more threads in production behaving in less predictable ways.

We plan to patch LRUMemoryCache and hopefully get it into JCS. If not it can still be used by specifying its class name for the jcs.default.cacheattributes.MemoryCacheName attribute in cache.ccf

I am interested in other's opinions on where we should put these fixes if we cannot get them accepted by the JCS project. Each is essentially a pluggable component for JCS and so does not necessarily have to be in the JCS source. But it would be useful for others who experience these problems.

Sorry, this has nothing directly to do with JCS. But since you've gone to the effort of contributing back your production experience with Hibernate...

Is your project currently on the "Who uses Hibernate" wiki page?
It's just that it seems like really good example of a medium-large project that uses Hibernate successfully (I hope). I understand you might not be able to put your project there for political/business reasons or whatever; but if you can, I'm sure the Hibernate Team would appreciate it.

Also, for your JCS issues and usage; have you considered contributing your experiences to the "JCS Usage in Hibernate" wiki page? It seems like really useful knowledge that anyone contemplating going to the effort of using JCS might find very useful.

Cheers,
Shorn.

Uggghhhh this is really, really bad - the sooner we get away from JCS the better.

So what are our options here:

For distributed caching things are looking good: we have a working patch for SwarmCache, and Bela and I will get JavaGroups TreeCache support working in the next week or so. This is great technology and will be a huge "selling" feature for us.

But we also need a good in-memory (ie. single machine) cache with proper expiry strategies. HashtableCache does no expiry so has only limited usefulness. Matthias Bogaert has recently added a patch allowing support for OSCache, so perhaps this should be what we emphasize from now on. I'll try to integrate this stuff today and do some proper evaluation.

I don't know if OSCache has the nice indexed-disk cache feature that JCS has however....

On the production experiences post, I will find out if we can go public with our usage. This could be commerically sensitive because the site had a thousand fold performance increase over the old asp version.

In terms of http://www.hibernate.org/61.html I was the original author of it. I have already updated it with warnings and links back to this forum thread.

I have tried to post a bug in the JCS bug system but it is down. I plan to put our patches up there. As each patch will be a pluggable component, there is no reason why they cannot go up somewhere else.

I would probably rather stay with the devil I know and continue to use a patched JCS. For others starting new projects another maybe OSCache is the answer. The commercial cache is too expensive for us and as you point out has features we do not need.

By the way the LRUMemoryCache is extremely non thread safe. I have been reviewing the code and there are lots of different problems I can see. The existing test for it is not multi-threaded so I can see how these issues could arise. I am also going to try to replace their doubly linked list hash table implementation with JDK1.4's LinkedHashMap. This gives us the benefit of production quality code at the core of the Memory Cache.

Perhaps we could add our new IndexedDiskCache and MemoryCache to the Hibernate project, or put them up on Sourceforge so that they are available to others.

Gluck,

Could you please update us on the issue. Did you ever get your patches into JCS? Any improvement in the multi-threaded issue, or the other issues you had? What did you decide in the end?

I am using JCS right now, mainly because of its nice admin servlet. Helps me to see what is being cached, how many objects, ability to remove objects from the cache from a web interface, etc. I also frequently change data on the db side, and need to be able to clear all my caches easily so I do not serve raw data. I am "lucky" that my web app has very few users so multi-threaded issues and memory leaks do not seem to ever have been a problem, but you never know....

I have been considering switching to OSCache, which seems pretty stable, but as far as I can tell there is no admin servlet available. (Or am I missing it?) Same with SwarmCache and TreeCache, none of those seem to have admin servlets either. (Any other suggestions?)

Thanks,
Daniel

Daniel

I started out with patches for JCS but then agreed with Gavin King to fork it into a new project. The new project is http://ehcache.sourceforge.net . EHCache bears little relation code wise to JCS but it has equivalent functionality for memory and disk caches.

EHCache is included in Hibernate 2.1. There are instructions there as to how to use it, as well as extensive documentation on the EHCache website.

JCS has been dropped from the Hibernate 2.1. We had severe issues with JCS and I do not recommend anyone use it in production.

EHCache has now been used for two months by many Hibernate users. We have run it in production for around three months. Some minor bug fixes were done at Xmas and are in EHCache 0.7. So, my advice would be to upgrade Hibernate to 2.1 and drop JCS in favour of EHCache. Note that Hibernate 2.1 now gives you a choice of caches, so there may be another one .e.g distributed that more suits your needs.

Sounds good. However I also need distributed functionality :-(

I also need the admin servlet I spoke about earlier. EHCache does not seem to have one but looking at the API it seems that an admin servlet would be possible. (Looking at the API for OSCache it seems it is not even possible, since there is no way to get a list of all the caches in the system. OSCache does not have a "CacheManager" like object, a central place where all caches are managed. So there would be no way to ask for a list of all the caches)

EHCache would be perfect if it had distributed capability. Are there plans in the future to support this?

There doesn't seem to be any caching system that has all these features. Hmm.

Thanks,
Daniel

Also, do you still have the patches you spoke about? Perhaps they could still be applied to the JCS code for the benefit of the people who still need to use JCS for whatever reason?

Also, I would be curious as to your reasons why you feel JCS can't be "saved". Looking at the site for EHCache three reasons are given, briefly the fact that some code is experimental, JCS has broad scope and the fact that JCS does not seem to be actively maintained. Personally I don't feel these reasons are much of a problem, except perhaps the last one.

JCS seems to have alot of promise as being a fully featured caching system, with perhaps a bit of work to iron out the bugs. Are there any other reasons you feel JCS is unsavable? Any other threads speak about this to avoid repetition?

I would be willing to put in some work to try to save JCS. Assuming we could apply your patches, I wonder what else would be involved to improve it, such as to make JCS faster (which you seem to also have an issue with on the EHCache site).

It would also be nice if JCS had a proper "release" with pre-built binaries to download. If no one else seems to be maintaining JCS perhaps I would be willing to take that over.

Perhaps this is not the best place to discuss all this. Let me know if we should take this discussion somewhere else.

One last note, please do not take this as me trying to belittle your work. EHCache seems to be a great solution for many projects, specifically for projects that only need a memory and disk cache. But unfortunatly it falls short if distributed functionality is also required.

I hope we can work together to iron this all out.

Thanks again,
Daniel

I do not think JCS can be saved. I spent a couple of weeks with the source code trying to do the minimum to it to have something that was correct.

There are many subtle threading and memory leak bugs in JCS.

Unfortunately the two patches I did only scratched the surface. Of course you are welcome to have a crack yourself.

In terms of scope JCS is a partial implementation of JCache, the JSR spec. It would be great if JCache was final and we had a reference implementation.

On being distributed, I will need to tackle this soon. I am not sure how I will do this yet. Because our prime motivation is to reduce database load (we have a real busy app), this will most likely me a master cache with pull replication to other caches. This will likely be done with JMX MBeans and EJB remoting.

I am also planning to do something to reduce the number of threads EHCache uses. At the moment each Disk Store uses its own shrinker thread. We have about 130 caches now with another hundred to go. On Linux each thread uses 512KB by default, so this is about 60MB of RAM. I am interested in whether others see this as a problem.

i meet the same as your ,i'm using Hibernate/jcs in my product ,when user more than 1000 ,my system tell me memory leak, i see U post ,but i don't know how to change attribute,
U write :It can be used by simply changing the class name of the jcs.auxiliary.DC.attributes attribute in cache.ccf.
i don't know how to opration
could you tell me
thx
email:liu_jie_bj@yahoo.com.cn

I tell U detail,My product use 50% memory of my System.when user more than 1000. then tell me 'out of memory' ,there are memory 1007140K free in my System,

We have made several improvements to JCS over the past couple of months. We

JCS is no longer directly supported by Hibernate, we switched to the EHCache rewrite of JCS.

JCS has incorporated several of the bug fixes that motivated the development of EHCache.

Since JCS is completely pluggable, any new memory manager, disk cache, etc. can be used. . . . A version of the style of memory plugin used by ehcache is in the experimental section of the JCS code base. We are also working on a Berkeley DB JE disk cache plugin.


Cheers,

Aaron