I recently hit a huge speedbump on my way to using oscache in a cluster with Hibernate. I don

You are welcome to improve the OSCache support. In fact, we don't have time to manage two enterprise-level cluster caching systems, so our work is focused on JBossCache (which is maturing nicely, btw).

Sorry, wasn't trying to belittle JBossCache. Bela, Ben, and others have been great help in getting me set up with it.

But there is a need for greater invalidation flexibility and clustered invalidation instead of full replication. OsCache also has some web features that make it worthwile even though I personally don't take advantage of them.

I have started to contribute heavily to OSCache but when I hit this problem I had to switch. I'm mainly concerned with the performance of replication under heavy load (> 100,000 users). It would be great to have 2 cache options available (I think swarmcache is not an option at all) when we start load testing our app. I am thankful to your team to be able to switch fairly easily between 2nd tier caches.

Btw, this post originally went to the oscache list but since I never saw this type of warning about clustering on the forums, I thought it best to cross post.

Forwarding to the forum since the devel list won't want this traffic but other users might:
------------------------------------------------------------------------------------
Frankly, I'm surprised that your application even starts up. Sure, ours did too sometimes but mostly not. The source of the problem could be debated but I think a JChannel per region/class is just plain wrong.

OSCache uses the term group instead of regions. In fact, groups may be new in oscache 2.1. Regardless, the hibernate provider for oscache should not create a GeneralCacheAdministrator per region. It is quite apparent that this class is meant for an application scope.

We have about 200 regions in our cache and that many channels just killed our app. When we switched to JBossCache, clustering worked beautifully. In fact, that is how I noticed that the problem of too many channels. But there is a performance hit, in our case huge, for using a replicated cache. I wish JBC had a feature to only cluster invalidation but AFAIK it does not.

So I rewrote the provider to use a single GeneralCacheAdministrator and use the groups to serve as regions. The only thind lost is the ability to specify capacity per region. You can still specify expiry policy per region. And OSCache's expiry policy features are somewhat more advanced than JBC.

Hope that clears things up.


------------------------------------------------------------------------------------
Forgive me if the list is the right place for this, but a co-worker forwarded this to me and I'm eager to get an answer. I may be interested in your patch.

What are the symptoms of your problem with OSCache in a cluster?

We plugged it into Hibernate to replace SwarmCache. (We may be experiencing a memory leak in SwarmCache.) We have no trouble starting up with OSCache. It registers its oodles of Jchannels. Both nodes in our cluster receive messages from each other. Everything seems fine, BUT sometimes we will get OutOfMemoryExceptions. It doesn't always even seem to be related to running our application. (By the way, Jprofiler shows no obvious leaks with either cache provider.) I am not sure I can pin the bad behaviour on the cache, and that's why I'm interested in your symptoms.

I'd like to hear more from you on this subject. I'm disappointed that it isn't covered in more detail on the Hibernate site.

Deleted everything not related to OSCache - If there was anything meaningful in it, please repost properly formated - michael

I just recently tried several configurations using several different cache solutions for a clustered cache. I found the JBoss TreeCache to be overkill at best and challengingly complex at worst. Fully transactional replicated caching may well be too much for many scenarios where simple distributed cache invalidation is all that is required. Unfortunately JBoss TreeCache does not support that. SwarmCache seems to be a dead product and the Hibernate documentation clearly states that OSCache cannot be used in a clustered environment.

What to do. What to do.

Well, to make a long story short, thanks to the work of several other developers, OSCache seems to be a viable clustered alternative, though it does not support distributed transactions. Basically, here's what you can do:

1) Get a 2.1 version of OSCache (at version RC1 as of this post) . You can download it from the Java.net site: https://oscache.dev.java.net/servlets/P ... cumentList

2) Modify the OSCache and OSCacheProvider in Hibernate to take advantage of OSCache's architecture. Basically the current OSCache implementation uses a single OSCache GeneralCacheAdministrator for every Hibernate cache region. This will cause out of memory errors among other things when using it clustered. You need to use a single GeneralCacheAdministrator for the entire cache, and then use an OSCache "group" for each region. There are code fragements available on the web and in the OSCache mailing lists.

3) Use your new provider and cache in your Hibernate configuration.

We're currently performing load testing on the customized cluster, so I don't know that everything will work perfectly yet, but it looks very promising so far.

BTW, why doesn't the OSCache team simple include a "new and improved" hibernate cache provider as part of their distribution?

Seems to me that now that Hibernate is established, there's no reason for the hibernate team to maintain all the cache system plugins for all the caches out there. The commercial systems provide their own connectors, makes sense that the open source crowd does to. It's ok for Hibernate to include some basic connectors, but it's kinda like the JDBC-ODBC bridge of old. Sure it comes in the box, but you shouldn't use it... you go get a high performance driver from the vendor who produced your DB or a third party vendor who specializes in JDBC drivers. The cache connector should probably work the same way. That will free up the hibernate team to work on the core product, which is where we all want them anyway.

Please post the modified cache providers on our JIRA system or on the Wiki. Thanks.

I disagree that the treecache is hard to use but agree that it cannot be used in all circumstances. I would love to see some #s about how much replication impacts performance. But with 200k users and expecting at least double that in the future, we think oscache is the right cache for our app (at the moment).

Thanks for the great summary. I would love to hear more about your success or problems with hibernate and oscache. We run a 30 node cluster and invalidation (due to the great work by Bela Ban and others) works almost flawlessly. Actually, the biggest problem we have encountered is the difficulty in restarting tomcat because of busy threads.

OFFTOPIC: Currently our largest issue (due to the load) is that the database (Oracle in a RAC) has difficulty keeping up. I would like to give more power back to the db developers to control the sql ran on their machines. It is my intent to add Stored Proc support to the query engine as soon as possible. I do not yet know how I will do that without making it oracle specific. I do not want a hack but we need this feature probably more than any other.

I once again express regret that the oscache provider will not be fixed in the hibernate 2.1.x line. If anybody needs it, they can search the oscache mailing list archives. I have considered the possibility of distributing it with oscache in the future.

I just opened an issue in JIRA according the performance improvements of Andres changes:

http://opensource.atlassian.com/projects/hibernate/browse/HB-1326

Thanks Lars. You have been quite supportive. Although I think performance improvement is a bit of a miscategorization given the fact that the current OSCache provider creates application caches for every single entity. To me that is a bug.

Please don't throw words like "bug" around, unless you actually understand the reasoning behind the current design. We really hate that.

If you are referring to the design of the OSCache provider, Hibernate OSCache class, and OSCache, I do.

I am sorry if you are offended but I think it is important to clarify the changes that I have suggested and posted. They do not impact the design of hibernate at all. THey solely deal with the way OSCache was being utilized.

If you were to inspect the source, it would be obvios that only one GeneralCacheAdministrator should be created for each application. That is it has application scope.

Regions are managed through groups in OSCache. I modified the Hibernate OSCache class to set its group (ie region) when created but not to create an additional application scoped cache.

The old (and I firmly believe) incorrect way lead to jgroup cluster channels being created for every entity because one channel is created for every application-scoped cache administrator.

bug is not a bad word :)

but performance improvement would be a misnomer.

bug is an horrible word for people who do their best to have the best code ;))))

If anybody, you guys/gals? should not be afraid of it.

Hibernate is some of the best written code I have seen. Better than most of mine but of course our boss does not always let us write the cleanest code. That is why I like contributing to open source, the freedom to write gode code.