UpdateTimestampsCache and cluster invalidation

awerder · **Joined:** Sun Sep 02, 2007 1:20 pm **Posts:** 5

Query caching doesn't work properly with a distributed cache, if cluster invalidation is used. It does work if cluster replication is used.

The reason is, that UpdateTimestampsCache places the timestamp of the last change to any table in the cache. With cluster replication this timestamp is transfered to any node in the cluster. With cluster invalidation this is not the case.

The method UpdateTimestampsCache.isUpToDate(...) should return false if the given tables are modified since the last query. But it does return true, if there is no timestamp for the given table. That's the code I'm talking about:

Code:

//$Id: UpdateTimestampsCache.java 11398 2007-04-10 14:54:07Z steve.ebersole@jboss.com $
package org.hibernate.cache;
...
public class UpdateTimestampsCache {
   ...
   public synchronized boolean isUpToDate(Set spaces, Long timestamp) throws HibernateException {
      Iterator iter = spaces.iterator();
      while ( iter.hasNext() ) {
         Serializable space = (Serializable) iter.next();
         Long lastUpdate = (Long) region.get(space);
         if ( lastUpdate==null ) {
            //the last update timestamp was lost from the cache
            //(or there were no updates since startup!)
            //updateTimestamps.put( space, new Long( updateTimestamps.nextTimestamp() ) );
            //result = false; // safer
         }
         else {
            if ( log.isDebugEnabled() ) {
               log.debug("[" + space + "] last update timestamp: " + lastUpdate + ", result set timestamp: " + timestamp );
            }
            if ( lastUpdate.longValue() >= timestamp.longValue() ) {
               return false;
            }
         }
      }
      return true;
   }
   ...
}

I think what should happen can be seen in the comment: Put a fresh timestamp in the cache and return false, so that the query is executed.

Does anybody know, why this comment is there and why this is just a comment and not real code?

Thanks a lot in advance.

eellis · **Joined:** Wed Jan 21, 2009 6:18 pm **Posts:** 5

Although locking would be nice, this is not the problem. The issue as I understand it is that when a new cache entry is placed for the StandardQueryCache there is no corresponding timestamp record placed in the cache relating the set of "spaces". This means that when a cache update message is passed from, let's say a CMS instance, to another site, the update is dropped because the UpdateTimestampsCache does not have the initial record. The easy workaround would be to place, if empty, a dummy timestamp entry into the spaces used by the StandardQueryCache entry.

I've created a patch against Hibernate trunk which has been tested to work against version: Hibernate Core 3.3.1.GA - compiled with Java1.5

This has been tested and verified to work on a very high volume cluster of 7 machines clustering Hibernate with EHCache using a CMS Tomcat virtual host to update the main site's virtual host.

Unfortunately, these forums won't allow me to attach files (a zip with the classes to overwrite). I'll see if I can post them elsewhere. I'll also see if Hibernate will be willing to accept the patch; though it includes a very minor API change which may be of concern.

eellis · **Joined:** Wed Jan 21, 2009 6:18 pm **Posts:** 5

Code:

Index: src/main/java/org/hibernate/cache/QueryCache.java
===================================================================
--- src/main/java/org/hibernate/cache/QueryCache.java   (revision 15793)
+++ src/main/java/org/hibernate/cache/QueryCache.java   (working copy)
@@ -43,7 +43,7 @@
 
    public void clear() throws CacheException;
    
-   public boolean put(QueryKey key, Type[] returnTypes, List result, boolean isNaturalKeyLookup, SessionImplementor session) throws HibernateException;
+   public boolean put(QueryKey key, Type[] returnTypes, List result, Set spaces, boolean isNaturalKeyLookup, SessionImplementor session) throws HibernateException;
 
    public List get(QueryKey key, Type[] returnTypes, boolean isNaturalKeyLookup, Set spaces, SessionImplementor session) throws HibernateException;
 
Index: src/main/java/org/hibernate/cache/StandardQueryCache.java
===================================================================
--- src/main/java/org/hibernate/cache/StandardQueryCache.java   (revision 15793)
+++ src/main/java/org/hibernate/cache/StandardQueryCache.java   (working copy)
@@ -82,6 +82,7 @@
          QueryKey key,
          Type[] returnTypes,
          List result,
+         Set spaces,
          boolean isNaturalKeyLookup,
          SessionImplementor session) throws HibernateException {
       if ( isNaturalKeyLookup && result.size() == 0 ) {
@@ -109,6 +110,7 @@
             }
          }
 
+         updateTimestampsCache.populate((Serializable[]) spaces.toArray(new Serializable[0]));
          cacheRegion.put( key, cacheable );
 
          return true;
Index: src/main/java/org/hibernate/cache/UpdateTimestampsCache.java
===================================================================
--- src/main/java/org/hibernate/cache/UpdateTimestampsCache.java   (revision 15793)
+++ src/main/java/org/hibernate/cache/UpdateTimestampsCache.java   (working copy)
@@ -85,6 +85,19 @@
          region.put( spaces[i], ts );
       }
    }
+    
+   public synchronized void populate(Serializable[] spaces) throws CacheException {
+      Long ts = new Long( System.currentTimeMillis() );
+      for ( int i=0; i<spaces.length; i++ ) {
+         Long lastUpdate = (Long) region.get(spaces[i]);
+         if (lastUpdate == null) {
+            if ( log.isDebugEnabled() ) {
+               log.debug( "Populating space [" + spaces[i] + "], timestamp: " + ts);
+            }
+            region.put( spaces[i], ts );
+         }
+      }
+   }    
 
    public synchronized boolean isUpToDate(Set spaces, Long timestamp) throws HibernateException {
       Iterator iter = spaces.iterator();
Index: src/main/java/org/hibernate/loader/Loader.java
===================================================================
--- src/main/java/org/hibernate/loader/Loader.java   (revision 15793)
+++ src/main/java/org/hibernate/loader/Loader.java   (working copy)
@@ -2160,6 +2160,7 @@
                session, 
                queryParameters, 
                resultTypes,
+               querySpaces,
                queryCache, 
                key, 
                result 
@@ -2201,11 +2202,12 @@
          final SessionImplementor session,
          final QueryParameters queryParameters,
          final Type[] resultTypes,
+         final Set querySpaces,
          final QueryCache queryCache,
          final QueryKey key,
          final List result) {
       if ( session.getCacheMode().isPutEnabled() ) {
-         boolean put = queryCache.put( key, resultTypes, result, queryParameters.isNaturalKeyLookup(), session );
+         boolean put = queryCache.put( key, resultTypes, result, querySpaces, queryParameters.isNaturalKeyLookup(), session );
          if ( put && factory.getStatistics().isStatisticsEnabled() ) {
             factory.getStatisticsImplementor()
                   .queryCachePut( getQueryIdentifier(), queryCache.getRegion().getName() );

eellis · **Joined:** Wed Jan 21, 2009 6:18 pm **Posts:** 5

I'll probably take this down... but for now:

You can download the patch file here:
http://eellis.info/hibernate-patch/hibernate-core-classes.zip

You can download the class file overlay zip here:
http://eellis.info/hibernate-patch/hibernate-core.patch

-- as in: unzip the hibernate jar, overwrite w/ these classes then re-zip.[/url]

nicholeuf · **Joined:** Fri Jun 19, 2009 6:24 pm **Posts:** 2

Did you submit a patch or JIRA for this?

eellis · **Joined:** Wed Jan 21, 2009 6:18 pm **Posts:** 5

Yep, nobody seems to care.

http://opensource.atlassian.com/project ... e/HHH-3734

eellis · **Joined:** Wed Jan 21, 2009 6:18 pm **Posts:** 5

FYI, we've been running this patch on our production servers (7 node cluster, 1 million req/per day) since Jan 09 w/o a single issue. It works perfectly.

andreasbester · **Joined:** Wed Aug 12, 2009 6:58 am **Posts:** 1

We have the same issue, of course with a different configuration.

Great work with the patch. Although we have another work around for it.

I've created a bug report for it.
So for those who would like it to be resolved please vote for it.

http://opensource.atlassian.com/project ... e/HHH-4080

juraj · **Joined:** Wed Aug 18, 2010 4:47 am **Posts:** 1

awerder: I agree with you. I also think that 2 lines of code

Code:

 
//updateTimestamps.put( space, new Long( updateTimestamps.nextTimestamp() ) );
//result = false; // safer

should be real code and not comment. I found this from year 2004 (http://www.javakey.net/source/hibernate ... .java.html):

Code:

if ( lastUpdate==null ) {
   //the last update timestamp was lost from the cache
   //(or there were no updates since startup!)
   
   //NOTE: commented out, since users found the "safe" behavior
   // counter-intuitive when testing, and we couldn't deal
   // with all the forum posts :-(
   //updateTimestamps.put( space, new Long( updateTimestamps.nextTimestamp() ) );
   //result = false; // safer
   
   //OR: put a timestamp there, to avoid subsequent expensive
   // lookups to a distributed cache - this is no good, since
   // it is non-threadsafe (could hammer effect of an actual
   // invalidation), and because this is not the way our
   // preferred distributed caches work (they work by
   // replication)
   //updateTimestamps.put( space, new Long(Long.MIN_VALUE) );
}

eellis: I think you are talking about some other problem. You said:

Quote:

This means that when a cache update message is passed from, let's say a CMS instance, to another site, the update is dropped because the UpdateTimestampsCache does not have the initial record.

As far as I know, when cluster invalidation receives update message it should delete the record from UpdateTimestampsCache. So if the update is dropped because there is no initial record, from cluster invalidation point of view, it is ok. You are probably using cluster replication. Our problem is the same as awerder pointed out: when there is no record for space (db table) in UpdateTimestampsCache, then the function UpdateTimestampsCache.isUpToDate returns true...