Interesting problem!
Quote:
Replication at the backing RDBMS level has been ruled out, as we need to replicate objects as stored by Hibernate, not individual rows in a table.
this puzzled me the most: if you have a full RDBMS, objects loaded by Hibernate will be identical.
I would use RDBMS replication, as it's easy to setup and proven.
Quote:
The replicated caching functionality I've been reading about seems to be a bit "all or nothing", i.e. it either only invalidates when something is changed (which is no use for offline availabilty), or pushes changes to all of the attached nodes.
The problem with caches, is that their contents tends to become outdated when you change the underlying database: to deal with this on a single node, Hibernate can notify the cache to update the correct entries when it updates the DB.
When having multiple nodes, these caches (which are otherwise pretty dumb and not understanding the relation between entries) can either make sure stale values on another node are updated or removed (invalidated); so if you have multiple nodes and they might not be able to communicate, you need to disable caching. The exceptional case in which caching could still be used is for read-only entities, or if the database happens to be used read-only when your central location is down, and you're prepared to reboot or programmatically invalidate the caches when it comes back online (for example via JMX).
If you're willing to experiment with real new technology, I think Hibernate OGM could help. With OGM we don't have the database at all, but replace it with a non-relational store; one of the supported stores is Infinispan which brings transactions and the speed of a cache, and is fully distributed. When a node fails in Infinispan, the others can continue working independently and transparently re-balance the stored values across nodes.
So you don't need the cache at all, but you still get its speed (actually quicker, as with a 2nd level cache you occasionally still have slow access to an RBDMS), and replication is cared for.
The problem I see with using OGM with Infinispan in your case is that you mentioned "offline work", rather than failover. Infinispan will fail if the cluster is split and then you write conflicting updates to the store, and then reconnect the clusters: it won't be able to decide which version to keep of your conflicting writes; to solve that you would need some eventual consistency policy controlled by your application, this is expected to be supported by Infinispan version 6 (not yet). But hopefully you don't need that, as it gets tricky!
http://www.hibernate.org/subprojects/ogm.html