Hi,

We are trying to write a Dialect for Coherence/Hazelcast for Hibernate OGM. I do understand that all the data is stored in the cache as Tuples (key/value pairs). Is there anyway I can store the actual portable format of the object (supported by Hazelcast/Coherence[POF format]) and then do the translation by implementing some Hibernate OGM specific interfaces in between. This would help in reducing the amount of data that is stored in the cluster as well as it will help to query the cache directly (using the native query API that is available in the griddialect interface e.g. we can use count/average etc. not supported by OGM or run aggregators/entryprocessors directly on the grid).

As an example this is what I want to do: Lets say there is an Order and OrderDetail object I wont to store it something like this
[orderpk_key_1],[Value is stored in POF format].

When I query, I could use Hibernate search or utilize a native search query like what is supported by Mongo for e.g. on HazelCast and get the required results. Internally if its Portable format, I can add specific indexes supported by the cache itself.

Please do let me know if its possible.

Thanks,
J

Hi,

(I have no experience with Hazelcast and just had a quick look in their docs to find out about the portable serialization, so take the following with a grain of salt).

As you say, Hibernate OGM uses a generic contract centered around a tuple-like representation of your data. So what won't work is to let your actual domain objects implement Portable because you will not get hold of these objects from within your dialect implementation.

But a dialect may decide to persist the passed tuples in whatever way it likes, it doesn't have to be the direct tuple representation. So one way may be to create a PortableTuple (implements Portable) class and persist such objects based on the tuples passed to insertTuple(). Your TupleSnapshot implementation would wrap such PortableTuple when reading data.

That'd still be a generic representation (basically storing the tuple column name array and the tuple column value array), I guess it's not really idiomatic and not ideal for querying. So you may try to implement PortableTuple in a way that it writes out the actual column values, but that'd require to know the persisted structure during read back; I'm not sure how that could be done (apart from encoding that structure somehow at the beginning of the record). Alternatively, you'd need to have one XyPortableTuple type for each entity type which may be doable by generating these classes upon start-up based on the registered entity types.

Hth,

--Gunnar

If you really want one POF per object type persisted, you could make use of the option system to enable such option. You could also pass some metadata to guess the POF for a given type. The dialect receives such option and can do whatever it pleases with it.

But the question behind is interesting. Is it worth to have a JPA provider store the actual objects in the grid (or a portable type safe equivalent of it). It is seducing for simple properties but what about object associations? Do you materialize them in a specific way or do you serialize the whole graph? What about duplication if you happen to store the object graph?

I'm interested into what you guys have in mind.

Hi Gunnar, Emmanel,

Thanks a lot for your responses. I will try to answer each question on what we are trying to do from a use-case perspective:

a) Is it worth to have a JPA provider store the actual objects in the grid (or a portable type safe equivalent of it)?

The reason why we want to store the actual objects in the grid is purely because we can query them using the native query form where OGM has limitations. Also, we can run EntryProcessors (I think Infinispan has DistributedCallable - sorry I havent used it but tried to find an equivalent) directly on the grid on the data. The Object graph against which we want to run these entry processors is around 8 levels deep and want to do processing on the actual objects based on the Object data. E.g. think of it as we want to run a MongoDB style Aggregation pipeline or a Map Reduce on the cluster where having the data in the Portable format helps for performance reasons. The generated objects will then be written to Oracle DB using a write-behind semantics.

Another reason is the total amount of memory that is required to store an Object is very high when you consider the Tuple format vs the portable type safe equivalent. E.g. the amount of memory required would grow exponentially as the column name and value are stored for each value. This is equivalent to storing JSON format. To give you an idea, the Object structure generates 8 MB JSON vs a 500KB for POF. This also has implications on Garbage collection and the size of JVMs we run.

Ability to run native queries directly on the cache is something that helps from a support perspective (application support) to check the data for consistency between cache and database.

b) It is seducing for simple properties but what about object associations? Do you materialize them in a specific way or do you serialize the whole graph?
This is a really good question and to be frank we have not thought about it completely. I think associations and the complex and connected nature of object graph is why we set out in evaluating the usage of Hibernate OGM in our project. We already have a rich Domain graph which is using Hibernate/JPA to persist and read objects from the database which works fine but have issues from a scalability perspective. The one thing why we are looking at OGM is that we should be able to take the same deeply nested object graph and persist it into a cache cluster (we do understand it has some limitations with respect to JPA annotations but we can work around it). I don't know whether this is a good idea and thats where we are evaluating OGM.

Technically we want the best of both worlds i.e. associations and objects stored in POF format but OGM handling all the association related complexities.

c) What about duplication if you happen to store the object graph?

We have 2 types of use cases i.e. one where we want to retrieve the entire object graph (just to give you a perspective hibernate fires around 16k queries to retrieve this entire object tree data on the database) and another where a part of object tree is required based on the queries fired. We want to use 2 strategies i.e. one where an entire object graph is stored as one entry and another where the object graph is stored in a format where parts of the graph could be queried.

Question : Is it possible to persist the entire graph as one single entry using OGM and then retrieve it using one call from the cache?

Duplication is not an issue. I do understand that we might face issues with respect to consistency but we are ready to handle it programmatically as long as we get the performance benefits.

Hi J.,

Your idea has intrigued me. We are not where you want to go I think, esp around the ability to store / retrieve data both as full graph and as split elements. But I don't think we are necessarily that far. Esp if you have plans to write the dialect, the additional seems conceptually doable to me.

I started a thread on the dev mailing list (registration needed) https://lists.jboss.org/mailman/listinfo/hibernate-dev

The thread is here: http://lists.jboss.org/pipermail/hibernate-dev/2015-January/012044.html

Are you fine to chime in there or is that a problem?

Emmanuel

Hi Emmanuel,


Sorry for the late reply. I will pitch in on the mailing lists as requested. Also, I would need some guidance for the strategy aspects on the extension points for writing the dialect. Will share the code on Github as soon as the strategy is finalized.

Thanks again for your reply!