Unwanted initialization non-dirty lazy references on-update

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

sanne.grinovero wrote:

Hi,
I got a bit confused on your last post. Are you saying that you are loading the user ids from a separate Lucene index rather than the database?

That surprises me a bit for two reasons:
- usually loading any stored field from the index is quite slow, it's CPU intensive and also quite IO heavy. People generally prefer to load from a database for such simple data extractions. Is this performing better in your case? We had recently introduced a FieldSelector to reduce the index extraction overhead: I'm wondering if that's helping with this.

>To be honest, I haven't yet done any comparisons in terms of response times and I just separated the user retrieval part which previously was stored at the Post level. So, it's a natural progression. The various filters to display the appropriate posts to the right user make it sensible to store the user ids and retrieve the users based on that (from the db or index). Otherwise I would have to retrieve the Posts themselves from the database which would kill the performance.
Honnestly, I never considered data retrieval from the index a constraint. I will have a look at the FieldSelector. Where can I find the documentation on this?

- How do you deal with transactions? Index operations might be stale (async) or generally just not as reliable as the database.

That is not a problem in this use case. If a change of photo or nickname doesn't come through immediately, it's not a disaster.

But I'm more interested in the other aspect which remains: if you store a variable in the index that may change and that is actually a selection attribute, the solution I chose for the User doesn't work. In that case, you actually need to keep that field updated resulting in the dreaded cascading database reads.

- Marc

sanne.grinovero · **Posted:** Wed Nov 21, 2012 8:01 am

Marc, did you consider applying a custom filter?

A filter can be cached, and contain complex logic for which you decide the match needs to be excluded or not. This decision can be evaluated connecting for example to the database or other external sources, so some things don't necessarily need to be encoded in the index.

Would that help?

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

Hi Sanne,

I never considered a filter because I tend to think of them as operating on the same index. Let's say I have a Post which can be part of a Group. So Posts can show up on the wall that are part of various groups. The Group can be hidden or not. If the GroupOwner sets the group to hidden, all the associated Posts have to be made invisible. I don't want this "hidden" attribute to be stored on the post level, because changing it would require updating all the posts and re-reading all the posts from the database. How could I use a filter to achieve this? I wasn't really aware that I could e.g. read from another index in this process but perhaps you're thinking in a different direction.

sanne.grinovero · **Posted:** Thu Nov 22, 2012 9:01 am

technically a filter could invoke anything.. I've heard of people invoking a remote web service.. not that I'm suggesting that as you'll be making many invocations! Just making sure you are aware you have this additional tool available while puzzling your solution.

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

I guess I don't really see how I could best use a Lucene filter to address the use case of the hidden group items. All the documentation seems to point to the use of a filter against the current index. Do you have an example of using a filter that e.g. queries another index for a particular value while processing result elements?

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

Can you give me some more hints on this. Are you suggesting that I pass a dao as a key reference to the filter so I can run queries? Would that even work session wise? And how would you avoid hitting the database/index for every record? I could really use an example on this because I think I might be completely misunderstanding you. There seems to be no documentation or forum posts whatsoever on this. This being: filtering the results of a Lucene query against something other than a value that's already available in the Document.

mschipperheyn · **Joined:** Wed Nov 05, 2003 7:22 pm **Posts:** 211

Looks like the new query time join does what I need when it comes to making selections on another index than the one I'm retrieving results from.
https://hibernate.onjira.com/browse/HSEARCH-1237