I'm trying to find the best way to index entities of the same type, based on an entity's property value.

For example, we are currently working on a project to make our (and other) SVN repositories searchable for code. In this project, we have an entity CodeFragment , and each code fragment has a property code, and project (the project in which the code resides).

To search specfic within a project, we apply a filter, which filters on the project field. With this solution, the ranking of results is not scoped to one project, but this is currently not a big issue.
We also give autocompletion suggestions, which should be scoped to the project a user is currently searching. The autocompleter uses the Hibernate Search Lucene index as source for its terms and frequencies to build its own index.
And here is where we'd like to fragment the index based on a entity's property value, namely project in our case.

I've looked at the documentation, and found a solution that might work: using a class bridge, which checks the value of the project property, and based on that, prefixes the field name with that value. This results in a (single) index for the CodeFragment entity, but with frequencies and terms scoped to the project the code fragment belongs to, by using different fields for each project.

I'm not sure if this is the best solution. The requirement is that we should be able to get terms and frequencies scoped to the project (entity property) a CodeFragment belongs to. Any other ideas?

Hi Elmer,
I think you should use sharding: you create a sharding policy such that each project has it's own index, and then you can make use of the awesome filter implementation which actually selects the proper shard:
http://docs.jboss.org/hibernate/stable/ ... lter-shard

Hmm, how could I've missed that :o
Let's see if I can set this up dynamically in our DSL.
Thanks!

almost forgot, there's one catch: it's currently not supported to create new shards dynamically: you'll have to restart it when adding a new project.

We have plans to introduce dynamic sharding; it's scheduled for the next release but it might slip as it is quite tricky.

That's actually a big catch in our case. Is it a problem, performancewise, to set the number of shards to a big number, if only one shard is searched at each query?
Still, then there is the problem of keeping a mapping from, in this case, string value to integer.
I think I'm gonna try dynamic fields instead of sharding, using a class bridge. That seems more robust to me at the moment.

Not a problem, you get actually great performance: better than a single index assuming they are all used frequently (a non "warmed up" index will not be very responsive as caches invalidate, so the first hit on such an index after some time might be slower, but overall better throughput).


That's easy to solve in your custom sharding implementation, a simple solution could use a properties file listing the project names, a better one could run a query on the database.


sure, very reasonable. If it doesn't work out, let's discuss alternatives. You're also welcome in changing whatever you need and send patches to be discussed and integrated.

Eventually I'm now implementing it using index sharding. It's better to have separate indices for each 'namespace', if a namespace A is updated more frequently than others, caching is more effective because indices of other namespaces stay untouched (and thus don't need to warm up again), if I understand correctly. I'll post an update if finished.

Correct.

yes please let us know how it goes, that's very useful to see where/what to improve.

I'm still bussy implementing it in our DSL, but I was thinking, what about inheritance?
If B : A , and A has the sharding strategy, will entity B be sharded the same way as entity A?

sharding is coupled to the index name, so it depends if the two types are both @Indexed and are using the same @Indexed(index= name )

Ok :) So if I search for entities of type A and namespace declarations (i.e. sharding) is defined on entity A, I also get B's for that particular namespace. This won't work if I search for B's, because my B is indexed using a different name. But that's not a big issue, I can inherit the same sharding strategy from B's superclass(es), and B might even override the property that is acting as namespace identifier (at most 1 namespace at each level of inheritance). Sounds cool!

(assuming that entities are indexed X times, where X is the number of ancestor classes (including itself) that have @Indexed annotation)

Don't mix up physical index configuration and sharding, which are performance options, with semantics. The search will always work the same, we make sure we target the proper shards/indexes and aggregate the needed results (or filter them out) according to the typesafe parameters provided at search time.
Maybe it's best if you write some tests to better understand how it will work ;)

Ok :)

Next question, how should I accomplish the following in a sharded environment where I use my name spaces?



How do I get a name space specific DirectoryProvider using the custom sharding strategy? Do I need to use the custom strategy method getDirectoryProvidersForQuery directly?

Why are you trying to get a reference to the DirectoryProvider ?
you just have to run the queries, applying a custom filter if you want to: the filters are passed over to the ShardingStrategy, so your custom strategy can look into the enabled filters, possibly read some of your custom filter parameter, and tell us which directories shall be queried according to your needs by returning the appropriate DirectoryProviders.

a rather complete example is here:
http://docs.jboss.org/hibernate/stable/ ... lter-shard

Because there is more than querying :) We also need references to the lucene directory / readers for MoreLikeThis queries, spell correction and auto completion.

So there is currently no easy way to get a sharded strategy depending DirectoryProvider?

(btw, our custom sharding strategy is almost finished, I fully understand its working, and it's based on the code in the doc. you mentioned)