-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 6 posts ] 
Author Message
 Post subject: Indexing into multiple ElasticSearch indices
PostPosted: Wed Mar 08, 2017 7:46 am 
Newbie

Joined: Wed Mar 08, 2017 5:31 am
Posts: 11
Hello,

I am migrating to Hibernate Search 5.6.1 in order to try using the database-to-Elasticsearch syncing capabilities of Hibernate Search.
I have already integrated with my ElasticSearch cluster and am indexing and querying using the low-level REST API, however I am searching for a managed syncing solution (only) and since you guys are working on it I figured it's better to leave it to the people with the know-how.

I already have my entity data indexed in ES. I split it into 3 indices (LIVE 1, LIVE 2 and ARCHIVE) where the two live indices hold more recent and relevant data (split into two halves for query performance reasons), and the archive index holds all old data. I need to be able to index an entity into one of these three indices, selectively, depending on my own custom logic. I know I can use @Indexed(index="INDEX_NAME") but how cat I choose the index name at index time?

Additionally, I use the 'routing' parameter for the ES Document APIs when doing CRUD operations against ES, and I configured my indices so that the 'routing' parameter is required. I use 'routing' to control in which shard the document should be placed, which is good for search performance. I use the value of a property in the document ast the value of the routing parameter. Can I use routing with Hibernate Search somehow?

Any help is appreciated!


Top
 Profile  
 
 Post subject: Re: Indexing into multiple ElasticSearch indices
PostPosted: Wed Mar 08, 2017 8:11 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hello,

splitting the data across indexes can be done using Hibernate Search's sharding capabilities - not to be confused with the sharding applied by Elasticsearch.

In particular see "Dynamic Sharding" as it allows you to define the name of the index you want to target for each operation:
- https://docs.jboss.org/hibernate/search ... c-sharding

The "routing" option is not supported yet. It seems like a good idea to add it! Please open a feature request:
- https://hibernate.atlassian.net/projects/HSEARCH/issues

Looks like we should consider matching the "dynamic sharding" capabilities of Hibernate Search to map directly to the routing API. I'd be most interested in your suggestions on this: if you could describe your use case on the feature request that will be a great help.

Thanks!

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Indexing into multiple ElasticSearch indices
PostPosted: Wed Mar 08, 2017 10:23 am 
Newbie

Joined: Wed Mar 08, 2017 5:31 am
Posts: 11
Thank you, it looks like dynamic sharding could work for selecting a single index to put the data in.

I created a feature request with my use case here: https://hibernate.atlassian.net/browse/HSEARCH-2634


Top
 Profile  
 
 Post subject: Re: Indexing into multiple ElasticSearch indices
PostPosted: Wed Mar 08, 2017 10:37 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
thanks that feature request, that's very useful feedback for our team.

I hope we can include this in the 5.8 version, as that's what we're working on. That will imply that you'll have to upgrade to Hibernate ORM 5.2.x, I hope that's not a problem?

This version will also support both Elasticsearch 2.x and Elasticsearch 5.x, and switch to the new Elasticsearch client so it should be worth it.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: Indexing into multiple ElasticSearch indices
PostPosted: Wed Mar 08, 2017 11:49 am 
Newbie

Joined: Wed Mar 08, 2017 5:31 am
Posts: 11
Unfortunately I don't think I will be able to upgrade to Hibernate 5.2 just yet, since it requires me upgrading to Java 8 and that is a project on its own. Any chance ES 5.x and this '_routing' change will be backported to 5.6.1?

Definitely looking forward to ES 5! I will give more feedback as I test


Top
 Profile  
 
 Post subject: Re: Indexing into multiple ElasticSearch indices
PostPosted: Wed Mar 08, 2017 12:00 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
I'm sorry we won't be able to backport these things, that's way too much work and we're a small team.

We spent a lot of energy to make sure that people would have a very smooth experience to upgrade to Java 8; if we're missing anything there which could help please let us know that as well.

For example - while I don't know about other libraries which you might be using - the versions you mentioned on HSEARCH-2634 are all already compatible with Java8.

Thanks!

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 6 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
cron
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.