Hibernate Books

All times are UTC - 5 hours [ DST ]



Post new topic Reply to topic  [ 11 posts ] 
Author Message
 Post subject: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Fri Aug 19, 2011 7:23 am 
Beginner
Beginner

Joined: Mon Apr 11, 2011 7:56 am
Posts: 38
Starting a new thread, continuing from this previous thread: Conditional indexing, multiple indexes / fields within index
I'm currently migrating from 3.4 to 4.0.0 alpha. The framework wrt index sharding changed in this new release, and now deals with IndexManagers instead of DirectoryProviders.
Changing the sharding strategy won't be that hard, just a matter of change in type if I look at the new CustomerShardingStrategy used in the tests.

Next problem I will have is to get the IndexManager for a specific shard (or 'namespace' in my case). The current 4.0 documentation doesn't show an example for getting an IndexManager. Not sure, but it seems that I currently need to cast the SearchFactory to an instance of SearchFactoryImplementor to have access to an index manager?

What I'm trying to accomplish is to get an IndexReader that uses a selection of shard. In HS3.4 it looked like this:
Code:
   public IndexReader getReader() {
      SearchFactory searchFactory = getFullTextSession().getSearchFactory();
      DirectoryProvider<?>[] providers = searchFactory
            .getDirectoryProviders(entityClass); // <-- How to get the index managers in 4.0 ?
      if(namespaceConstraint.isEmpty()) {
         return searchFactory.getReaderProvider().openReader(providers);
      } else {
         int index = directoryProviderIndexForNamespace();
         return searchFactory.getReaderProvider().openReader(providers[index]);
      }      
   }


How should I do this in HS4.0? Maybe there is already a better way to get an index reader using filters, that uses the sharding strategy (similar to what is done at query time)?

Thanks again :)


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Fri Aug 19, 2011 4:20 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
Hi Elmer,
that's great feedback: indeed I didn't think on how to open an IndexReader on a sharding subset only.
These are the available methods on the SearchFactory:
Code:
org.hibernate.search.SearchFactory.openIndexReader(Class<?>...)
org.hibernate.search.SearchFactory.closeIndexReader(IndexReader)

They are mentioned on the Migration Guide, and as said there it's experimental.
I'm quite fond of the new simple API, better than before. Do you have any suggestion to keep it simple and provide the flexibility you need?

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Sun Aug 21, 2011 9:25 am 
Beginner
Beginner

Joined: Mon Apr 11, 2011 7:56 am
Posts: 38
I agree the api is simpler. Getting a reader is cleaner/easier now that we don't have to bother about the DirectoryProviders/IndexManagers, in case no custom sharding strategy is used.

To get a reader with a subset of the indices (based on sharding strategy), I have currently 2 solutions in mind:

  • (prefered) Have a SearchFactory.openIndexReader(Class<?> c, FullTextFilterImplementor[] filters): This is similar to how the indexmanagers are gathered at query time, and is probably therefore easier to understand and more robust (using the same sharding strategy instance as HS does during querying).
  • Have a SearchFactory.getIndexManagers(Class<?> c), returning the array of indexManagers. This solution is similar to the implementation of <HS 4.0 (except that we're now dealing with IndexMangers instead of DirectoryProviders), where I get the targeted indices by using my own mapping to get the right array entry from the index managers. Or is this already possible? See Q2.

I think this will open up at least the flexibilty as we had in HS3.4.

Q2: Another thing that is still unclear to me: Is there a way/How to get the IndexManager(s) for a class? The docs say that the IndexManager's can be used, but now how they can be accessed?:
Quote:
9.3. Accessing a Lucene Directory

A Directory is the most common abstraction used by Lucene to represent the index storage; Hibernate Search doesn't interact directly with a Lucene Directory but abstracts these interactions via an IndexManager: an index does not necessarily need to be implemented by a Directory.

If you know your index is represented as a Directory and need to access it, you can get a reference to the Directory via the IndexManager. Cast the IndexManager to a DirectoryBasedIndexManager and then use getDirectoryProvider().getDirectory() to get a reference to the underlying Directory. This is not recommended, we would encourage to use the IndexReader instead.


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Thu Dec 01, 2011 7:50 am 
Newbie

Joined: Thu Dec 01, 2011 7:43 am
Posts: 5
Elmer wrote:
Q2: Another thing that is still unclear to me: Is there a way/How to get the IndexManager(s) for a class? The docs say that the IndexManager's can be used, but now how they can be accessed?:


This is the way I have done it:

Code:
      FullTextSession fullTextSession = Search.getFullTextSession(this
            .getSession());

      MutableSearchFactory searchFactory = (MutableSearchFactory) fullTextSession
            .getSearchFactory();

      Map<Class<?>, EntityIndexBinder> indexBindingsForEntity = searchFactory
            .getIndexBindingForEntity();

      for (EntityIndexBinder eib : indexBindingsForEntity.values()) {
         IndexManager[] indexManagers = eib.getIndexManagers();
         DirectoryBasedIndexManager indexManager = (DirectoryBasedIndexManager) indexManagers[0];

         Directory indexDirectory = indexManager.getDirectoryProvider()
               .getDirectory();

         //Do something...
      }


Regards!


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Thu Dec 01, 2011 8:49 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
Hi davidpolo,
that's correct, but why would you need to do that?

to easily access the IndexReader we exposed now

Code:
org.hibernate.search.SearchFactory.getIndexReaderAccessor()


which provides both:

Code:
org.hibernate.search.indexes.IndexReaderAccessor#open(Class<?>... entityTypes)
org.hibernate.search.indexes.IndexReaderAccessor#open(String... indexNames)


Should we accomodate for more use cases?

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Thu Dec 01, 2011 10:08 am 
Newbie

Joined: Thu Dec 01, 2011 7:43 am
Posts: 5
Hi s.grinovero,

Thank you for your reply.

I was trying to get the Directory in order to unlock it.

This code is run during the startup to assure there are no lock files:

Code:
FullTextSession fullTextSession = Search.getFullTextSession(this
            .getSession());

      MutableSearchFactory searchFactory = (MutableSearchFactory) fullTextSession
            .getSearchFactory();

      IndexReader indexReader = fullTextSession.getSearchFactory()
            .getIndexReaderAccessor().open("");

      Map<Class<?>, EntityIndexBinder> indexBindingsForEntity = searchFactory
            .getIndexBindingForEntity();

      for (EntityIndexBinder eib : indexBindingsForEntity.values()) {
         IndexManager[] indexManagers = eib.getIndexManagers();
         DirectoryBasedIndexManager indexManager = (DirectoryBasedIndexManager) indexManagers[0];

         Directory indexDirectory = indexManager.getDirectoryProvider()
               .getDirectory();

         if (IndexWriter.isLocked(indexDirectory)) {
            IndexWriter.unlock(indexDirectory);
         }
      }


A better solution is welcome...


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Thu Dec 01, 2011 12:32 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
ok I see.
we could add an option which forces unlock at boottime.. would you really want that?
I'm asking as it sounds dangerous: if you start two apps and force unlock to an index which is being used, you'll corrupt the index.

But I guess we could add an option, then people might be free to avoid it or use it as they want. Feel free to create a JIRA to ask this; if you want to code it yourself even better ;)

Consider that your current approach might not know if your same application is actually already writing on the index; make sure you have some external protection / lock around that.

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Thu Dec 01, 2011 1:05 pm 
Newbie

Joined: Thu Dec 01, 2011 7:43 am
Posts: 5
I have a single application that executes this code at startup so I think it's safe to unlock the Directory at this time.

I'm doing this because I'm using SimpleFSLockFactory and I was getting the exception after an unexpected shutdown of the application:

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out

I need to unlock at startup because I think using NativeFSLockFactory is not an option for me (I'm using NFS).


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Thu Dec 01, 2011 2:00 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
I think you're correct. open a JIRA please :)

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Fri Dec 02, 2011 4:45 am 
Newbie

Joined: Thu Dec 01, 2011 7:43 am
Posts: 5
I have created a JIRA:

https://hibernate.onjira.com/browse/HSEARCH-998

I don't know if it's correct, it's the first time I open a JIRA...


Top
 Profile  
 
 Post subject: Re: ShardingStrategy migration HSearch 3.4 to 4.0
PostPosted: Fri Dec 02, 2011 6:07 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2296
Location: Third rock from the Sun
thanks a lot!
No worries, it's good, only change I made is to change it into a "feature request" as I wouldn't consider it a bug, and so it doesn't affect a previous version (removed that).

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.