ShardingStrategy migration HSearch 3.4 to 4.0

Elmer · **Joined:** Mon Apr 11, 2011 7:56 am **Posts:** 38

Starting a new thread, continuing from this previous thread: Conditional indexing, multiple indexes / fields within index
I'm currently migrating from 3.4 to 4.0.0 alpha. The framework wrt index sharding changed in this new release, and now deals with IndexManagers instead of DirectoryProviders.
Changing the sharding strategy won't be that hard, just a matter of change in type if I look at the new CustomerShardingStrategy used in the tests.

Next problem I will have is to get the IndexManager for a specific shard (or 'namespace' in my case). The current 4.0 documentation doesn't show an example for getting an IndexManager. Not sure, but it seems that I currently need to cast the SearchFactory to an instance of SearchFactoryImplementor to have access to an index manager?

What I'm trying to accomplish is to get an IndexReader that uses a selection of shard. In HS3.4 it looked like this:

Code:

   public IndexReader getReader() {
      SearchFactory searchFactory = getFullTextSession().getSearchFactory();
      DirectoryProvider<?>[] providers = searchFactory
            .getDirectoryProviders(entityClass); // <-- How to get the index managers in 4.0 ?
      if(namespaceConstraint.isEmpty()) {
         return searchFactory.getReaderProvider().openReader(providers);
      } else {
         int index = directoryProviderIndexForNamespace();
         return searchFactory.getReaderProvider().openReader(providers[index]);
      }      
   }

How should I do this in HS4.0? Maybe there is already a better way to get an index reader using filters, that uses the sharding strategy (similar to what is done at query time)?

Thanks again :)

sanne.grinovero · **Posted:** Fri Aug 19, 2011 4:20 pm

Hi Elmer,
that's great feedback: indeed I didn't think on how to open an IndexReader on a sharding subset only.
These are the available methods on the SearchFactory:

Code:

org.hibernate.search.SearchFactory.openIndexReader(Class<?>...)
org.hibernate.search.SearchFactory.closeIndexReader(IndexReader)

They are mentioned on the Migration Guide, and as said there it's experimental.
I'm quite fond of the new simple API, better than before. Do you have any suggestion to keep it simple and provide the flexibility you need?

Elmer · **Joined:** Mon Apr 11, 2011 7:56 am **Posts:** 38

I agree the api is simpler. Getting a reader is cleaner/easier now that we don't have to bother about the DirectoryProviders/IndexManagers, in case no custom sharding strategy is used.

To get a reader with a subset of the indices (based on sharding strategy), I have currently 2 solutions in mind:

(prefered) Have a SearchFactory.openIndexReader(Class<?> c, FullTextFilterImplementor[] filters): This is similar to how the indexmanagers are gathered at query time, and is probably therefore easier to understand and more robust (using the same sharding strategy instance as HS does during querying).
Have a SearchFactory.getIndexManagers(Class<?> c), returning the array of indexManagers. This solution is similar to the implementation of <HS 4.0 (except that we're now dealing with IndexMangers instead of DirectoryProviders), where I get the targeted indices by using my own mapping to get the right array entry from the index managers. Or is this already possible? See Q2.

I think this will open up at least the flexibilty as we had in HS3.4.

Q2: Another thing that is still unclear to me: Is there a way/How to get the IndexManager(s) for a class? The docs say that the IndexManager's can be used, but now how they can be accessed?:

Quote:

9.3. Accessing a Lucene Directory

A Directory is the most common abstraction used by Lucene to represent the index storage; Hibernate Search doesn't interact directly with a Lucene Directory but abstracts these interactions via an IndexManager: an index does not necessarily need to be implemented by a Directory.

If you know your index is represented as a Directory and need to access it, you can get a reference to the Directory via the IndexManager. Cast the IndexManager to a DirectoryBasedIndexManager and then use getDirectoryProvider().getDirectory() to get a reference to the underlying Directory. This is not recommended, we would encourage to use the IndexReader instead.

davidpolo · **Joined:** Thu Dec 01, 2011 7:43 am **Posts:** 5

Elmer wrote:

Q2: Another thing that is still unclear to me: Is there a way/How to get the IndexManager(s) for a class? The docs say that the IndexManager's can be used, but now how they can be accessed?:

This is the way I have done it:

Code:

      FullTextSession fullTextSession = Search.getFullTextSession(this
            .getSession());

      MutableSearchFactory searchFactory = (MutableSearchFactory) fullTextSession
            .getSearchFactory();

      Map<Class<?>, EntityIndexBinder> indexBindingsForEntity = searchFactory
            .getIndexBindingForEntity();

      for (EntityIndexBinder eib : indexBindingsForEntity.values()) {
         IndexManager[] indexManagers = eib.getIndexManagers();
         DirectoryBasedIndexManager indexManager = (DirectoryBasedIndexManager) indexManagers[0];

         Directory indexDirectory = indexManager.getDirectoryProvider()
               .getDirectory();

         //Do something...
      }

Regards!

sanne.grinovero · **Posted:** Thu Dec 01, 2011 8:49 am

Hi davidpolo,
that's correct, but why would you need to do that?

to easily access the IndexReader we exposed now

Code:

org.hibernate.search.SearchFactory.getIndexReaderAccessor()

which provides both:

Code:

org.hibernate.search.indexes.IndexReaderAccessor#open(Class<?>... entityTypes)
org.hibernate.search.indexes.IndexReaderAccessor#open(String... indexNames)

Should we accomodate for more use cases?

davidpolo · **Joined:** Thu Dec 01, 2011 7:43 am **Posts:** 5

Hi s.grinovero,

Thank you for your reply.

I was trying to get the Directory in order to unlock it.

This code is run during the startup to assure there are no lock files:

Code:

FullTextSession fullTextSession = Search.getFullTextSession(this
            .getSession());

      MutableSearchFactory searchFactory = (MutableSearchFactory) fullTextSession
            .getSearchFactory();

      IndexReader indexReader = fullTextSession.getSearchFactory()
            .getIndexReaderAccessor().open("");

      Map<Class<?>, EntityIndexBinder> indexBindingsForEntity = searchFactory
            .getIndexBindingForEntity();

      for (EntityIndexBinder eib : indexBindingsForEntity.values()) {
         IndexManager[] indexManagers = eib.getIndexManagers();
         DirectoryBasedIndexManager indexManager = (DirectoryBasedIndexManager) indexManagers[0];

         Directory indexDirectory = indexManager.getDirectoryProvider()
               .getDirectory();

         if (IndexWriter.isLocked(indexDirectory)) {
            IndexWriter.unlock(indexDirectory);
         }
      }

A better solution is welcome...

sanne.grinovero · **Posted:** Thu Dec 01, 2011 12:32 pm

ok I see.
we could add an option which forces unlock at boottime.. would you really want that?
I'm asking as it sounds dangerous: if you start two apps and force unlock to an index which is being used, you'll corrupt the index.

But I guess we could add an option, then people might be free to avoid it or use it as they want. Feel free to create a JIRA to ask this; if you want to code it yourself even better ;)

Consider that your current approach might not know if your same application is actually already writing on the index; make sure you have some external protection / lock around that.

davidpolo · **Joined:** Thu Dec 01, 2011 7:43 am **Posts:** 5

I have a single application that executes this code at startup so I think it's safe to unlock the Directory at this time.

I'm doing this because I'm using SimpleFSLockFactory and I was getting the exception after an unexpected shutdown of the application:

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out

I need to unlock at startup because I think using NativeFSLockFactory is not an option for me (I'm using NFS).

sanne.grinovero · **Posted:** Thu Dec 01, 2011 2:00 pm

I think you're correct. open a JIRA please :)

davidpolo · **Joined:** Thu Dec 01, 2011 7:43 am **Posts:** 5

I have created a JIRA:

https://hibernate.onjira.com/browse/HSEARCH-998

I don't know if it's correct, it's the first time I open a JIRA...

sanne.grinovero · **Posted:** Fri Dec 02, 2011 6:07 am

thanks a lot!
No worries, it's good, only change I made is to change it into a "feature request" as I wouldn't consider it a bug, and so it doesn't affect a previous version (removed that).