Filter Implementation Example

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

Hi ,

I have written the following filter, where the idea is to return all results which match the words from the passed array. When i debug through i can see values being returned into the bitset, however the filter doesnt' return the expected result from the query, from the data i know there should be at least one value in the list returned. Have i coded this correctly?

Code:

public class CityFilter extends Filter {
   
   private String[] cities;
   
   public CityFilter(String[] cities) {
      this.cities = cities;
   }

   @Override
   public BitSet bits(IndexReader indexReader) throws IOException {
      BitSet bitSet = new BitSet( indexReader.maxDoc() ); 
      int[] docs = new int[1];
      int[] freqs = new int[1];
      TermDocs termDocs = null;
      for (int i = 0; i < cities.length; i++) {
         String city = cities[i];
         if (city != null) {
            termDocs = indexReader.termDocs( new Term( "profile.filter.city", city ) );
            int count = termDocs.read(docs, freqs);
            if (count == 1) { 
               bitSet.set(docs[0]); 
            }
         }
         while (termDocs != null && termDocs.next()) {
            bitSet.set(termDocs.doc());
         }
      }

      return bitSet;
   }

}

I'm not sure if the while-loop is in the correct place and/or if the bitSet.set(docs[0]); is correct.

and the actual Filter Factory:

Code:

public class CityFilterFactory {
   
   private String[] cities;
   

   @Factory
   public Filter buildCityFilterFactory() {
      
      CityFilter cityFilter = new CityFilter(cities);
      
      return new CachingWrapperFilter(cityFilter);
   }
   
   @Key
   public FilterKey getKey() {
      StandardFilterKey key = new StandardFilterKey();
      key.addParameter(cities);
      return key;
   }

   public String[] getCities() {
      return cities;
   }

   public void setCities(String[] cities) {
      this.cities = cities;
   }

   
}

Any help appreciated.

Thanks

sanne.grinovero · **Posted:** Wed Mar 09, 2011 11:35 am

Have a look at QueryWrapperFilter , much easier ;)

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

Hi Sanne,

Thanks, will do but can you recommend an example, where i can pass in an array?

Thanks

sanne.grinovero · **Posted:** Wed Mar 09, 2011 12:12 pm

don't pass in an array, build a BooleanQuery having a SHOULD condition for each element of the array, then wrap the Query in the QueryWrapperFilter to make a filter of it.

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

Hi Sanne,

That's a much easier and elegant solution :)

Code:

@Factory
   public Filter buildCityFilterFactory() {
      
       BooleanQuery booleanQuery = new BooleanQuery();
       for(int i=0; i<cities.length; i++) {
          booleanQuery.add(new TermQuery(new Term("profile.filter.city", cities[i].toLowerCase())), 
                            BooleanClause.Occur.SHOULD);
       }
       Filter filter = new QueryWrapperFilter(booleanQuery);
       
       return filter;

   }

Although my results are still not being returned, as in the list is empty, will this behave as an OR operand. hence returning which ever matches or is it an AND computation on each value passed into the BooleanQuery?

Thanks

sanne.grinovero · **Posted:** Wed Mar 09, 2011 12:43 pm

that's an OR, so it shouldn't be empty.
I guess you have the city names analyzed in a different way, so "lowercase" is not enough: you need an exact match of the strings in the TermQuery with those in the index.
you can quickly check this by indexing you city names as UN_TOKENIZED, then make sure you match the case.

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

Hi Sanne,

That's interesting. I removed the store=Store.YES from the field and specifed the analyzer to user (@Field(analyzer=@Analyzer(impl=StandardAnalyzer.class)) and that returned the correct result.

I was have the understanding that store=Store.YES stores the value as entered and it is not affected by the analyzer (only as storage), hence, at search time it should match say "cheshire" even if it is stored as Cheshire. Is this not the case?

Thanks

sanne.grinovero · **Posted:** Wed Mar 09, 2011 12:58 pm

you're confusing Index.YES with Store.YES - they are independent

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

yes, appologies. i updated the post just as you were replying.

sanne.grinovero · **Posted:** Wed Mar 09, 2011 1:23 pm

so what you Store is relevant only for projections, to be able to extract your data - or most likely part of it - back from the resulting hits.
What you Index is what will identify your results - so to match a Query it has nothing to do with Store but with how you index it.

In most cases (when using the QueryParser, or Search's own DSL to create queries) the corresponding analyzer is used to tokenize the search string (input), in cases as yours where you directly build the BooleanQuery - that's considered a bit more advanced - you have to make sure the terms you use do actually match the output of your tokenizer/analyzer.

Use Luke to look into the index, or checkout the unit tests in Search to find out how in some cases we do get a reference to the analyzer directly to use it.

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

Hi Sanne,

That's great thanks. Okay so if i wanted to store "Manchester and Bolton" but searched just manchester would that match using the BooleanQuery (i dont' see that it would) - would this mean indexing as:
@Field(index=Index.UN_TOKENIZED, analyzer=@Analyzer(impl=StandardAnalyzer.class))? Or does this mean searching for exactly "manchester and bolton" using the BooleanQuery?

Thanks

sanne.grinovero · **Posted:** Thu Mar 10, 2011 7:09 am

Index.UN_TOKENIZED --> skips the Analyzer, so it won't care which Analyzer you define.
So if you index "Manchester and Bolton" you will need to build the TermQuery exactly with "Manchester and Bolton", same case, same spacing. it's like String.equals.

In the case Index.TOKENIZED it will use an Analyzer, so to match "Manchester and Bolton" assuming you use StandardAnalyzer (varies according to analyzer, make sure you take it in consideration!) the hit will be found to match either of "manchester", "and", "bolton" (in no specific order, but needs to be lowercase).

If you need to split the text using a specific analyzer, have a look at how it's done in
org.hibernate.search.test.util.AnalyzerUtils.tokensFromAnalysis(Analyzer, String, String), you'll find it in the sourcecode, tests.

Also consider that org.hibernate.search.SearchFactory
has methods as

Code:

Analyzer getAnalyzer(String name);
Analyzer getAnalyzer(Class<?> clazz);

so you can ask for the proper analyzer you might need.

please have a look in the tests, it will clarify a lot of your doubts :)

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

Cool,

Been through the tests and yes it has clarified numerous issues.

However, i tried both solutions:
1, @Field(index=Index.UN_TOKENIZED, store = Store.YES) and when searching using the BooleanQuery i pass in the exact words ("Manchester and Bolton") and yes it works

2, @Field(store = Store.YES, analyzer=@Analyzer(impl=StandardAnalyzer.class)) and when searching using the BooleanQuery i lower case the search string ("manchester and bolton") and this fails to return anyhing. Not sure what is happening here as the value is tokenized it should match either values in the search string.. TermQuery created as follows:

Code:

booleanQuery.add(new TermQuery(new Term("cities", cities[i].toLowerCase())), 
                                   BooleanClause.Occur.SHOULD);

Thanks

sanne.grinovero · **Posted:** Thu Mar 10, 2011 8:30 am

I don't think you looked at org.hibernate.search.test.util.AnalyzerUtils.tokensFromAnalysis. you have to use that, to split the query string in tokens, you can't just lowercase it.

Code:

booleanQuery.add(new TermQuery(new Term("cities", "manchester")),
                                   BooleanClause.Occur.SHOULD);

HibSearchUser80 · **Joined:** Wed Feb 09, 2011 6:49 am **Posts:** 22

aha, ok i'm using v3.0 downloaded 3.3..see what you mean!

thanks