MultiField search - how to?

James Minter · **Joined:** Tue Oct 09, 2007 4:48 am **Posts:** 11

Hi,

I have a question about Hibernate Search stuff. I've tried to make this as short as I can, but it's hard! :

My basic requirement is to allow a free text search over a set of fields, but also adding addition filters to either add usefulness or enforce security requirements. i.e. at the front end there is a textbox for the users' search and a selectable date range.

I want to be able to search over my objects with the users query, and then filter/limit those results by the date range and my own requirements (i.e. something like: +isPublished:true).

So far, I have been testing this manually by building up a search query like this:

Code:

MultiFieldQueryParser parser = new MultiFieldQueryParser(new String[]{"modifiedDate", "memberName", 
      "agendaItemName", "agendaKeywords", "isPublished"}, new StandardAnalyzer());

String queryString = "+modifiedDate:["+bottomDateString+" TO "+ topDateString + "] +("+
           "member.memberName:"+keywords+" "+
           "agendaItems.agendaItemName:"+keywords+" "+
           "agendaItems.agendaKeywords:"+keywords+" "+
           ") " +
           "+isPublished:true";

Query luceneQuery = parser.parse(queryString);

org.hibernate.Query hibQuery = fullTextSession.createFullTextQuery(luceneQuery, Asset.class);

Although this works for simple cases, it's undeniably horrible and has a number of major show-stoppers in that it will fail completely if the user does a search like "+somesearchquery -someotherquery".

So, my question is - how do I do this programatically with Hibernate search. I notice in the "Lucene in Action" book some interesting bits:

Code:

Query query = MultiFieldQueryParser.parse("development", new String[]{"title", "subjects"}, new SimpleAnalyser());

Translating into my scenario, I have:

Code:

Query lucQ = MultiFieldQueryParser.parse(
   
   new String[] {"isPublished:true","modifiedDate:["+bottomDateString+" TO "+ topDateString + "]",
           keywords, keywords, keywords},
   
   new String[] {"isPublished","modifiedDate", "member.memberName", "agendaItems.agendaItemName", "agendaItems.agendaKeywords"},
   
   new BooleanClause.Occur[]{BooleanClause.Occur.MUST, BooleanClause.Occur.MUST, BooleanClause.Occur.SHOULD, BooleanClause.Occur.SHOULD, BooleanClause.Occur.SHOULD},
   
   new StandardAnalyzer());

Which is supposed to allow me to search over multiple fields. However, when I try this in Hibernate with a query like "-james", it generates me (correctly, I believe) the following:

Code:

+isPublished:true +modifiedDate:[20080223 TO 20080419] (-member.memberName:james) (-agendaItems.agendaItemName:james) (-agendItems.agendaKeywords:james)

The trouble with that is that the first two clauses in the query overpower the remaining clauses. In other words, I will still get results with 'james' in the fields because it is published and validates against a date.

What I need it to generate is:

Code:

+isPublished:true +modifiedDate:[20080223 TO 20080419] +((-member.memberName:james) (-agendaItems.agendaItemName:james) (-agendItems.agendaKeywords:james))

(notice extra "+(" )

I can't see any way of doing this short of parsing once (without my filtered bits) manually adding the "+(" and ")" and then appending my filtered stuff before parsing again and executing the query.

i.e.:

Code:

Query lucQ = MultiFieldQueryParser.parse(
   
   new String[] {keywords, keywords, keywords},
   
   new String[] {"member.memberName", "agendaItems.agendaItemName", "agendaItems.agendaKeywords"},
   
   new StandardAnalyzer());
   
String genQry = lucQ.toString();

String newQry = "+isPublished:true +modifiedDate:[20080223 TO 20080419] +(" + genQry + ")";

Query luceneQuery = parser.parse(newQry);

org.hibernate.Query hibQuery = fullTextSession.createFullTextQuery(luceneQuery, Asset.class);

Am I missing something? While it seems to work, it still feels like a bit of a hack to me...

Many thanks for any help and thanks for reading.

James.

matzetronic · **Joined:** Mon Mar 31, 2008 5:22 am **Posts:** 4

Hi,

I'm facing a similar problem here. I'm using a MultifieldQueryParser to search over an index which contains fields of two different hibernate DAOs. _One_ of these DAOs contains also a Date field.

Since I red that using a RangeQuery is not the best option to restrict a query to a date range (changes the score?! see also http://wiki.apache.org/lucene-java/FilteringOptions), I wrote a FullTextFilter and annotated the DAO which holds the Date type to use that FullTextFilter.

If I now apply that Filter to a query, not only the Objects which are not within the specified date range but also the objects of the other DAO (which don't contain a date type at all and are also not annotated with a @FullTextFilterDef) are filtered out.

So, the question is what's the best way to search over different objects, if one or more hold a specific type(a date for example) you wanna filter out, and the other objects don't contain that type.

Regards,
Mathias

hardy.ferentschik · **Posted:** Mon Mar 31, 2008 7:40 am

Hi,

I think a more programmatic approach is easier to handle than one single string in combination with a MultiFieldQueryParser.

For example you can use a query parser for the freetext search field the user can fill in. This will allow the user to use meta characters like '+' and '-'. For other search criteria you can then construct TermQuerys. In the end you can combine all queries using a BooleanQuery. Check out the Lucene Query hierarchy http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc//org/apache/lucene/search/Query.html and you will see that the possibilities are numerous.

When using date ranges you might want to consider using a Filter, since otherwise you might end up with a TooManyClausesException (http://wiki.apache.org/lucene-java/LuceneFAQ#head-06fafb5d19e786a50fb3dfb8821a6af9f37aa831). Hibernate Search offers several ways to specify filters: http://www.hibernate.org/hib_docs/search/reference/en/html_single/#d0e1982.

I guess bottom line is that you don't have to construct these massive strings. Just work with Lucene's Query classes.

--Hardy

matzetronic · **Joined:** Mon Mar 31, 2008 5:22 am **Posts:** 4

Hi,

I chose a MultifieldQueryParser because I wanted to have all indexed Objects be treated equally in the results. It should be the perfect fit for that use case.

I also already have a filter which works as expected. But it seems that every type is filtered by the FullTextFilter, regardless in which type the @FullTextFilterDef annotation is. This is a bit confusung though, because annotating a type with @FullTextFilterDef makes me think that the filter only applies for that type.

Anyway, I'll try to build another (likely more complex) query, to get this to work as intended. I'll post my solution to this topic when I found a reasonably good one.

Mathias

emmanuel · **Posted:** Mon Apr 07, 2008 5:34 pm

matzetronic wrote:

But it seems that every type is filtered by the FullTextFilter, regardless in which type the @FullTextFilterDef annotation is. This is a bit confusung though, because annotating a type with @FullTextFilterDef makes me think that the filter only applies for that type.

Open a JIRa issue, the doc should be clearer

matzetronic · **Joined:** Mon Mar 31, 2008 5:22 am **Posts:** 4

emmanuel wrote:

matzetronic wrote:

But it seems that every type is filtered by the FullTextFilter, regardless in which type the @FullTextFilterDef annotation is. This is a bit confusung though, because annotating a type with @FullTextFilterDef makes me think that the filter only applies for that type.

Open a JIRa issue, the doc should be clearer

Done ( http://opensource.atlassian.com/project ... SEARCH-179 )

matzetronic · **Joined:** Mon Mar 31, 2008 5:22 am **Posts:** 4

To bring this to an end, I found the following solution acceptable:

As Hardy already noted, I used a hierarchy of boolean queries together with a multifieldqueryparser and a rangequery:

Code:

             // this type needs a special treatment
             if(searchClasses[i] == EventDate.class)
             {
                if(startDate!=null && endDate!=null)
                {
                   BooleanQuery innerBooleanQuery = new BooleanQuery();
                   innerBooleanQuery.add(parser.parse(searchTerm),BooleanClause.Occur.MUST);
                   
                   RangeQuery rq = new RangeQuery( new Term("startDate",DateTools.dateToString(startDate, DateTools.Resolution.MINUTE)),new Term("startDate",DateTools.dateToString(endDate, DateTools.Resolution.MINUTE)), true );
                   innerBooleanQuery.add(rq,BooleanClause.Occur.MUST);
                   outerBooleanQuery.add(innerBooleanQuery, BooleanClause.Occur.SHOULD);
                }
                else
                   outerBooleanQuery.add(parser.parse(searchTerm), BooleanClause.Occur.SHOULD);
             }
             else
                outerBooleanQuery.add(parser.parse(searchTerm), BooleanClause.Occur.SHOULD);

Hope you find it useful.

Regards,
Mathias