Search query searching over all fields

mueller · **Joined:** Mon Mar 10, 2008 6:40 pm **Posts:** 114

Hibernate search works great, I'm very happy with the whole mechanism. But I can't seem to figure out how to search over multiple fields very well. So obviously it's pretty easy to qualify search terms with fields and search multiple fields that way:
If I have 3 fields to search on and somebody searches for "word":
field1:word field2:word field3:word
If I have 3 fields to search on and somebody searches for "word1 word2":
field1:word1 field1:word2 field2:word1 field2:word2 field3:word1 field3:word2

But this gets pretty ugly and I imagine inefficient... I have MANY more than 3 fields to search on! Is this the appropriate way?

sanne.grinovero · **Posted:** Sat Feb 20, 2010 7:07 pm

Use a MultiFieldQueryParser, like from the example code in the maven archetype:

Code:

private Query searchQuery(String searchQuery) throws ParseException {

        String[] bookFields = {"title", "subtitle", "authors.name", "publicationDate"};

        //lucene part
        Map<String, Float> boostPerField = new HashMap<String, Float>(4);
        boostPerField.put(bookFields[0], (float) 4);
        boostPerField.put(bookFields[1], (float) 3);
        boostPerField.put(bookFields[2], (float) 4);
        boostPerField.put(bookFields[3], (float) .5);

        FullTextEntityManager ftEm = org.hibernate.search.jpa.Search.getFullTextEntityManager((EntityManager) em);

        QueryParser parser = new MultiFieldQueryParser(bookFields, ftEm.getSearchFactory().getAnalyzer("customanalyzer"),
                boostPerField);

        org.apache.lucene.search.Query luceneQuery;
        luceneQuery = parser.parse(searchQuery);

        final FullTextQuery query = ftEm.createFullTextQuery(luceneQuery, Book.class);

        return query;
    }

BTW I'd suggest to use a QueryParser also for single field queries, so that you can apply the same Analyzer you use for indexing and parse complex user imput properly

mueller · **Joined:** Mon Mar 10, 2008 6:40 pm **Posts:** 114

exactly what I was looking for! Thanks.

mueller · **Joined:** Mon Mar 10, 2008 6:40 pm **Posts:** 114

This seems to be exactly what I feared I'd have to do manually... which I guess is good in that I don't have to write the code for this, but I was hoping there was a more elegant and efficient solution to querying Lucene with many many fields. Am I using MultiFieldQueryParser the right way?

Code:

MultiFieldQueryParser.parse(query,
    new String[] {"names.full", "names.fullPhonetic", "names.prefix"},
    new BooleanClause.Occur[] {BooleanClause.Occur.SHOULD, BooleanClause.Occur.SHOULD, BooleanClause.Occur.SHOULD},
    new StandardAnalyzer())

Calling toString() on this with a query of "word1 word2 word3" yields:

Code:

(names.full:word1 names.full:word2 names.full:word3) (names.fullPhonetic:word1 names.fullPhonetic:word2 names.fullPhonetic:word3) (names.prefix:word1 names.prefix:word2 names.prefix:word3)

which is right I guess, just naively hoping for something more elegant and efficient since that was only 3 fields and I have a large number...

sanne.grinovero · **Posted:** Sun Feb 21, 2010 9:11 am

that's right; don't worry, it might not look elegant but that's efficient in Lucene's world.

mueller · **Joined:** Mon Mar 10, 2008 6:40 pm **Posts:** 114

Thank you Sanne for explaining this. The MultiFieldQueryParser is very useful, I like how it does not apply the multiple field prefixes to query terms which are already qualified with field prefixes:
word1 field:word2
becomes
(multiField1:word1 field:word2) (multiField2:word1 field:word2)

excellent. Except there are issues. If you pass in only a term that has a field prefix:
field:word
it becomes
(field:word field:word)

That's if there are 2 multifield fields, if there are 10 then you get 10 copies of the same search term... So I'm worried I'm not using MultiFieldQueryParser correctly. Basically, I want to allow users to search specific fields, but if they don't specify a field qualifier/prefix, then I search a bunch of fields for them. So far, MultiFieldQueryParser seems to do this, though with strange inefficiencies like above.

sanne.grinovero · **Posted:** Mon Feb 22, 2010 4:10 am

Hi,
consider that the multifieldQueryParser is just a parser, after that it's still optimized.
To have it rewrite the query, either use Luke or write some code like:

Code:

SearchFactory searchFactory = session.getSearchFactory();
DirectoryProvider[] directoryProviders = searchFactory.getDirectoryProviders( Product.class );
IndexReader indexReader = searchFactory.getReaderProvider().openReader( directoryProviders );
System.out.println("toString2: " + query.rewrite( indexReader ).toString());
searchFactory.getReaderProvider().closeReader( indexReader );

But this is only useful for inspection, it's done automatically before the Query is performed.
Look at how a RangeQuery is rewritten, that is scary.

Quote:

(field:word field:word)

that doesn't seem right, but I don't think it's your fault. It will work correctly; it's likely going to affect the scoring but will be relative to all results so it won't affect order of results either.
For better answers about the QueryParsers I'd suggest you to send a mail to Lucene mailing list (thay don't have a forum, but do archive emails):
http://lucene.apache.org/java/docs/mailinglists.html
I think they could fix this, or explain why it does like you saw.