Newbie: tokenize or not

francof · **Joined:** Fri Aug 17, 2007 2:11 pm **Posts:** 19

I have this indexed property

Quote:

@Column(name = "ITEM_CODE")
@NotNull
@Field(index = Index.UN_TOKENIZED, store = Store.YES)
public String getItemCode() {
return this.itemCode;
}

and 2 other fields which are tokenized.
Itemcode will have unique values with no spaces like DSCH2, RMAV3000 etc

and this query in my search method.

Code:

String searchQuery = this.itemKeyFields ; 
      
String[] itemFields = {"itemCode", "modelName","modelDescription"};

Map<String,Float> boostPerField = new HashMap<String,Float>(3);
      boostPerField.put( "itemCode", 4f); 
      boostPerField.put( "modelName", 2f); 
      boostPerField.put( "modelDescription", 2f); 

QueryParser parser = 
         new MultiFieldQueryParser( itemFields,  new StandardAnalyzer(), boostPerField ); 

With this setup, if I put dsc as a search keyword, I get no results, but if I put in dsc*, I get my desired results because it does a wildcard card search.

My question is should my itemcode field be tokenized too?
If I do that, I would get my results but is it the right way for field like this.

Other than that, the speed of Hibernate Search is phenomenal !!!

Thanks
Franco

francof · **Joined:** Fri Aug 17, 2007 2:11 pm **Posts:** 19

I now set the field to

Code:

@Field
public String getItemCode() {
return this.itemCode;
} 

reindexed the entity, used only this field in the MultiFieldQueryParser and tried a search again.

I still have to put a wildcard at the end of the search term.

So how do I do a wildcard query (without the user needing to put a asterisk) on multiple fields where as in my example 1 of these fields is a unique identifier ?

sanne.grinovero · **Posted:** Tue Feb 17, 2009 3:27 am

Hi, you could try using Luke to "see" and experiment with Analyzers.
Generally they cleanup the noise and split the text in pieces, so

Code:

"Hibernate Search in Action" -> { "hibernate","search","in","action" }

or even

Code:

"Hibernate Search in Action" -> { "hibernate","search","action" }

when using smarter Analyzers.

The same process is applied to your query (of course you should use the same analyzer), and then Lucene will see how many tokens your query has in common with each document.
These terms however have to fully match, or you have to explicitly use a wildcard expression.

In your case the codes shouldn't be analyzed: usually you don't want codes to be processed as they could end up split by numbers, dots or other symbols. You may want to manage the case however.

If you don't want your users having to add the "*" wildcard you can either add it yourself, make your own custom QueryParser, or build the Query programmatically combining WildcardQuery with others.

derkoe · **Joined:** Fri Feb 13, 2009 6:17 am **Posts:** 5

For us the best way to search in multiple fields was to introduce a new field (e.g. "summary" or "all") containing all the values to be searched in. This also helps when you want to search across different types of entities (e.g. Vehicle and BusinessPartner).

In your case this would be:

Code:

@Field
public String getSummary()
{
  return itemCode + " " + modelName + " " + modelDescription;
}

I would not tokenize fields like itemCode - because it is just a code. Only fields containing more words should be tokenized.

To search you can change the query parsing - just extend QueryParser and use your own.

Something like:

Code:

// no special Lucene parsing
if (StringUtils.containsNone(query, "+-~:\"%"))
{
   query = query.toLowerCase();
   String[] strings = query.split("\\s");

   BooleanQuery wild = new BooleanQuery();
   for (String term : strings)
      wild.add(new WildcardQuery(new Term(FIELD_SUMMARY, term + "*")), BooleanClause.Occur.MUST);

   return wild;
}
else
{
   return super.parse(query);
}

--
Chris

francof · **Joined:** Fri Aug 17, 2007 2:11 pm **Posts:** 19

@Chris, I used your solution with a wildcard query and it worked great. Thanks

@Sanne, it was my misunderstanding - did not realized that codes are split by numbers and other symbols. My item code could have a / in it.

In the book, Hibernate Search, a field EAN is indexed untokenized. I assumed my item code had similar meaning that is why I went that direction.

I have lots more reading to do...

Thanks to both of you for your replies, I appreciate your time.

Franco