Matching untokenised fields

seemarich · **Joined:** Tue Feb 03, 2009 12:29 pm **Posts:** 49

I have a field with the following mapping

@Field(index = Index.UN_TOKENIZED, store = org.hibernate.search.annotations.Store.YES)
@Column(name = "TRIM")
private String trim;

I know that if a field is untokenised, no analyzers are applied. So I'm assuming that in this case, the field is stored in the index as it is. In Luke, I can see the value of the field correctly.

Now the problem is when I search by this field. There are records which have the field's value as 'AWD'. But when I search using Hibernate search passing the query string "trim:AWD", no records are returned.

I can see the value 'AWD' in the index using Luke, but it also does not give any results for the query string "trim:AWD". (However in the case of Luke, I'm not able to search without any analyzer. It automatically takes StandardAnalyzer and I'm not able to turn off that)

What could I be doing wrong? Does Hibernate search also use analyzer even though I specified Index.UN_TOKENIZED

Thanks,
Seema

sanne.grinovero · **Posted:** Fri Mar 06, 2009 5:40 pm

Hibernate Search will not use an Analyzer during index creation/updating for untokenized fields, however for searching you have to make sure you are preparing the Lucene Query correctly.
How do you create the query? could you post some code?

seemarich · **Joined:** Tue Feb 03, 2009 12:29 pm **Posts:** 49

hi,

This is how I construct the query. I do pass the analyzer to the query parser, but I assume that it would not be used for the untokenized fields. Please advice.

Thanks,
Seema

Code:

   FullTextEntityManager fullTextEntityManager = Search
            .getFullTextEntityManager(entityManager);
      entityManager.getTransaction().begin();

      // create native Lucene query
      String[] fields = new String[] { "trim" };
      MultiFieldQueryParser parser = new MultiFieldQueryParser(fields,
            new StandardAnalyzer());
      Query query = parser.parse("trim:AWD");

      // wrap Lucene query in a org.hibernate.Query
      FullTextQuery hibQuery = fullTextEntityManager.createFullTextQuery(
            query, SaleEventItem.class);

      // execute search
      List<SaleEventItem> list = hibQuery.getResultList();
      System.out.println("size is " + list.size());
      for (SaleEventItem saleEventItem : list) {
         System.out.println(saleEventItem.getId() + " "
               + saleEventItem.getMake() + " " + saleEventItem.getYear()
               + " " + saleEventItem.getModel());
      }
      entityManager.getTransaction().commit();
      entityManager.close();

sanne.grinovero · **Posted:** Mon Mar 09, 2009 7:28 am

The MultiFieldQueryParser is a Lucene-provided helper class, it is not aware of the annotations used by Search.
In this case it is applying the analyzer you apply, and so it is lowercasing "AWD".

Use something like

Code:

TermQuery ftQuery = new TermQuery( new Term( "trim","AWD" ) );

or change the Analyzer you are in the QueryParser to something simpler not messing with your cases, or a custom one.

seemarich · **Joined:** Tue Feb 03, 2009 12:29 pm **Posts:** 49

hi,

I have multiple fields to match, which is why I used the MultiFieldQueryParser. I didn't show it in the code just to make it simple. But actually my query needs to be constructed combining expressions that query different fields - some of which might be tokenized and the others untokenized. I cannot do that using TermQuery, right? So in that case, I guess the only option I have is using a custom analyzer for the untokenized field?

Thanks,
Seema

sanne.grinovero · **Posted:** Thu Mar 12, 2009 5:33 am

Hi,
sorry for the delay.

You may want to try using org.apache.lucene.analysisPerFieldAnalyzerWrapper or org.hibernate.search.util.ScopedAnalyzer

so you can reuse Lucene's parser but specify which Analyzer to use for each field.

jkaufma · **Joined:** Tue Mar 31, 2009 10:03 am **Posts:** 2

Hi,

I'm new here, however, I'm experiencing exactly the same problems. I am using Hibernate Search 3.1.0.CR1 with Lucene 2.4.

In my entity bean I have different @AnalyzerDef annotations. For example, one field is annonated with

Code:

@Fields({
   @Field(),   //   not stored, tokenized with default Analyzer of entity "ElementStandardAnalyzer")
   @Field(name = "titlePatterns", analyzer = @Analyzer(definition = "ElementPatternAnalyzer")),
   @Field(name = "titlePlain",    analyzer = @Analyzer(definition = "ElementPlainAnalyzer"))
})
public String getTitle()
{
   return title;
}

The "ElementStandardAnalyzer" is the default analyzer for the entity (@Analyzer annotation at the entity itself).
I also have fields which are not tokenized with an analyzer:

Code:

@Field(store = Store.YES, index = Index.UN_TOKENIZED)
public String getAssignedEAN()
{
   return assignedEAN;
}

I build my query using the QueryParser:

Code:

FullTextEntityManager fem = Search.getFullTextEntityManager(em);
Analyzer luceneAnalyzer = fem.getSearchFactory().getAnalyzer(MyEntity.class);
QueryParser parser = new MultiFieldQueryParser(fields, luceneAnalyzer);

According to Hibernate Search in Action, section 7.2.6 "the ScopedAnalyzer should apply the specified analyzer to its matching field" automatically. And indeed,

Code:

parser.parse("Blink182").toString()

evaluates to something like

Code:

title:blink182 titlePatterns:"blink 182" titlePlain:blink182 assignedEAN:blink182

However, it seems that my "ElementStandardAnalyzer" has been applied to the field assignedEAN even though it is annotated as un-tokenized (you see that it's converted to lower-case). If there really was an EAN "Blink182" in the index it would not be found.

Did I miss something or is it a mistake to use the QueryParser when searching in un-tokenized fields?

Many thanks in advance for any help,
Joachim