Query DSL and class bridge

magix · **Posted:** Tue Jan 04, 2011 12:21 pm

Hi,

I indexed a JPA-entity via a Class-Bridge annotation, and so far everything is fine (BTW I'm using Hibernate Search 3.3.0 Final).
But when I want to search for a indexed property, I get the following error:

Code:

org.hibernate.search.SearchException: Unable to find field detail.BE_en in it.hsearch.test.CollectDetail
   at org.hibernate.search.engine.DocumentBuilderIndexedEntity.objectToString(DocumentBuilderIndexedEntity.java:607)
   at org.hibernate.search.query.dsl.impl.ConnectedMultiFieldsTermQueryBuilder.buildSearchTerm(ConnectedMultiFieldsTermQueryBuilder.java:133)
   at org.hibernate.search.query.dsl.impl.ConnectedMultiFieldsTermQueryBuilder.createQuery(ConnectedMultiFieldsTermQueryBuilder.java:92)
   at org.hibernate.search.query.dsl.impl.ConnectedMultiFieldsTermQueryBuilder.createQuery(ConnectedMultiFieldsTermQueryBuilder.java:73)

It seems that Hibernate Search cannot find or doesn't know of the field detail.BE_en although via Luke I can see it in the index.

Here's my entity:

Code:

@Entity
@Table( name = "COLLECT_DETAIL" )
@Indexed
@ClassBridge(name="detail",
      index=Index.TOKENIZED,
      store=Store.YES,
      impl = CollectDetailBridge.class)
public class CollectDetail {


   private Long pk;
    private String tag;
    private String lang;
    private Long   occ;
    private String content;
    
   @Id    
   public Long getPk() {
      return pk;
   }
   public void setPk(Long pk) {
      this.pk = pk;
   }
    
   public String getTag() {
      return tag;
   }
   public void setTag(String tag) {
      this.tag = tag;
   }
   public String getLang() {
      return lang;
   }
   public void setLang(String lang) {
      this.lang = lang;
   }
   public Long getOcc() {
      return occ;
   }
   public void setOcc(Long occ) {
      this.occ = occ;
   }
   public String getContent() {
      return content;
   }
   public void setContent(String content) {
      this.content = content;
   }

}

and this is the used class bridge

Code:

public class CollectDetailBridge implements FieldBridge {

   public void set(String name, Object value, Document document, LuceneOptions luceneOptions) {
      CollectDetail detail = (CollectDetail) value;
      String tag     = detail.getTag();
      String lang    = detail.getLang();
      Long   occ     = detail.getOcc();
      String content = detail.getContent();
      String fieldName = getFieldName(name, tag, lang);
      
      
      Field field = new Field(fieldName, content, luceneOptions.getStore(),
      luceneOptions.getIndex(), luceneOptions.getTermVector() );
      field.setBoost( luceneOptions.getBoost() );
      document.add( field );
   }
   
   private String getFieldName(String name, String tag, String lang) {
      if (lang.equals(""))
         return name+"."+tag;
      else
         return name+"."+tag+"_"+lang;
   }
}

The query code goes as follows:

Code:

      EntityManagerFactory entityManagerFactory = Persistence.createEntityManagerFactory("pu");
      EntityManager em = entityManagerFactory.createEntityManager();
      FullTextEntityManager fullTextEntityManager = Search.getFullTextEntityManager(em);
      
      em.getTransaction().begin();
      QueryBuilder qb = fullTextEntityManager.getSearchFactory()
         .buildQueryBuilder().forEntity( CollectDetail.class ).get();
      org.apache.lucene.search.Query query = qb.keyword()
         .onFields("detail.BE_en")
         .matching("test")
         .createQuery();
      // wrap Lucene query in a javax.persistence.Query
      Query persistenceQuery = fullTextEntityManager.createFullTextQuery(query, CollectDetail.class).setMaxResults(100);
      // execute search
      List<CollectDetail> result = persistenceQuery.getResultList();
      System.out.println(result.size());
      
      em.getTransaction().commit();
      em.close();   

Is this not possible with Hibernate Search or am I missing something?

Thanks
Matthias

hardy.ferentschik · **Posted:** Tue Jan 04, 2011 1:03 pm

Since the field you are searching for is not a entity property, but a field you generate via the class bridge you have to use the ignoreFieldBridge() option. Try something like this:

Code:

      EntityManagerFactory entityManagerFactory = Persistence.createEntityManagerFactory("pu");
      EntityManager em = entityManagerFactory.createEntityManager();
      FullTextEntityManager fullTextEntityManager = Search.getFullTextEntityManager(em);
      
      em.getTransaction().begin();
      QueryBuilder qb = fullTextEntityManager.getSearchFactory()
         .buildQueryBuilder().forEntity( CollectDetail.class ).get();
      org.apache.lucene.search.Query query = qb.keyword()
         .onFields("detail.BE_en").ignoreFieldBridge()
         .matching("test")
         .createQuery();
      // wrap Lucene query in a javax.persistence.Query
      Query persistenceQuery = fullTextEntityManager.createFullTextQuery(query, CollectDetail.class).setMaxResults(100);
      // execute search
      List<CollectDetail> result = persistenceQuery.getResultList();
      System.out.println(result.size());
      
      em.getTransaction().commit();
      em.close();   

--Hardy

magix · **Posted:** Tue Jan 04, 2011 1:15 pm

Thanks for your immediate reply.

This way it works, but which Analyzer will be used now?

hardy.ferentschik · **Posted:** Tue Jan 04, 2011 1:26 pm

It is still the same analyzer used. In your example since I cannot see anything else it is your default analyzer. It is also important to notice why the code looks up the field bridge here. It allows you to using matching() with a non string value, eg a Date. In plain Lucene you would have to create a string from the date. Using the query DSL the object passed to the matching() method is passed through the field bridge of the field you are searching for. This way you don't have to this conversion manually. Have a look at http://docs.jboss.org/hibernate/stable/ ... y-querydsl , especially the Keyword queries section.
Maybe you have some suggestions on how to improve the documentation.

magix · **Posted:** Wed Jan 05, 2011 4:08 am

But how can I change the analyzer per field, for example for the field detail.BE_en I would like to use an English stemmer, while for
the field detail.BE_de a German stemmer.

In the documentation I found a section about Dynamic Analyzer. Is this the way to go?

Matthias

magix · **Posted:** Wed Jan 05, 2011 5:40 am

Mmh, I tried to put an @AnalyzerDisriminator and analyzers on the entity like this

Code:

@Entity
@Indexed
@ClassBridge(name="detail",
      index=Index.TOKENIZED,
      store=Store.YES,
      impl = CollectDetailBridge.class)
@AnalyzerDiscriminator(impl = LanguageDiscriminator.class)      
@AnalyzerDefs({
  @AnalyzerDef(name = "de",
          tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
          filters = {
            @TokenFilterDef(factory = LowerCaseFilterFactory.class),
            @TokenFilterDef(factory = GermanStemFilterFactory.class)
  }),
  @AnalyzerDef(name = "it",
        tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
          filters = {
            @TokenFilterDef(factory = LowerCaseFilterFactory.class),
            @TokenFilterDef(factory = SnowballPorterFilterFactory.class, 
                  params = { @Parameter(name="language", value= "Italian") }),
  }),          
  @AnalyzerDef(name = "en",
    tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
    filters = {
      @TokenFilterDef(factory = LowerCaseFilterFactory.class),
      @TokenFilterDef(factory = EnglishPorterFilterFactory.class
      )
    })
})

In the index the gernan word "Mäuse" is reduced to "mau", so the stemmer is used.
But if I search on this field, the analyzer is not applied and I don't find "Mäuse".
If I use overridesForField() to set the correct analyzer for the specific field then it works:

Code:

      QueryBuilder qb = fullTextEntityManager.getSearchFactory()
         .buildQueryBuilder()
         .forEntity( CollectDetail.class )
            .overridesForField("detail.BE_de", "de") //use german stemmer
         .get();

However I thought that the same analyzers should be applied out of the box?

hardy.ferentschik · **Posted:** Wed Jan 05, 2011 6:58 am

Hi Matthias,

seems you really give the query DSL a good testing, especially in connection with some quite advanced Search features like analyzer discriminators. In your case where the used analyzer depends on some language property using @AnalyzerDiscriminator is the right way forward.
And you are right that the right analyzer should be applied to the matching parameter of the query. However, in the context of the query we don't have an entity context which can be used to determine the analyzer to be used. You have to "know" which analyzer to apply on which field. overridesForField is the right solution. Analyzer discriminator really only helps when creating/updating the index. Does this explanation make sense to you?

--Hardy

magix · **Posted:** Wed Jan 05, 2011 9:44 am

Hi Hardy,

well, I don't understand this and I'm confused.
I mean, I'm constructing a QueryBuilder like this

Code:

      QueryBuilder qb = fullTextEntityManager.getSearchFactory()
         .buildQueryBuilder()
         .forEntity( CollectDetail.class )
         .get();

so I would suppose that Hibernate Search knows that I'm searching on the CollectDetail entity. Why then can't he use the analyzer discriminator for CollectDetail?
Is this only a missing feature or are you saying that this is a general, not resolvable limitation?

Thanks for any clarification,
Matthias

magix · **Posted:** Wed Jan 05, 2011 11:45 am

Ah, maybe I got the point now.
As the AnalyzerDiscriminator can depend on the entity state, it is not possible to use it on query time.
Right?

Perhaps it would be an option to have an AnalyzerDiscriminator which doesn't depend on state but only on the field names (useful only at class level? or also on field level if a FieldBridge split up the content?) by implementing this method instead:

Code:

public String getAnalyzerDefinitionName(String field)

Then it should be possible for HibernateSearch to use it also at query time?

Thanks
Matthias

hardy.ferentschik · **Posted:** Wed Jan 05, 2011 11:57 am

I think it is really a general limitation. Have a look at the Discriminator interface:

Code:

public interface Discriminator {

   /**
    * Allows to specify the analyzer to be used for the given field based on the specified entity state.
    *
    * @param value The value of the field the <code>@AnalyzerDiscriminator</code> annotation was placed on. <code>null</code>
    * if the annotation was placed on class level.
    * @param entity The entity to be indexed.
    * @param field The document field.
    * @return The name of a defined analyzer to be used for the specified <code>field</code> or <code>null</code> if the
    * default analyzer for this field should be used.
    * @see org.hibernate.search.annotations.AnalyzerDef
    */
   String getAnalyzerDefinitionName(Object value, Object entity, String field);
}

The idea is to return the definition name of an analyzer based on an actual entity instance and/or property value (depending on whether you placed the annotation on class or property level).

In the query case you just specify an entity type not an actual instance. For this reason we cannot really know which analyzer to apply unless you explicitly use overridesForField. Theoretically we could use the LanguageDiscriminator provided the implementation only depends on the passed field name, but this would be against the whole idea of analyzer discriminator.

One way to solve your problem without overridesForField or analyzers discriminator would be to use @ClassBridges like this:

Code:

@ClassBridges ( {
   @ClassBridge(name="detail.BE_de",
              index= Index.TOKENIZED,
              impl = CollectDetailBridge.class,
                              analyzer = @Analyzer(definition="de")
            ),
           @ClassBridge(name="detail.BE_en",
              index= Index.TOKENIZED,
              impl = CollectDetailBridge.class,
                               analyzer = @Analyzer(definition="en"
            )
})

Maybe it would make sense to have an additional interface class bridges could implement and allow us to query the bridge which fields it created and which analyzer to use for these fields. That would be a new feature though. Feel free to create a Jira issue for that. Something like this might be useful for other problems as well.

I hope this helps,
--Hardy

hardy.ferentschik · **Posted:** Wed Jan 05, 2011 12:02 pm

Hi Matthias,

seems we posted our answers pretty much at the same time and I see you understood the problem now.
Having some "special" analyzer discriminator which only depends on the field name might be one solution, but not supported atm.
The other is to use @ClassBridges as in my other answer not last but not least the option of an additional interface for class bridge. As I mentioned this might be the most sensible solution in the long run, especially since we have been discussing before that we might need to extend the class/field bridge api.

magix wrote:

Ah, maybe I got the point now.
As the AnalyzerDiscriminator can depend on the entity state, it is not possible to use it on query time.
Right?

Perhaps it would be an option to have an AnalyzerDiscriminator which doesn't depend on state but only on the field names (useful only at class level? or also on field level if a FieldBridge split up the content?) by implementing this method instead:

Code:

public String getAnalyzerDefinitionName(String field)

Then it should be possible for HibernateSearch to use it also at query time?

Thanks
Matthias

emmanuel · **Posted:** Wed Jan 05, 2011 2:26 pm

We could add a new concept that returns an analyzer based on a field name (assuming the field name is not found - like a email catch-all in a way) but it seems that the ability to use .overridesForField solves the problem (albeit in a procedural way as opposed to a declarative / systematic way).

Matthias, can you first try the .overridesForField approach and tell us if it's really an issue that way?

magix · **Posted:** Sat Jan 08, 2011 12:49 pm

Yes, .overridesForField solves the problem, but nonetheless I think this is a missing feature.

If I can put bridges on field and on class-level, why should I put analyzers only on fields and not also on classes? Currently this is possible only statically for the whole class, or dynamically via @AnalyzerDiscriminator, but in this case only 1 analyzer can be selected.

Take the example from Hibernate Search in Action, where a Map is "bridged" to the index generating n fields based on the keys in the map. The keys are languages, if I recall correctly, and the values are some text. So how would you analyze those values using the correct stemmers for each language?

Matthias

hardy.ferentschik · **Posted:** Sat Jan 08, 2011 2:37 pm

Hi Matthias,

good to hear that the overridesForField approach worked for you. I agree that there is still a gap in Search when it comes to class bridges (and custom field bridges) and analyzers. There are several ways to address this, but it will be in upcoming releases of Search. It might be an extended bridge api or as Emmanuel hinted some sort of new concept based on field names only.
This discussion will hopefully just be the beginning of another Search feature :)

--Hardy

emmanuel · **Posted:** Mon Jan 10, 2011 9:14 am

magix wrote:

Yes, .overridesForField solves the problem, but nonetheless I think this is a missing feature.

If I can put bridges on field and on class-level, why should I put analyzers only on fields and not also on classes? Currently this is possible only statically for the whole class, or dynamically via @AnalyzerDiscriminator, but in this case only 1 analyzer can be selected.

Take the example from Hibernate Search in Action, where a Map is "bridged" to the index generating n fields based on the keys in the map. The keys are languages, if I recall correctly, and the values are some text. So how would you analyze those values using the correct stemmers for each language?

To be fair, you can write an analyzer that delegates to other analyzers based on the field name today. Check ScopedAnalyzer for an example. But I do agree that this is not easy and does not feel natural with the rest of HSearch's configuration.

Can somebody open a JIRA and summarize the discussion so we don't forget that conversation?