Analyzer annotation added to field together with FieldBridge

Andrzej Kojder · **Joined:** Thu Nov 25, 2010 5:04 am **Posts:** 1

Hello,
this is my first post on this forum, so welcome all :)

My problem is that analyzer's annotation doesn't work when added to a field together with FieldBridge (works if Analyzer is set in property file or added on a class level), eg.:

Code:

@Entity
@Indexed
public class MyEntity {

   private Long id = -1L;
   private Address address;

   @Id
   @GeneratedValue(strategy = GenerationType.AUTO)
   @Column(name = "id")
   public Long getId() {
      return id;
   }

//edit: here have @ManyToOne, @JoinColumn and @Cascade annotations
   @Fields({
         @Field(name = "address", bridge = @FieldBridge(impl = AddressBridge.class), index = Index.TOKENIZED, analyzer = @Analyzer(impl = AddressAnalyzer.class)),
         @Field(name = "addressEx", bridge = @FieldBridge(impl = AddressBridge.class), index = Index.TOKENIZED, analyzer = @Analyzer(impl = OtherAddressAnalyzer.class)) })
   public Address getAddress() {
      return address;
   }
}

It does work, when I add the same Analyzer annotation on class level instead field level.
IMHO the implementation of the analyzer doesn't matter, because it does't work even for KeywordAnalyzer.
As a bridge I use simple implementation of FieldBridge, something like:

Code:

public class AddressBridge implements FieldBridge {

   @Override
   public void set(String name, Object value, Document document, LuceneOptions luceneOptions) {
      /*
       * here null- and instanceof checking
       */
      Address address = (Address) value;
      luceneOptions.addFieldToDocument(name + ".street", address.getStreet(), document);
      luceneOptions.addFieldToDocument(name + ".zip", address.getZip(), document);
   }
}

Bridge works ok. To check results i use Luke 1.0.1.
I use hibernate core 3.6.0 Final and hibernate search 3.3.0 CR1.

Is this a bug or I'm doing something wrong?
Does anyone have any idea how to fix it?

Thank you in advance for your help.

sanne.grinovero · **Posted:** Fri Nov 26, 2010 6:33 am

Hi,
welcome to the forums.
I would expect the code you posted to work, could you please open a JIRA ticket and attach a patch with a testcase?
There are other testcases relating to analyzers in the package org.hibernate.search.test.analyzer, it should be easy to adapt one to verify this. One to cover this seems to be missing, so this might be a bug, in that case if you help with a test it shouldn't be hard to fix.

m.feichtegger · **Joined:** Thu Dec 30, 2010 3:09 am **Posts:** 2

Hi everybody,

is there a solution for this problem or an according JIRA issue. I've the same problem and wondered if it is me or a bug :-)

kind regards
Michael

m.feichtegger · **Joined:** Thu Dec 30, 2010 3:09 am **Posts:** 2

Hi once more,

after some more research I can finally say: it was me!

There were two aspects preventing the analyzer to be invoked:

1) I've annotated the field with

Code:

index=Index.UN_TOKENIZED

2) My field bridge did not add any content using

Code:

options.addFieldToDocument(fieldName, fieldValue, document);

hope that helps

kind regards
Michael

sanne.grinovero · **Posted:** Sat Jan 01, 2011 11:45 am

Quote:

is there a solution for this problem or an according JIRA issue. I've the same problem and wondered if it is me or a bug :-)

I added a new unit test to make sure, wasn't happy the test was missing but indeed it looks like there's no bug.

To recap for everybody's sake:
1) always check the value is not null/empty or the field will be skipped
2) the field must be tokenized (indexed)
3) custom FieldBridges must not be broken :)

mkkumar79 · **Joined:** Fri Jun 11, 2010 3:00 am **Posts:** 14

Hi Sanne,

I have similar problem. FieldBridge is not broken, but analyzer defined for the field is not applied.

Versions:
Hibernate Search - 3.4.0.Final
Hibernate Search Analyzers - 3.4.0.Final
Hibernate Core - 3.6.3.Final

Code:

Apps.java
-------------------------------------------------------------------------------

Code:

@AnalyzerDefs({
@AnalyzerDef(name="stemmedAnalyzer", 
             tokenizer = @TokenizerDef(factory= StandardTokenizerFactory.class),
             filters = {
                        @TokenFilterDef(factory=LowerCaseFilterFactory.class),
                        //@TokenFilterDef(factory=StandardFilterFactory.class),
                        @TokenFilterDef(factory=SnowballPorterFilterFactory.class, 
                              params = @Parameter(name="language", value="English"))
}),
@AnalyzerDef(name="normalAnalyzer", 
        tokenizer = @TokenizerDef(factory= StandardTokenizerFactory.class),
        filters = {
                   @TokenFilterDef(factory=LowerCaseFilterFactory.class)
})
})


@Entity
@Table(name = "apps_readonly", uniqueConstraints = @UniqueConstraint(columnNames = {
        "name", "appid" }))
@Indexed(index="Apps")
@Analyzer(definition="stemmedAnalyzer")
public class Apps implements java.io.Serializable  {

    @OneToOne
    @JoinColumn(referencedColumnName="appid", name="appid")
    @IndexedEmbedded(depth=1, prefix="metadata_")
    public Metadata getMetadata(){
        return this.metadata;
    }
    
}

Metadata.java
--------------------------------------------------------

Code:

@Entity
@Table(name = "metadata", uniqueConstraints = @UniqueConstraint(columnNames = "appid"))
public class Metadata implements java.io.Serializable {
   @Column(name = "description", nullable = false, length = 10000)
   
   @Fields({
      @Field(name = "stemmed_desc", index = Index.TOKENIZED, store = Store.NO),
      @Field(name = "unstemmed_desc", index = Index.TOKENIZED, store = Store.NO, analyzer=@Analyzer(definition="normalAnalyzer"))             
   })
   @FieldBridge(impl=ChunkDescriptionBridge.class, params=
             {@Parameter( name="chunkSize", value="100" ), @Parameter( name="delimeter", value=" " )})
    @Boost(value=0.0f)
    public String getDescription() {
      return this.description;
   }
}

ChunkDescriptionBridge
--------------------------------------------------------------------

Code:

public class ChunkDescriptionBridge implements ParameterizedBridge, FieldBridge {

   private Integer chunkSize = null;
   private String delimeter = null;
   private Integer noOfChunks = null;
   
   @Override
   public void set(String name, Object value, Document document,
         LuceneOptions luceneOptions) {
      String description = (String) value;
      String desc1 = "";
      String desc2 = "";
      String desc3 = "";
      String desc4 = "";
      
      if(StringUtils.isNotEmpty(description)){
         String[] descSplitArray = description.split("\\s+");
         Integer descArrayLength = descSplitArray.length;
         
         desc1 = StringUtils.join(descSplitArray, delimeter, 0, descArrayLength > chunkSize ? chunkSize: descArrayLength);
         
         if(noOfChunks > 1 && descArrayLength >= chunkSize){
            desc2 = StringUtils.join(descSplitArray, delimeter, chunkSize, descArrayLength > 2 * chunkSize ? 
                  2 * chunkSize: descArrayLength);
            
            if(descArrayLength >= 2 * chunkSize){
               desc3 = StringUtils.join(descSplitArray, delimeter, 2 * chunkSize, 
                     descArrayLength > 3 * chunkSize ? 3 * chunkSize: descArrayLength);               
            }
            
            if(descArrayLength > 3 * chunkSize){
               desc4 = StringUtils.join(descSplitArray, delimeter, 3 * chunkSize, descArrayLength);               
            }
         }
      }
      
      Field descField = new Field(name + "1", desc1, luceneOptions.getStore(), luceneOptions.getIndex(),
            luceneOptions.getTermVector());
      descField.setBoost(0.3f);
      document.add(descField);

      
      if(noOfChunks > 1 ) {
         descField = new Field(name + "2", desc2, luceneOptions.getStore(), luceneOptions.getIndex(),
               luceneOptions.getTermVector());
         descField.setBoost(0.03f);
         document.add(descField);

         
         descField = new Field(name + "3", desc3, luceneOptions.getStore(), luceneOptions.getIndex(),
               luceneOptions.getTermVector());
         descField.setBoost(0.003f);
         document.add(descField);

         
         descField = new Field(name + "4", desc4, luceneOptions.getStore(), luceneOptions.getIndex(),
               luceneOptions.getTermVector());
         descField.setBoost(0.003f);
         document.add(descField);      

      }
      
   }

   @Override
   public void setParameterValues(Map parameters) {
      
      this.delimeter = (parameters.get("delimeter") == null ? " " : (String)parameters.get("delimeter"));
      this.chunkSize = (parameters.get("chunkSize") == null ? 100 : Integer.parseInt((String)parameters.get("chunkSize")));
      this.noOfChunks = (parameters.get("noOfChunks") == null ? 1 : Integer.parseInt((String)parameters.get("noOfChunks")));
   }

}

Though I have applied normalAnalyzer on Metadata.description field, it still creates stemmed indices. But the field bridge was worked.

NOTE: Just to make sure I have tried with the option suggested by Michael. But it also did not work.

Code:

if(isNotEmpty(desc1)){
         luceneOptions.addFieldToDocument(name + "1", desc1, document);         
      }

sanne.grinovero · **Posted:** Tue Jan 31, 2012 10:59 am

Hi mkkumar79,
could you please test that in a recent version? 4.0.0.Final or 4.1.0.Alpha1

qambard · **Joined:** Sun Apr 15, 2012 11:07 am **Posts:** 5

Hi,
We are having the same issue, with the 4.0.0.Final version
We have 2 Fields using the same bridge, but with differents parameter and different analyzer.
When opening with Luke, it appears that all the fields are correctly stored, but the specified analyzer (here our NGramAnalyzer.class) isn't used.
field are not empty, index == YES and the NGramAnalyzer works as expected (we use the same in our workaround).

Here is our mapping :

Code:

    @Fields({
            @Field(index = Index.YES, store = Store.YES, bridge = @FieldBridge(impl = CompanyBusinessIdBridge.class, params = @Parameter(name = "fieldSubName", value = "ngram")), analyzer = @Analyzer(impl = NGramAnalyzer.class)),
            @Field(index = Index.YES, store = Store.YES, bridge = @FieldBridge(impl = CompanyBusinessIdBridge.class)) })
    private final Map<CompanyBusinessIdType, CompanyBusinessId> businessIds = new HashMap<CompanyBusinessIdType, CompanyBusinessId>();

and our CompanyBusinessIdBridge implementation :

Code:

public class CompanyBusinessIdBridge implements FieldBridge, ParameterizedBridge {
    public static String FIELD_SUB_NAME = "fieldSubName";
    private String fieldSubName = "";

    @Override
    @SuppressWarnings("unchecked")
    public void set(String name, Object value, Document document, LuceneOptions luceneOptions) {
        Map<CompanyBusinessIdType, CompanyBusinessId> businessIds = (Map<CompanyBusinessIdType, CompanyBusinessId>) value;
        for (Entry<CompanyBusinessIdType, CompanyBusinessId> e : businessIds.entrySet()) {
            luceneOptions.addFieldToDocument(name + fieldSubName + e.getKey().name(), e.getValue().getValue(), document);
        }
    }

    @Override
    @SuppressWarnings("rawtypes")
    public void setParameterValues(Map parameters) {
        String subName = (String) parameters.get(FIELD_SUB_NAME);
        if (subName != null) {
            fieldSubName = "_" + subName;
        } else {
            fieldSubName = "";
        }
    }
}

For now we work with a workaround (a unique fieldBrige with our 2 analyzer inside, and we add directly the value to the document, without using the luceneOptions)