Hibernate search: capture unique stemmed words in document

bwalsh · **Joined:** Sat Jan 31, 2009 7:39 pm **Posts:** 5

Hello

I have the latest version of hibernate search (3.1.0.GA). Very easy to get running. Thanks for a great product.

I need to capture the documentId and the list of unique, stemmed words. Proximity is not important.

Is there a combination of custom analyzer &/or PostInsertEventListener that will work for this ?

Thanks,

-brian

sanne.grinovero · **Posted:** Sun Feb 01, 2009 6:00 am

Hi, glad you like it.
Could you elaborate on your requirement?
what do you mean by "capture"? you want to store the Lucene Document somewhere before it gets written to the index?

bwalsh · **Joined:** Sat Jan 31, 2009 7:39 pm **Posts:** 5

given ...

Code:

public class Post {
   @Id()
   @GeneratedValue(strategy = GenerationType.AUTO)
   @DocumentId
   private Long     articleId ;

   @Column(columnDefinition = "MEDIUMBLOB")
       @Field(index=Index.TOKENIZED, store=Store.NO)   
   private String cleanedContent ;

        // ........
}

public class GenericDAO <T, PK extends Serializable> extends
      HibernateDaoSupport {

   public PK create(T o) {
      Serializable pk = getHibernateTemplate().save(o);
      return (PK) pk;
   }
}

I have several use cases that require the list of unique, stemmed words from Post.cleanedContent. Ideally we would fire off these use cases based on the insert into the index.

My challenge is in understanding the API and determining a place to insert our code.

It seems the @AnalyzerDef / @TokenizerDef / @TokenFilterDef might be one place. However, it didn't seem that a TokenFilter knew the @DocumentId of the field it was processing ?

Another place might be FullTextIndexEventListener.onPostInsert. However, it wasn't clear to me what org.hibernate.event.PostInsertEvent
contains regarding tokens. Perhaps PostInsertEvent.getState() contains tokens?

I'd really prefer not to repeat the token/stem process that hibernate search already does.

[/code]

hardy.ferentschik · **Posted:** Tue Feb 03, 2009 11:00 am

Hi,

I am still not quite sure what you want to achieve. Are you looking for a way to implement word stemming for your [i]cleanedContent[i] field or do you actually need the generated (and potentially stemmed tokens) from the indexing process in a separate part of your application?

For the former @AnalyzerDef together with an appropriate chain of tokenizers and token filters will work. For the latter you might want to implement a custom analyzer which gives you access to the indexing process. You could for example intercept the tokens form the indexing process and do whatever you want with them. You can specify a custom analyzer as part of the @Field annotation.

--Hardy

bwalsh · **Joined:** Sat Jan 31, 2009 7:39 pm **Posts:** 5

>actually need the generated (and potentially stemmed tokens) from the indexing process in a separate part of your application?

yes exactly, I'll look at creating the custom analyzer