Stop words and fuzzy search

j.joslet · **Joined:** Thu Dec 03, 2009 10:07 am **Posts:** 5

Hello,

I try to make a search on a street field wich can contain words such as "street" (or "rue", "de", "la" in french).

These words are discarded by the analyzer and my index is build correctly without them. I use the Stop Filter Analyzer to do this.

For example, the following fields:

Code:

rue de la paix
neighbour street

are indexed as follow:

Code:

paix
neighbour

I'd like to allow user searching on this field by typing "rue de la paie" or "naighbour street" and retrieve the correct record by using a fuzzy search.

How can I build a query that does not use words that must be skipped by the analyzer ?

Here is more explanations and some examples I have try.

If I do something like this:

Code:

Analyzer analyzer = fullTextSession.getSearchFactory().getAnalyzer("personAnalyzer");
QueryParser p = new QueryParser("person.street", analyzer);
Query q = p.parse( "rue de la paix" );

I get the following query:

Code:

person.street:paix

But now, if I had the fuzzy operator on each words, like this:

Code:

Analyzer analyzer = fullTextSession.getSearchFactory().getAnalyzer("personAnalyzer");
QueryParser p = new QueryParser("person.street", analyzer);
Query q = p.parse( "rue~ de~ la~ paix~" );

I get this query:

Code:

person.street:rue~0.5 person.street:de~0.5 person.street:la~0.5 person.street:paix~0.5

wich is not what I really want.

I'd like to get the following query

Code:

person.street:paix~0.5

when the user type

Code:

rue de la paix

.

How can I get that result ? Is it possible ?

[EDIT] I also get another problem using fuzzy search and accented character.

I try to search the firstname with the following user input "jérô" and my code add a fuzzy operator to the text, resulting in "jérô~".

The query returned by the parser is :

Code:

person.firstname :jérô~0.5

I use the ISOLatin1AccentFilter analyzer to remove these accents but when using the parser and this query, nothing is removed.

How can I remove the accents in this case ?

Thanks in advance for responses.

j.joslet · **Joined:** Thu Dec 03, 2009 10:07 am **Posts:** 5

It seems that analyzer isn't used when fuzzy search is enable. Any workaround to correct that ?

sanne.grinovero · **Posted:** Fri Dec 11, 2009 8:50 pm

Hi, yes I confirm I have had this same problem. It's Lucene's QueryParses which doesn't apply the Analyzer, as you have also verified.
The solution is to avoid the QueryParser and build the equivalent Query programmatically