Hi,

How can I sort text coming in multiple languages? I have a DB filed called Country which is entered by users from 15 different countries. In the UI we use Hibernate Search to display data and sort. However sorting is creating an issue. Its not sorting based on multiple languages. Only English seems to get acknowledged. Example if I have 3 texts like
1. 中國
2. China
3. Germany
4. France

When sorted it is coming like this

1. China
2. 中國
3. France
4. Germany


I was expecting 中國 to be at the end because its unicode is greater.

I am using the following code to index the city field
@AnalyzerDef(name = "keyword", tokenizer = @TokenizerDef(factory = KeywordTokenizerFactory.class), filters = {
@TokenFilterDef(factory = CollationKeyFilterFactory.class, params = {
@Parameter(name = "language", value = ""),
@Parameter(name = "strength", value = "primary")
}),
@TokenFilterDef(factory = LowerCaseFilterFactory.class)
})


@Transient
@Field(name="city", index = Index.UN_TOKENIZED,store = Store.YES, analyzer = @Analyzer(definition = "keyword")) {
public String getCity() {
......
}
However this does not seem to work

Hi,

I think your problem is that you don't specify a language parameter in the CollationKeyFilterFactory. Leaving this parameter blank means that the system Locale is used for indexing/searching. I am not sure whether there is a Collator which purely sorts looking at a unicode defined order.

--Hardy

Maybe a RuleBasedCollator is something you could look into, but I am not sure how much effort it is to create the required sorting rules. Have a look at the java.text.RuleBasedCollator docs.

Well leaving the language parameter empty implies that it is going to support all the available languages as per the Solr wiki.

Wondering if there is any other way. Interestingly even empty values are not in correct order in the sort.

Is there an ASCII sort available in Lucene?