In FullTextFilter why do field names not have same count?

Mocchi · **Joined:** Fri Mar 19, 2010 7:00 pm **Posts:** 14

Ok so I'm almost done understanding filters in Hibernate Search and have one more roadblock. In the following method:

Code:

public DocIdSet getDocIdSet(IndexReader reader) throws IOException {
    OpenBitSet bitSet = new OpenBitSet(reader.maxDoc());

    TermDocs termDocs = reader.termDocs();
    TermEnum termEnum = reader.terms();

    while (termEnum.next()) {

        String fieldName = termEnum.term().field();
        System.out.println("FIELDS OUTPUT: " + fieldName);
        if (fieldName.equals(field)) {
            termDocs.seek(termEnum.term());

            if (termDocs.next()) {
                    
                    int doc = termDocs.doc();

..........

notice in particular the following print and IF estatements:

Code:

        System.out.println("FIELDS OUTPUT: " + fieldName);
        if (fieldName.equals(field)) {

When I print the field names some of the fields get printed out dozens of times while others are only printed out a couple of times. And when I do the check "fieldName.equals(field)" this is passing up other rows from the filtering and thus my data set is incomplete in my main.

So my question is WHY is that check needed to check if the field names are equal and WHY aren't the field names matching up in number of occurrences? I would think that if 20 rows are given back in a SELECT query that each row has an identical number of field names (say you select first_name, last_name, user_name and there are 20 rows that get returned, then EACH ROW SHOULD have a first_name, last_name, user_name - but I'm not getting that, more like two first_name, 15 first_name, and 20 user_name, AND YET I do have all the rows and fields if I just comment out the IF check field names....).

Does this make sense? Any help is greatly appreciated.

* I'm using lucene 2.4.1 I believe - not sure how to check

sanne.grinovero · **Posted:** Sat Mar 27, 2010 9:21 pm

Quote:

I would think that if 20 rows are given back in a SELECT query that each row has an identical number of field names

Welcome to the inverted index world :)
Each text when analyzed is tokenized in several tokens, depending on the string contents and on the Analyzer you selected.
A simple Analyzer would split the sentence "Hello World" into two tokens hello and world, and you'll find this as two fields in the Document. Same concept applied for when you index a graph of objects and have contained fields which are repeated - the repetition is reflected in the index as multiple fields.

Mocchi · **Joined:** Fri Mar 19, 2010 7:00 pm **Posts:** 14

Thank you for the reply. So how about the check if field equals the field name? Is that needed? How do I make sure i check every single row? Thanks

Mocchi · **Joined:** Fri Mar 19, 2010 7:00 pm **Posts:** 14

Can anyone chime in? is if (fieldName.equals(field)) required in filters? I have taken it out of my filters and they seem to work now...