Weird Lucene behaviour: The same "document" w/ multiple ids

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

Hi all,

i have a strange behaviour here:

I have an Entity with a composite key. The key is mapped as component and annotated with
@EmbeddedId
@DocumentId
@FieldBridge(impl = ProductTextBridge.class)
@Field(index=Index.UN_TOKENIZED,store=Store.YES)
protected ProductText_Id id;

After having indexed all items, if i now search the index with Luke (or from the application), i get the identical items (same key) with different DocumentIds stored in the index.

What's going wrong ?

Br, Mark

hardy.ferentschik · **Posted:** Thu Nov 05, 2009 3:49 am

Hi Mark,

first try removing the @Field annotation. If this does not help post your code for the ProductTextBridge. Is your bridge implementing TwoWayFieldBridge or TwoWayStringBridge? It has to.

--hardy

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

Hi hardy,

thanks for the reply. I meanwhile changed a lot of things and now it seems to work. Yes, it did implement the TwoWayFieldBridge, but the individual properties were marked as Index_TOKENIZED. (I think, this was the cause)

I also tried marking a different field (hascode) as documentID but that threw interesting exceptions (first, hibernate tried to cast the hashcode to a string. So i implemented to hashcodestring function that provided a string. But then hibernate tried to cast the string to ProductText_Id... Strange strange....)

So, left this topic is the question: How can I define what the documentID should be ? If i use Luke to browse the index, the documentIDs look like they are incrementally generated by Lucene. But I really would prefer to use something related to the field i marked as DocumentId. Is this possible and if yes - how ?

-- Mark

hardy.ferentschik · **Posted:** Thu Nov 05, 2009 7:09 am

Quote:

thanks for the reply. I meanwhile changed a lot of things and now it seems to work. Yes, it did implement the TwoWayFieldBridge, but the individual properties were marked as Index_TOKENIZED. (I think, this was the cause)

What do you mean with individual properties were marked as Index_TOKENIZED? Depending of what you are indexing it is probably a good thing to tokenize. When it comes to your document id, then of course not. The document id must be un-tokenized. It might actually help if you post your fully annotated entities together with your bridge implementation.

Quote:

So, left this topic is the question: How can I define what the documentID should be ? If i use Luke to browse the index, the documentIDs look like they are incrementally generated by Lucene. But I really would prefer to use something related to the field i marked as DocumentId. Is this possible and if yes - how ?

The document id is whatever you make it in your ProductTextBridge? I don't understand what you mean.

--Hardy

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

OK, problem is not solved. I have still the same issue. :-(

This is my hibernate-mapping file:

Code:

<class name="de.bauerlive.ngshop.data.ProductText" table="product_text">
    <cache usage="nonstrict-read-write"/>
    <composite-id class="de.bauerlive.ngshop.data.identifiers.ProductText_Id" name="id">
      <key-property column="MANDT" length="4" name="mandt" type="string"/>
      <key-property column="LANG" length="2" name="lang" type="string"/>
      <key-property column="PRODID" length="40" name="prodid" type="string"/>
    </composite-id>
    <property column="STEXT" length="80" name="stext" type="string"/>
    <property name="ltext" type="text"/>
    <many-to-one class="de.bauerlive.ngshop.data.Product" fetch="select" insert="false" name="product" update="false">
      <column length="4" name="mandt" not-null="true"/>
      <column length="40" name="prodid" not-null="true"/>
    </many-to-one>
  </class>
<class name="de.bauerlive.ngshop.data.Product" table="product">
    <cache usage="nonstrict-read-write"/>
    <composite-id class="de.bauerlive.ngshop.data.identifiers.Product_Id" name="id">
      <key-property column="MANDT" length="4" name="mandt" type="string"/>
      <key-property column="PRODID" length="40" name="prodid" type="string"/>
    </composite-id>
    <property column="DATBI" name="datbi" type="timestamp"/>
    <property column="DATAB" name="datab" type="timestamp"/>
    <property column="FOLGEPROD" length="40" name="folgeprod" type="string"/>
    ...
  </class>

Here are the two Id-component classes:

Code:

@Analyzer(impl=de.bauerlive.ngshop.logic.BLNGAnalyzer.class)
public class ProductText_Id implements Serializable {
    private static final long serialVersionUID = 8194101847993970529L;
    @Field(index=Index.UN_TOKENIZED, store=Store.YES)
    protected String mandt;
    @Field(index=Index.UN_TOKENIZED,store=Store.YES)
    protected String lang;
    @Field(index=Index.UN_TOKENIZED, store=Store.YES, boost=@Boost(2f))
    protected String prodid;

    @Override
    public String toString() {
        return Integer.toString(this.hashCode());
    }

    /**
     * Get the value of prodid
     *
     * @return the value of prodid
     */
    public String getProdid() {
        return prodid;
    }

    /**
     * Set the value of prodid
     *
     * @param prodid new value of prodid
     */
    public void setProdid(String prodid) {
        this.prodid = prodid;
    }

    /**
     * Get the value of lang
     *
     * @return the value of lang
     */
    public String getLang() {
        return lang;
    }

    /**
     * Set the value of lang
     *
     * @param lang new value of lang
     */
    public void setLang(String lang) {
        this.lang = lang;
    }

    /**
     * Get the value of mandt
     *
     * @return the value of mandt
     */
    public String getMandt() {
        return mandt;
    }

    /**
     * Set the value of mandt
     *
     * @param mandt new value of mandt
     */
    public void setMandt(String mandt) {
        this.mandt = mandt;
    }

    @Override
    public boolean equals(Object obj) {
        if (obj == null) {
            return false;
        }
        if (getClass() != obj.getClass()) {
            return false;
        }
        final ProductText_Id other = (ProductText_Id) obj;
        if ((this.mandt == null) ? (other.mandt != null) : !this.mandt.equals(other.mandt)) {
            return false;
        }
        if ((this.lang == null) ? (other.lang != null) : !this.lang.equals(other.lang)) {
            return false;
        }
        if ((this.prodid == null) ? (other.prodid != null) : !this.prodid.equals(other.prodid)) {
            return false;
        }
        return true;
    }

    @Override
    public int hashCode() {
        int hash = 3;
        hash = 19 * hash + (this.mandt != null ? this.mandt.hashCode() : 0);
        hash = 19 * hash + (this.lang != null ? this.lang.hashCode() : 0);
        hash = 19 * hash + (this.prodid != null ? this.prodid.hashCode() : 0);
        return hash;
    }

}

and

Code:

public class Product_Id implements Serializable {
    private static final long serialVersionUID = -1647812012628341082L;

    @Override
    public boolean equals(Object obj) {
        if (obj == null) {
            return false;
        }
        if (getClass() != obj.getClass()) {
            return false;
        }
        final Product_Id other = (Product_Id) obj;
        if ((this.mandt == null) ? (other.mandt != null) : !this.mandt.equals(other.mandt)) {
            return false;
        }
        if ((this.prodid == null) ? (other.prodid != null) : !this.prodid.equals(other.prodid)) {
            return false;
        }
        return true;
    }

    @Override
    public int hashCode() {
        int hash = 7;
        hash = 89 * hash + (this.mandt != null ? this.mandt.hashCode() : 0);
        hash = 89 * hash + (this.prodid != null ? this.prodid.hashCode() : 0);
        return hash;
    }

    protected String mandt;
    protected String prodid;

    /**
     * Get the value of prodid
     *
     * @return the value of prodid
     */
    public String getProdid() {
        return prodid;
    }

    /**
     * Set the value of prodid
     *
     * @param prodid new value of prodid
     */
    public void setProdid(String prodid) {
        this.prodid = prodid;
    }

    /**
     * Get the value of mandt
     *
     * @return the value of mandt
     */
    public String getMandt() {
        return mandt;
    }

    /**
     * Set the value of mandt
     *
     * @param mandt new value of mandt
     */
    public void setMandt(String mandt) {
        this.mandt = mandt;
    }

}

And here the field bridge:

Code:

public class ProductTextBridge implements TwoWayFieldBridge {

    public Object get(String fieldName, Document doc) {
        ProductText_Id questionPK = new ProductText_Id();
        Field field = null;
        field = doc.getField(fieldName + ".mandt");
        questionPK.setMandt(field.stringValue());
        field = doc.getField(fieldName + ".prodid");
        questionPK.setProdid(field.stringValue());
        field = doc.getField(fieldName + ".lang");
        questionPK.setLang(field.stringValue());
        return questionPK;
    }

    public String objectToString(Object obj) {
        ProductText_Id questionPK = (ProductText_Id) obj;
        StringBuilder sb = new StringBuilder();
        sb.append(questionPK.getMandt()).append(" ").append(questionPK.getLang()).append(" ").append(questionPK.getProdid());

        return sb.toString();
    }

    public void set(String fieldName, Object obj, Document doc, Store store, Index index, Float boost) {
        ProductText_Id questionPK = (ProductText_Id) obj;
        Field field = new Field(fieldName + ".mandt", questionPK.getMandt(), store, index);

        if (boost != null) {
            field.setBoost(boost);
        }

        doc.add(field);

        field = new Field(fieldName + ".lang", questionPK.getLang(), store, index);

        if (boost != null) {
            field.setBoost(boost);
        }

        doc.add(field);

        field = new Field(fieldName + ".prodid", questionPK.getProdid(), store, index);

        if (boost != null) {
            field.setBoost(boost);
        }

        doc.add(field);
    }

    @Override
    public void set(String arg0, Object arg1, Document arg2, LuceneOptions arg3) {
        set(arg0, arg1, arg2, arg3.getStore(), arg3.getIndex(), arg3.getBoost());
    }


}

and

Code:

public class ProductBridge implements TwoWayFieldBridge {

    public Object get(String fieldName, Document doc) {
        Product_Id questionPK = new Product_Id();
        Field field = null;
        field = doc.getField(fieldName + ".mandt");
        questionPK.setMandt(field.stringValue());
        field = doc.getField(fieldName + ".prodid");
        questionPK.setProdid(field.stringValue());

        return questionPK;
    }

    public String objectToString(Object obj) {
        Product_Id questionPK = (Product_Id) obj;
        StringBuilder sb = new StringBuilder();
        sb.append(questionPK.getMandt()).append(" ").append(questionPK.getProdid());

        return sb.toString();
    }

    public void set(String fieldName, Object obj, Document doc, Store store, Index index, Float boost) {
        Product_Id questionPK = (Product_Id) obj;
        Field field = new Field(fieldName + ".mandt", questionPK.getMandt(), store, index);

        if (boost != null) {
            field.setBoost(boost);
        }

        doc.add(field);

        field = new Field(fieldName + ".prodid", questionPK.getProdid(), store, index);

        if (boost != null) {
            field.setBoost(boost);
        }

        doc.add(field);
    }

    @Override
    public void set(String arg0, Object arg1, Document arg2, LuceneOptions arg3) {
        set(arg0, arg1, arg2, arg3.getStore(), arg3.getIndex(), arg3.getBoost());
    }


}

If i now search with

Code:

FullTextSession fullTextSession = Search.getFullTextSession(session);
            Transaction tx = fullTextSession.beginTransaction();
            Analyzer anal = new BLNGAnalyzer();

            String[] fields = new String[]{"prodid", "stext", "ltext", "product.eans", "product.manufacturer_aid", "product.manufacturerName"};
            MultiFieldQueryParser parser = new MultiFieldQueryParser(fields, anal);

            //QueryParser parser = new QueryParser("prodid", anal);
            parser.setAllowLeadingWildcard(true);
            parser.setDefaultOperator(Operator.AND);

            BooleanQuery lquery = new BooleanQuery();

            try {
                lquery.add(parser.parse(entered), BooleanClause.Occur.MUST);
                lquery.add(new TermQuery(new Term("mandt", mvh.getUser().getMandt())), BooleanClause.Occur.MUST);
                lquery.add(new TermQuery(new Term("lang", mvh.getUser().getLang().toLowerCase())), BooleanClause.Occur.MUST);
                lquery.add(parser.parse(catSearch), BooleanClause.Occur.MUST);
                
            } catch (ParseException ex) {
                Logger.getLogger(ProductDAO.class.getName()).log(Level.SEVERE, null, ex);
            }
            FullTextQuery hibQuery = fullTextSession.createFullTextQuery(lquery, ProductText.class);
            hibQuery.setCacheable(true);

            hibQuery.setMaxResults(aThis.getMaxResults());
            hibQuery.setFirstResult(aThis.getFirstResult());
            List l = hibQuery.list();

I get multiple times the SAME product (they definitely have the same values of their identifiers). I really don't get it.... :-(

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

Sorry, i forgot the class itself:

Code:

@Entity
@Indexed
@Analyzer(impl = de.bauerlive.ngshop.logic.BLNGAnalyzer.class)
public class ProductText implements Serializable {

    private static final long serialVersionUID = -8429603356751414390L;
    @EmbeddedId
    @DocumentId
    @Field(index = Index.UN_TOKENIZED, store = Store.YES)
    @FieldBridge(impl = ProductTextBridge.class)
    protected ProductText_Id id;
    private Product product;

    
    public String getHashcode() {
        return Integer.toString(this.hashCode());
    }

    /**
     * Get the value of id
     *
     * @return the value of id
     */
    public ProductText_Id getId() {
        if (id == null) {
            id = new ProductText_Id();
        }
        return id;
    }

    /**
     * Set the value of id
     *
     * @param id new value of id
     */
    public void setId(ProductText_Id id) {
        this.id = id;
    }

    @ManyToOne(fetch = FetchType.EAGER)
    @JoinColumns({
        @JoinColumn(name = "prodid", nullable = false),
        @JoinColumn(name = "mandt", nullable = false)
    })
    @IndexedEmbedded
    public Product getProduct() {
        if (product == null) {
            product = new Product(getMandt(), getProdid());
        }
        return this.product;
    }

    public void setProduct(Product product) {
        this.product = product;
    }
    @Field(store = Store.YES, boost =
    @Boost(2f))
    private String stext;
    @Field
    private String ltext;

    public String getLtext() {
        return ltext;
    }

    public void setLtext(String ltext) {
        this.ltext = ltext;
    }

    public String getStext() {
        return stext;
    }

    public void setStext(String stext) {
        this.stext = stext;
    }

    public void setProdid(String prodid) {
        getId().setProdid(prodid);
    }

    public void setMandt(String mandt) {
        getId().setMandt(mandt);
    }

    public void setLang(String lang) {
        getId().setLang(lang);
    }

    @Field(index = Index.TOKENIZED, store = Store.YES)
    public String getProdid() {
        return getId().getProdid();
    }

    @Field(index = Index.TOKENIZED, store = Store.YES)
    public String getMandt() {
        return getId().getMandt();
    }

    @Field(index = Index.TOKENIZED, store = Store.YES)
    public String getLang() {
        return getId().getLang();
    }

    @Override
    public boolean equals(Object obj) {
        if (obj == null) {
            return false;
        }
        if (getClass() != obj.getClass()) {
            return false;
        }
        final ProductText other = (ProductText) obj;
        if (this.id != other.id && (this.id == null || !this.id.equals(other.id))) {
            return false;
        }
        if ((this.stext == null) ? (other.stext != null) : !this.stext.equals(other.stext)) {
            return false;
        }
        if ((this.ltext == null) ? (other.ltext != null) : !this.ltext.equals(other.ltext)) {
            return false;
        }
        return true;
    }

    @Override
    public int hashCode() {
        int hash = 7;
        hash = 37 * hash + (this.id != null ? this.id.hashCode() : 0);
        return hash;
    }
}

The problem lies not within the search - the results are correct.
The problem seems to be the storage, because if I use Luke to browse the index and perform the search there, i get the same results:

Multiply times the same "Object" but with different DocId's.

hardy.ferentschik · **Posted:** Fri Nov 13, 2009 4:16 am

Hi,

that a lot of code. First of all I would start with removing superfluous annotations. There should be no @Field annotation when you use @DocumentId. All the @Field annotations in ProductText_Id are also not needed.

Not sure why you posted the ProductBridge. It stands a little in isolation here. Unless Product defines its own index and you want search for Product instances directly you don't need it.

I recommend to enable trace output for org.hibernate.search.backend.impl.lucene.works.AddWorkDelegate. Enabling trace for this class will write the contents of the Lucene documents which are getting stored. This might help to narrow down the problem.

Are you otherwise sure that your key generation is unique? If you index one single instance, what happens to be in the index?

--Hardy

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

Hi,

yes, you are right. Product und ProductBridge are just included for completeness - but unnecessary.

So, in

Code:

@EmbeddedId
    @DocumentId
    @FieldBridge(impl = ProductBridge.class)
    @Field(index = Index.TOKENIZED, store = Store.YES)
    protected Product_Id id;

The @Field is unnecessary, and it is stored and tokenized anyway ?
The same for the ProductText_Id ?

I need to search these fields by +(id.prodid:CAN) and want to find elements like CAN-C-023. Of course, the same results should show up if i search for +(id:prodid:can). I of course search also the other fields.

A possible key for this is (mandt:0001,prodid:CAN-C-023,lang:DE). This is also the primary key of my table and is far is I understood all i have to do is to give it a TwoWayFieldBridge. But the generated DocIds (seen by Luke) are ascending numeric values that do not seem to have any relation to the above key. (They start from 1).

My problem is, that i get the same product multiple times in my search results (they are not distinct). I assume this happens because everytime i store the ProductText object with the above key, it does not overwrite the "old" document, but instead creates a new one.

What did i oversee ?

--Markus

hardy.ferentschik · **Posted:** Fri Nov 13, 2009 6:10 am

Quote:

The @Field is unnecessary, and it is stored and tokenized anyway ?
The same for the ProductText_Id ?

I need to search these fields by +(id.prodid:CAN) and want to find elements like CAN-C-023

That's fine, but your bridge implementation is taking care of this. That's what you are doing in "public void set(String fieldName, Object obj, Document doc, Store store, Index index, Float boost){...}" You are creating these fields programmatically. Having a @Field annotation is not prerequisite for making a property searchable. Think of a ClassBridge. You could define a single ClassBridge for a indexed entity and in this bridge you could then add arbitrary field names (not related to the actual entity properties). You still will be able to search for these fields. Obviously, the code is not as transparent, since you will have to know the field names. Using @Field makes it more transparent what the searchable field names are.
Besides, the @Field annotations in ProductText_Id are not used. They would only make sense in a indexed embedded case (which is not the case here).

Quote:

A possible key for this is (mandt:0001,prodid:CAN-C-023,lang:DE). This is also the primary key of my table and is far is I understood all i have to do is to give it a TwoWayFieldBridge. But the generated DocIds (seen by Luke) are ascending numeric values that do not seem to have any relation to the above key. (They start from 1).

Given your custom bridge you should have a field with the key "__HSearch_id" and the value of "questionPK.getMandt()).append(" ").append(questionPK.getLang()).append(" ").append(questionPK.getProdid()"

--Hardy

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

Hi Hardy,

yet - thanks a lot. That solves my (minor ;-) ) issue that the field "id.mandt" is two times in the document (if I look at it with Luke). I always wondered why :-)

Nevertheless, the Object (0001,CAN-C-053,DE) is still duplicated with a new DocID everytime I store it with session.saveOrUpdate(pt).
Any idea why Lucene does not recognize it as already stored and just overwrites it ? Is there a bug in an update function ?

hardy.ferentschik · **Posted:** Fri Nov 13, 2009 6:47 am

Hi,

Good that we removed one issue :)
For your actual problem I recommend you are looking at your session/transaction management. Are you sure that Hibernate treats the update as an update and not as a new insertion. In fact I think the latter is what's happening here. Lucene cannot recognize it as already stored since Lucene does not have a concept of Document identity (or whatever you would like to call it).
In fact Lucene Documents can also not just be updated. An update is the removal of the old document and addition of a new (modified) Document. When an object gets updated via Hibernate, Hibernate Search first creates a so called DeleteLuceneWork instance followed by a AddLuceneWork instance. However, if Hibernate does not know that there was an instance update, it might just trigger a AddLuceneWork which would lead to "duplicate" Lucene Documents.
This kind of problem can occur when you use wrong Session and Transaction management. It is one of the most common errors and occurs quite often on this forum.
In which environment are you using Hibernate/Hibernate Search? How does your indexing/updating code look like? I recommend you review some of the old post regarding session management on this forum.

--Hardy

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

Hi Hardy,

wow - thanks for that insight ! This indeed gives me a better idea on what's happening.

In the database of course is only stored one instance (because elsewise the primary key would be duplicate). But as I use MySQL (it's a JSF web application with TomCat 6), it could be that "under the hood" they just use the REPLACE-command.

This is how i store the attributes (it is a general "save"-Method):

Code:

public static void save(Object sc) {
        SessionFactory sessionFactory = HibernateUtil.getSessionFactory();
        Session session = sessionFactory.openSession();
        session.beginTransaction();
        try {
            session.saveOrUpdate(sc);
            session.getTransaction().commit();
        } catch (Exception e) {
            Logger.getLogger(ShopDAO.class.getName()).log(Level.SEVERE, null, e);
            session.getTransaction().rollback();
        }
        session.close();
    }

So, storing a mapped object is just a simple "dao.save(o);"

I think i remember a flag "select before update" for the mapping - would this solve my problem ?

hardy.ferentschik · **Posted:** Fri Nov 13, 2009 7:54 am

Hi Mark,

Personally I would use the Open Session in View pattern in a web application. You are opening/closing a session for each operation.
Maybe enable trace output for org.hibernate.event.def.DefaultSaveOrUpdateEventListener. This will give you valuable information about the entity state. Do you capture SQL log messages? Maybe that's enough to give you an indication on what's going on.

--Hardy

mbauer77 · **Joined:** Sun Jul 12, 2009 9:53 am **Posts:** 21

Hi Hardy,

it's not only JSF, but also Facelets and Richfaces with a lot of includes (both ui and a4j) and ajax requests, so using the SessionInView-Pattern is quite difficult to use in this environment. So for the current version, I'll stick to opening a session per Request and maybe in a later version changing this. The performance loss is not so big (in comparison with the current problem ;-) )

In the SQL log it performs an update. :-/ is there a listener where i can view what Hibernate Search is doing (deleting and adding documents) ?

sanne.grinovero · **Posted:** Tue Nov 24, 2009 6:38 pm

Quote:

it's not only JSF, but also Facelets and Richfaces with a lot of includes (both ui and a4j) and ajax requests

looks like a job for Seam, would really help in managing state correctly.

To see what Search is doing, besides debugging, did you try to enable TRACE level logging on org.hibernate.search package?