hibernate search erased documents

darioc · **Joined:** Wed Aug 06, 2014 10:53 am **Posts:** 30

I have verified that the erased documents are flagged as deleted by lucene index. When the merge process or optimazation process start the flagged documents are erased from index.
That's ok, but is there some way to exclude the deleted object from hibernate search query ?

darioc · **Joined:** Wed Aug 06, 2014 10:53 am **Posts:** 30

remove this topic! The problem is not reproducible

sanne.grinovero · **Posted:** Wed Feb 04, 2015 9:06 pm

Hi Darioc, correct deleted documents are still stored until the next merge, but they are "hidden" and should never be returned by Hibernate Search.
That's just an implementation detail of the storage format.

darioc · **Joined:** Wed Aug 06, 2014 10:53 am **Posts:** 30

Hi,

I think there's something wrong during the delete of the documents.
Unfortunately it's not reproducible in fixed way, but it happens often (about two runs on three).
My workflow is: remove and update two different bean (jpa object/lucene document) in the same Spring transaction.
After that transaction was completed I try to search the previously object deleted, and I found it!!! It's very strange.
If I rebuild the index and try to search again, the document disappears.
What's happening? Someone has any idea?

As Sanne explains, Lucene keeps the deleted object until the merge task starts; but in the meantime the document should be not accessible.

My configuration:
Spring: 4.1.4.RELEASE
Hibernate Search: 5.0.1.Final
Hibernate: 4.3.8.Final

Lucene access type: mmap
JDK: 1.8
OS: Windows

darioc · **Joined:** Wed Aug 06, 2014 10:53 am **Posts:** 30

It might depend on this issue https://issues.apache.org/jira/browse/LUCENE-6166 ??

darioc · **Joined:** Wed Aug 06, 2014 10:53 am **Posts:** 30

Ok, I've found the problem. The migration guide of lucene helped me:
http://lucene.apache.org/core/4_10_3/MIGRATE.html

It writes: "Deleted docs are no longer implicitly filtered from docs/positions enums. Instead, you pass a Bits skipDocs (set bits are skipped) when obtaining the enums. Also, you can now ask a reader for its deleted docs."

So I've decided to apply a filter to exclude deleted documents.

Code:

public class DeletedDocFilter extends Filter {

   @Override
   public DocIdSet getDocIdSet(AtomicReaderContext context, Bits acceptDocs) throws IOException {

      OpenBitSet obs = new OpenBitSet(context.reader().maxDoc());
      Bits liveDocs = MultiFields.getLiveDocs(context.reader());

      for (int i = 0; i < context.reader().maxDoc(); i++) {
         if (liveDocs != null && !liveDocs.get(i)) {
            // document deleted
         }
         else
            obs.set(i);
      }
      return obs;
   }
}

Code:

...
CachingWrapperFilter deletedDocsFilter = new CachingWrapperFilter(new DeletedDocFilter());
FilteredQuery filteredQuery = new FilteredQuery(this.myLuceneQuery, deletedDocsFilter);
FullTextQuery fullTextQuery = fullTextEntityManager.createFullTextQuery(filteredQuery, filteredClasses);
... 