Bulk manual indexing

apparna · **Joined:** Fri Aug 01, 2008 1:11 pm **Posts:** 3

Hi, I'm using Hibernate Search 3.0.1 GA and I'm running into a strange issue while trying to bulk index some records in the database.

I have a field bridge on a property (Name) within an entity class(Customer). When I'm trying to manually index, somehow these field bridges aren't invoked.

Code:

         ScrollableResults results = fullTextSession.createCriteria(Customer.class).scroll(ScrollMode.FORWARD_ONLY);

Transaction transaction = fullTextSession.beginTransaction();
         int index=0;
         while(results.next())
         {
             index++;
             fullTextSession.index((Customer)results.get(0));
             if (index % 1000 == 0) 
             {
                // hibernateSession.clear();
                 fullTextSession.flush(); //apply changes to indexes
                 fullTextSession.clear(); //clear since the queue is processed
                 System.out.print(new Date() + " ... batch flush ..."); 
             }
             
         }
         transaction.commit();

The same field bridge works properly if I use

Code:

Customer customer = new Customer(name, phonenumber);
em.persist(customer);

What am i missing here?

Thanks,
Apparna

hardy.ferentschik · **Posted:** Wed Aug 06, 2008 9:35 am

Hi,

is name the only property you are trying to index or are there more. If so, are the other properties indexed when bulk indexing?

Could you maybe provide the code for your entities and bridges?

--Hardy

emmanuel · **Posted:** Thu Aug 07, 2008 9:54 am

and your configuration file. 1000 must also be used on the batch size property.
Actually a better approach has been added to Hibernate 3.1 with an explicit session.flushToIndexes() method.

apparna · **Joined:** Fri Aug 01, 2008 1:11 pm **Posts:** 3

Thanks Emmanuel and Hardy for your response.

Hardy: I solved the problem. We have a custom backend and the run method of the backendqueueprocessor for some reason gets DeleteLuceneWork when I'm doing this bulk update. That was the source of the problem.

Code:

public void run() {
      List<LuceneWork> filteredQueue = new ArrayList<LuceneWork>(queue);
      for (LuceneWork work : queue)
      {
         if (work instanceof OptimizeLuceneWork || work instanceof DeleteLuceneWork) 
         {
            // we don't want optimization to be propagated
            filteredQueue.remove(work);
         }
      }
:
}

for now, I'm removing the DeleteLuceneWork from the queue but I still need to find out why I'm getting the DeleteLuceneWork when all I'm trying to do is bulk index.

Emmanuel: I'm using batch size as 1000

In 3.0.1 GA, is there a need to flush the session as well? I'm doing it though Hibernate Search in Action book, you had mentioned just session.clear

Code:

if (index % 1000 == 0)
             {
                // hibernateSession.clear();
              fullTextSession.flush(); //Is this needed?
                 fullTextSession.clear(); //clear since the queue is processed
                 System.out.print(new Date() + " ... batch flush ...");
             } 

Quote:

Actually a better approach has been added to Hibernate 3.1 with an explicit session.flushToIndexes() method.

But Hibernate Search 3.1, requires hibernate core 3.3 and Hibernate annotations and EM 3.4 and I've no idea when JBoss will migrate to the latest versions so we can actually start using Hibernate Search the way its supposed to be. Since my company has a enterprise licence with JBoss we can only use what JBoss provides us. So unfortunate, since they haven't still upgraded the hibernate annotations/em jars which are from dec 2006! :(

BTW, I'm very impressed with whatever I have seen so far with Hibernate Search
Flexibility: Write your own backend if you need to
Annotations: So cool! :)
Custom Bridges: You guys thought of everything that ppl will need out of the box.

I just hope JBoss upgrades their hibernate annotations/em for their EAP 4.3 version so I can officially start using Hibernate Search.

Thanks,
Apparna

emmanuel · **Posted:** Fri Aug 08, 2008 4:44 pm

Ah interesting, can you give us a bit more info about your backend and the reason you chose to write a custom one (or even better post that on the community section of the wiki)?

I understand, EAP 5 will target the latest version of Hibernate Core and annotations. We might even have Hibernate Search officially in there :)

If you don't change your objects, you do not need to flush() the session.

When indexing an object, Hibernate Search does not know if the object was indexed before, so we apply two operations, a delete and an addition.