Excessive Indexing

jarnis · **Joined:** Mon May 23, 2011 12:41 pm **Posts:** 3

Hey Guys

I'm new to this forum (and fairly new to hibernate search) so please fogive me if my question is triviel or has been answered before. Here goes:

I have a hibernate-persisted class, which I'm trying to make searchable using hibernate search. When loading the classes from the database, the search index is appearantly being updated heavily. When starting with an empty index folder (I'm using FSDirectoryProvider until I resolve this problem), after application startup (webapp under tomcat) the folder contains 2 files:

segments.gen
segments_1

After I load the page, which contains a list with the searchable objects on it, the content of the folder is:

_17.cfs
_18.cfs
_19.cfs
_1a.cfs
_1b.cfs
_1c.cfs
_1d.cfs
_1e.cfs
_1f.cfs
_a.cfs
_l.cfs
_w.cfs
segments.gen
segments_2x

Notice, no search in the index has been performed yet. The list of objects is retreived using HibernateDaoSupport/HibernateTemplate. After loading the page a second time, the folder contains this:

_17.cfs
_17_1.del
_1i.cfs
_1t.cfs
_24.cfs
_2f.cfs
_2q.cfs
_2r.cfs
_2s.cfs
_2t.cfs
_2u.cfs
_2v.cfs
_2w.cfs
_a.cfs
_a_1.del
_l.cfs
_l_1.del
_w.cfs
_w_1.del
segments.gen
segments_5w

In my search, the list of all the searchable objects is loaded again, which makes me unsure if the changes are due to the search, or just due to reloading the list:

_32.cfs
_3d.cfs
_3o.cfs
_3z.cfs
_4a.cfs
_4b.cfs
_4c.cfs
_4d.cfs
_4e.cfs
segments.gen
segments_8x

Furthermore, loading the list page takes significantly longer than it did with my previous search solution Compass, which also uses Lucene as backend (the reason I'm switching away from Compass is that it requires Terracotta DSO/Custom and we are switching to Terracotta Express/Standard, and after I resolve this problem I intend to make or modify a DirectoryProvider to use the Terracotta Express as storage). I'm not 100% sure, but to me it seems that my objects are indexed each time they are loaded via hibernate.

Can anyone point me in the direction to look for a solution? I will gladly provide any relevant info if the above is not enough to go by, just let me know what info is needed. My current hibernate dependencies:
hibernate: 3.2.5.ga (can't easily be upgraded because of other dependencies)
hibernate-search: 3.0.0.GA
hibernate-commons-annotations 3.3.0.ga

Any help will be appreciated. Thanks in advance,
Jarnis Bertelsen

jarnis · **Joined:** Mon May 23, 2011 12:41 pm **Posts:** 3

Extra info:
Further testing has revealed that the reindexing occurs only once each time the page is reloaded (or the struts Action.execute method is called). It does not matter if the list with all the searchable objects is retrieved more than once or a search is performed more than once. Only the first call the the hibernate DAO seems to cause reindexing, and it does not matter if it is a search or a "normal" DAO call.

sanne.grinovero · **Posted:** Tue May 24, 2011 5:17 am

Hi jarnis,
welcome.

Quote:

hibernate: 3.2.5.ga (can't easily be upgraded because of other dependencies)
hibernate-search: 3.0.0.GA
hibernate-commons-annotations 3.3.0.ga

wow.. that's quite old indeed, as it was the first release ever of Search, you're going to miss a ton of improvements: I'll assume your search requirements are very limited?

Some hints on the internal design of Search might help:
Hibernate Search will never reindex anything unless Hibernate (core) detects that you have updated the entities. So it looks like that something during your page load is affecting the state of entities?
From the file list you posted, it looks like it's reindexing a lot of stuff, not just a single entity (or maybe a single entity, but multiple times).
A common error I see is people having getters and setters on their entities which actually contain some logic; that will confuse Hibernate core.

example (wrong) :

Code:

setName(String name) { this.name = name.toLowerCase(); }
getName() { return this.name; }

should be

Code:

setName(String name) { this.name = name }
getName() { return this.name; }
@Transient setNameNormalize(String name) { this.name = name.toLowerCase(); }

As in the first case when Hibernate detects dirty entities, it will notice that the value it loaded was changed in some way, you can also detect this case when enabling SQL logging: you'll see unneeded database update statements. Please enable such logging, so you should be able to verify which entities are considered dirty and being flushed, and your page should be much faster as you'd be avoiding both index updates and database updates.

generally, at least upgrade to version 3.0.1.GA as it's 3.0.0 compatible.
More recent versions (like 3.4) have much smarter dirty checking detection routines! Some interesting reason to report why you can't update hibernate core?

jarnis · **Joined:** Mon May 23, 2011 12:41 pm **Posts:** 3

Solved!

Turned out one of the setter methods on the indexed/persisted object did not directly assign the passed value to the object's field. Instead it did some date arithmetic and assigned a different Date object (created within the setter). When hibernate called the setter while fetching the object from the database, it resulted in a changed value, which resulted in reindexing. With 5 fields with that kind of assignment and 475 records, that was appearantly enough to cause the massive reindexing, though I suspect it might have been made worse due to some kind of feedback loop.

Changing the setters to make a direct assignment (at least in the cases, where hibernate is calling the setter during object loading) solved the problem.

Thanks for the help s.grinovero.

The reason we use Hibernate 3.2.5.ga is because AppFuse depends on that version, which does suck as AppFuse is no longer maintained and is not easily removed from the app.

sanne.grinovero · **Posted:** Mon May 30, 2011 2:16 pm

great!
and thanks for reporting your experience.