hibernate.AssertionFailure collection [] was not processed

afishler · **Joined:** Wed Jul 30, 2008 5:50 am **Posts:** 15

Hi,

I have a batch process pushing objects into the db and indexing them at the same time using Hibernate Search.

From time to time I get an exception hibernate.AssertionFailure collection [] was not processed by flush() for a certain collection within one of the classes involved in the indexing.

The collection is anyways not involved in the indexing and is probably empty and not relevant.

I have researched the web for this and found no distinct direction.

When I move into manual indexing - which means I stop indexing and leave it for a later, other offline process....the exceptions disappear.

Seems like a burden not to index while pushing the objects...it can save me lots of time.

does anyone have a direction?

Cheers,
Arie

sanne.grinovero · **Posted:** Mon Jan 12, 2009 3:12 pm

please, a stacktrace? which versions of the hibernate and lucene jars?
maybe some entity code?

afishler · **Joined:** Wed Jul 30, 2008 5:50 am **Posts:** 15

Hibernate 3.3.1
Hibernate search 3.0.1
Lucene 2.4.0

Stack Trace:
org.hibernate.AssertionFailure: collection [com.zibaba.catalog.domain.Dictionary
Category.taggedDictionaryCategoryData] was not processed by flush()
at org.hibernate.engine.CollectionEntry.postFlush(CollectionEntry.java:2
28)
at org.hibernate.event.def.AbstractFlushingEventListener.postFlush(Abstr
actFlushingEventListener.java:356)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlus
hEventListener.java:51)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java
:137)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doComm
it(HibernateTransactionManager.java:655)
at org.springframework.transaction.support.AbstractPlatformTransactionMa
nager.processCommit(AbstractPlatformTransactionManager.java:709)
at org.springframework.transaction.support.AbstractPlatformTransactionMa
nager.commit(AbstractPlatformTransactionManager.java:678)
at org.springframework.transaction.interceptor.TransactionAspectSupport.
commitTransactionAfterReturning(TransactionAspectSupport.java:321)
at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:116)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:204)
at $Proxy70.processShopProductWithReturnStatus(Unknown Source)
at com.zibaba.catalog.connector.MultiThreadedConnector$PushTask.run(Mult
iThreadedConnector.java:325)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:907)
at java.lang.Thread.run(Thread.java:619)

The class with the annotations

@Entity
@Inheritance(strategy = InheritanceType.SINGLE_TABLE)
@DiscriminatorColumn(
name = "Category_Type",
discriminatorType = DiscriminatorType.STRING
)
@DiscriminatorValue("Open")
@GenericGenerator(
name="category-hilo",
strategy = "hilo",
parameters = {
@Parameter(name="table", value = "categories_unique_keys"),
@Parameter(name="column", value = "next"),
@Parameter(name="max_lo", value = "1000")
}
)

@Table(name="dictionary_category")
// We do not keep unique index on the long name although we should. This is in order
// to avoid the limit of the key length.
@org.hibernate.annotations.Table(appliesTo="dictionary_category", indexes={
@org.hibernate.annotations.Index(name="index_1", columnNames={"Long_Name", "Source_Shop"}),
@org.hibernate.annotations.Index(name="index_2", columnNames={"Name"})
})
@Indexed(index="categories")
@Analyzer(impl = EnglishSnowballAnalyzer.class)
@BatchSize(size=100)
public abstract class DictionaryCategory implements java.io.Serializable {

@Id
@GeneratedValue(generator="category-hilo")
@Column(name = "Id")
@DocumentId
private Integer id;

@Column(name = "Name", length = 255)
@Field(index=Index.TOKENIZED, store=Store.NO)
@Boost(4)
protected String name;

@ManyToOne
@JoinColumn(name = "Parent_Id")
@org.hibernate.annotations.Cascade(value = org.hibernate.annotations.CascadeType.DELETE_ORPHAN)
private DictionaryCategory parent;

@OneToMany(mappedBy="dictionaryCategories")
@ContainedIn
private Set<Product> productsWithThisCategory = new HashSet<Product>(0);

@OneToMany(cascade={CascadeType.ALL}, mappedBy="dictionaryCategory")
//@JoinColumn(name="Dictionary_Category_Id")
private Set<TaggedDictionaryCategoryData> taggedDictionaryCategoryData = new HashSet<TaggedDictionaryCategoryData>(0);

Would Appreciate any help!

afishler · **Joined:** Wed Jul 30, 2008 5:50 am **Posts:** 15

Just to emphasize again....the collection assumed not to be flushed is empty all the time. We currently have no data for is...so nothing is actually changing there!

mwiesner · **Posted:** Sat Jan 17, 2009 5:13 am

I experience the same problem with a different domain model, having even newer versions of the Hibernate/Hibernate Search jars as above:
Hibernate-Core 3.3.1.GA
Hibernate Annotations 3.4.0 GA
Hibernate search 3.1GA
Lucene 2.4.0

I get the same Exception, with different stacktrace...:

Quote:

org.hibernate.AssertionFailure: collection [com.xyz.model.Category.Internal_Advertisement] was not processed by flush()
at org.hibernate.engine.CollectionEntry.postFlush(CollectionEntry.java:228)
at org.hibernate.event.def.AbstractFlushingEventListener.postFlush(AbstractFlushingEventListener.java:356)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:124)
at org.hibernate.collection.PersistentSet.size(PersistentSet.java:162)
....

Where entity Category contains:

Code:

private java.util.Set<com.xyz.model.Internal_Advertisement> Internal_Advertisement 

The Internal_Advertisment is not marked as @Indexed at this point.

So i can confirm this "bug"!

When i tried to annotate all the entities representing the type within the corresponding Set (Collection) as @Indexed this error message disappears but I get the next complaint from hibernate that:

Quote:

"A duplicate collection was found in Type ..."

(something like that don't remember exactly). Already reverted my changes to that point, sry.

I even hooked in (for debug purposes) and extended from

Code:

public class MyFullTextIndexEventListener extends
      FullTextIndexEventListener {
   
   private static final long serialVersionUID = 1L;

   public void onPostRecreateCollection(PostCollectionRecreateEvent event) {
      super.onPostRecreateCollection( event );
   }

   public void onPostRemoveCollection(PostCollectionRemoveEvent event) {
      super.onPostRemoveCollection(event);   
   }

   public void onPostUpdateCollection(PostCollectionUpdateEvent event) {
      super.onPostUpdateCollection(event);
   }
   
   protected void processCollectionEvent(AbstractCollectionEvent event) {
      super.processCollectionEvent(event);
   }

and then registering this one as EventListener:

Code:

      <!-- START HIBERNATE SEARCH / LUCENCE STUFF -->
         <event type="post-update">
         <listener class="com.xyz.search.MyFullTextIndexEventListener" />
      </event>
       <event type="post-insert">
         <listener class="com.xyz.search.MyFullTextIndexEventListener" />
      </event>
      <event type="post-delete">
         <listener class="com.xyz.search.MyFullTextIndexEventListener" />
      </event>
      <event type="post-collection-recreate">
         <listener class="com.xyz.search.MyFullTextIndexEventListener" />
      </event>
      <event type="post-collection-remove">
         <listener class="com.xyz.search.MyFullTextIndexEventListener" />
      </event>
      <event type="post-collection-update">
         <listener class="com.xyz.search.MyFullTextIndexEventListener" />
      </event>  

But that did not help. I simply don't understand yet, why the empty sets are touched ?!!?

Any ideas on how to solve the problem? I agree, that my sets are empty and should not be index or "touched" at all...

Regards,
Martin

sanne.grinovero · **Posted:** Thu Jan 22, 2009 7:29 am

yes this is definitely not the meant behaviour.
could you please open a Jira request?
http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH

thanks

mwiesner · **Posted:** Mon Feb 16, 2009 11:45 am

Could you please do that. Have no account and no time ATM. But this is nevertheless an important isssue.

Regards,
Martin

sanne.grinovero · **Posted:** Wed Feb 18, 2009 10:32 am

my fault, it is reported already: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-178
Unfortunately it depends on a Hibernate Core issue.

mwiesner · **Posted:** Wed Feb 18, 2009 12:11 pm

I just read the stuff behind http://opensource.atlassian.com/projects/hibernate/browse/HHH-3225 but I don't quite understand where the proposed "fix" should be applied in conjunction with the hibernate search problem as described above. Could anyone of the hibernate-search cracks in this forum please help with this issue?

Regards,
Martin

jlspea01 · **Joined:** Thu May 21, 2009 1:37 pm **Posts:** 2

I am on hibernate-3.3.1, annotations-3.4.0 and search-3.1.0 and was having this issue too. My app is web based using struts2 and I also use spring. I have 'OpenSessionInView' set up so that I would never have to deal with garbage like this.

According to HSEARCH-178 the issue is resolved only quite recently but I am not certain whether it is in the 3.1.0 release and am not going to download and build since I like to suffer with the herd when I don't have time to dig into code myself

The workaround was quite simple... Make sure that your ManyToOne associations for indexed (and indexedembedded) objects have fetch=FetchType.Lazy set explicitly.

-- begin long story --

Firstly, I use spring and I was SSSSOOOOO tempted to set @Transactional at the top of all my Daos and be done with it, but I
don't like having a problem like this and not knowing why the fix works... cargo-cult is not for me. setting @Transactional DID help me do the troubleshooting however and I could get past the point where I was getting a stack trace and see where my operation was starting and stopping.

For 2 days, this drove me absolutely nuts. So much so that I considered dropping @Indexed from my class and just reindexing when the classes changed. (Not a bad option for me since 90% of my data is not indexed and the 10% that is RARELY changes)... but that particular out did not work anymore with search 3.1.0

During that time I saw about 4 different ways that people have been trying to hack up search and hibernate to make the problem go away. In my case I took the long (old school) way. I turned logging waaaay up and noticed that when my class is @Indexed, I am generating a TON more sql than when it is not... as I dug in a bit deeper, I noticed that when my class is @Indexed, that lazy loading is not the default behaviour for a @ManyToOne association.

Luckily for me, setting @ManyToOne(fetch=FetchType.Lazy) was respected and worked... oh lucky day and the issue was gone.

Oh, the pain, the pain. And now, I have a new Jira to close if I upgrade to the next search and hibernate core since I will want to remove all of this fetch annotations from my own code.

sanne.grinovero · **Posted:** Fri May 22, 2009 3:16 am

Quote:

@ManyToOne(fetch=FetchType.Lazy)

right, that's recommended when possible.

Quote:

According to HSEARCH-178 the issue is resolved only quite recently but I am not certain whether it is in the 3.1.0 release and am not going to download and build since I like to suffer with the herd when I don't have time to dig into code myself

No it's not in 3.1.0 it will be in 3.1.1 and 3.2.0; I've updated the issue to make this clear.

Still you should always have a transaction, whatever you're doing, and the fix is only a workaround: it will prevent the nasty cryptical stacktrace and manage to index correctly, but still it will log warnings complaining that you should use transactions. Basically the purpose of the fix is to let people better understand what's wrong.

jlspea01 · **Joined:** Thu May 21, 2009 1:37 pm **Posts:** 2

Quote:

you should always have a transaction

I have transactions around everything that I 'expect' to be transactional (CRUD stuff). In this particular case, a very, very simple query was failing (HQL follows)

Code:

From MyIndexedThing

I would NEVER issue vanilla sql like this in a transaction:

Code:

select * from my_indexed_thing_t;

so I am still hesitant to use the @Transactional notation for a non-search service... (although my 'SearchServiceImpl' uses @Transactional)

I wasn't aware of the suggestion about ManyToOne mappings. I will keep that in mind from now on... maybe even find a way to tell Jalopy about it.

My problem with the whole thing is that the fetching strategy for ManyToOne changed when hibernate search came into the mix.... but now it works and I have learned a few more valuable things about hibernate.

Thank you for your quick response.

sanne.grinovero · **Posted:** Fri May 22, 2009 5:49 am

Quote:

I would NEVER issue vanilla sql like this in a transaction:

All databases I know of will create an implicit transaction if you didn't, avoiding transactions has absolutely no benefit and many dangers, even if you're just reading.
Whatever you're going to do with this "vanilla sql" it needs for sure to be consistent with other operations, or under load you'll get unpredicted behaviour which you can't test and verify as a developer. For example if rendering a web page you want all information to be consistent one with the other, and avoid stuff like " 3 results found: A,B.". Correct design needs to think in terms of unit of work and properly define an application transaction, participating and synchronized with the database transaction.