-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

View unanswered posts | View active topics


Board index » Projects » Search, Validator, Shards

All times are UTC - 5 hours [ DST ]


Forum locked This topic is locked, you cannot edit posts or make further replies.  Page 1 of 1
  [ 5 posts ] 
  Previous topic | Next topic 
Author Message
koenhandekyn
 Post subject: could not advance using next() while batch reindexing
PostPosted: Tue Mar 11, 2008 6:47 am 
Newbie

Joined: Wed Jul 04, 2007 12:03 pm
Posts: 14
hello

i'm following the instructions from the hibernate search documentation for batch reindixing. i have configured batch_sizes as below. i discovered with trial and error that if hibernate.search.worker.batch_size is larger than hibernate.jdbc.batch_size i get issues with lazy loaded properties.

with the below configuration, i don't run into memory exceptions anymore, and i don't have issues with lazy loaded transacstion anymore but i run into another issue : "could not advance using next()" (tracktrace below)

<property
name="hibernate.jdbc.batch_size"
value="50"
/>

<property
name="hibernate.search.worker.batch_size"
value="25"
/>

i'm running the current version of seam201ga with hibernate 326 and hibernate search 301, and i'm using mysql.

thanx for any feedback.

Code:
  public void reindex() {

    log.info(" REINDEX ");
    final FullTextSession fullTextSession = getFullTextSession();

    fullTextSession.purgeAll(Document.class);
    log.info(" PURGED ALL ");

    fullTextSession.setFlushMode(FlushMode.MANUAL);
    fullTextSession.setCacheMode(CacheMode.IGNORE);
    final StopWatch s = new StopWatch();
    s.start();
    // Scrollable results will avoid loading too many objects in memory
    final ScrollableResults results = fullTextSession.createCriteria(Document.class).scroll(ScrollMode.FORWARD_ONLY);
    int index = 0;
    while (results.next()) {
      index++;
      fullTextSession.index(results.get(0)); // index each element
      if (index % Constants.JDBC_BATCH_SIZE == 0) {
        fullTextSession.clear(); // clear every batchSize since the queue is processed
        log.info("REINDEXED #0 DOCUMENTS duration #1", index, s.getTime());
      }
    }
    results.close();
    log.info("REINDEXED ALL DOCUMENTS duration #0", s.getTime());
    // fullTextSession.close();
  }

Quote:
11:35:29,354 INFO [IndexManager] REINDEXED 3050 DOCUMENTS duration 288850
11:35:34,131 INFO [IndexManager] REINDEXED 3100 DOCUMENTS duration 293627
11:35:37,166 ERROR [MailActivation] Failed to execute folder check, spec=MailActivationSpec(mailServer=10.254.0.41, storeProtocol=pop3, mailFolder=INBOX, pollin
gInterval=60000, messageSelector=null, userName=ontvangersdemo, maxMessages=1, debug=false)
11:35:38,748 INFO [IndexManager] REINDEXED 3150 DOCUMENTS duration 298244
11:35:41,260 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fd9d:d540:47d659d1:d9 invoked while multiple thr
eads active within it.
11:35:41,264 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fd9d:d540:47d659d1:d9 abortin
g with 1 threads active!
11:35:41,270 INFO [ConnectionManager] forcing batcher resource cleanup on transaction completion; forgot to close ScrollableResults/Iterator?
11:35:41,390 WARN [JDBCExceptionReporter] SQL Error: 0, SQLState: null
11:35:41,392 ERROR [JDBCExceptionReporter] The result set is closed.
11:35:41,396 ERROR [SeamPhaseListener] uncaught exception
javax.el.ELException: org.hibernate.exception.GenericJDBCException: could not advance using next()
at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:333)
at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:342)
at org.jboss.el.parser.AstPropertySuffix.invoke(AstPropertySuffix.java:58)
at org.jboss.el.parser.AstValue.invoke(AstValue.java:96)
at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
at org.jboss.seam.core.Expressions$2.invoke(Expressions.java:173)
at org.jboss.seam.navigation.Pages.callAction(Pages.java:636)
at org.jboss.seam.navigation.Pages.preRender(Pages.java:289)
at org.jboss.seam.jsf.SeamPhaseListener.preRenderPage(SeamPhaseListener.java:549)
at org.jboss.seam.jsf.SeamPhaseListener.beforeRenderResponse(SeamPhaseListener.java:460)
at org.jboss.seam.jsf.SeamPhaseListener.beforeServletPhase(SeamPhaseListener.java:144)
at org.jboss.seam.jsf.SeamPhaseListener.beforePhase(SeamPhaseListener.java:114)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:222)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
at org.jboss.seam.debug.hot.HotDeployFilter.doFilter(HotDeployFilter.java:68)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:141)
at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:281)
at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:60)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:856)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:566)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1508)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.hibernate.exception.GenericJDBCException: could not advance using next()
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.impl.ScrollableResultsImpl.next(ScrollableResultsImpl.java:104)
at up.docstore.IndexManager.reindex(IndexManager.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.seam.util.Reflections.invoke(Reflections.java:21)
at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:31)
at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:31)
at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:46)
at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:42)
at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:166)
at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:102)
at up.docstore.IndexManager_$$_javassist_22.reindex(IndexManager_$$_javassist_22.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:329)
... 52 more
Caused by: java.sql.SQLException: The result set is closed.
at org.jboss.resource.adapter.jdbc.WrappedResultSet.checkState(WrappedResultSet.java:1939)
at org.jboss.resource.adapter.jdbc.WrappedResultSet.next(WrappedResultSet.java:1181)
at org.hibernate.impl.ScrollableResultsImpl.next(ScrollableResultsImpl.java:99)
... 75 more
11:35:41,610 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_40] - Abort called on already aborted atomic action -3f57fd9d:d540:47d659d1:d9
Top
 Profile  
 
koenhandekyn
 Post subject: type error
PostPosted: Tue Mar 11, 2008 8:07 am 
Newbie

Joined: Wed Jul 04, 2007 12:03 pm
Posts: 14
it's not lazy loaded transactions of course but lazy loaded properties.
Top
 Profile  
 
sanne.grinovero
 Post subject:
PostPosted: Wed Mar 12, 2008 7:30 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
what happens if you comment/skip "fullTextSession.clear();" ?

of course you'll need it, I just would like to know if it could help.
Top
 Profile  
 
koenhandekyn
 Post subject: fixed
PostPosted: Wed Mar 12, 2008 8:46 am 
Newbie

Joined: Wed Jul 04, 2007 12:03 pm
Posts: 14
i'm discovering some strange behaviour. the below routine works. but if i query slightly differently it runs out of memory. see comments in the code

works fine with criteria API but ...
    out of memory : "select o from " + entityClass.getName() + " o left join fetch o.labels left join fetch o.properties left join fetch o.to inner join fetch o.from";
    out of memory : "select o from " + entityClass.getName() + " o fetch all properties";


kind regards
koen

Code:
// Now re-index with HSearch
      userTx.begin();
      fullTextSession.setFlushMode(FlushMode.MANUAL);
      fullTextSession.setCacheMode(CacheMode.IGNORE);

      // Use HQL instead of Criteria to eager fetch lazy properties


      // final String queryString = "select o from " + entityClass.getName() + " o left join fetch o.labels left join fetch o.properties left join fetch o.to inner join fetch o.from";
      // final String queryString = "select o from " + entityClass.getName() + " o fetch all properties";
      // Scrollable results will avoid loading too many objects in memory
      // final Query query = fullTextSession.createQuery(queryString);
      // query.setLockMode("o", LockMode.NONE);
      // query.setCacheMode(CacheMode.IGNORE);
      // query.setFlushMode(FlushMode.MANUAL);

       final Criteria criteria = fullTextSession.createCriteria(entityClass);
       criteria.setCacheable(false);
       criteria.setCacheMode(CacheMode.IGNORE);
       criteria.setFetchMode("o", FetchMode.EAGER);
       criteria.setLockMode(LockMode.NONE);

      ScrollableResults results = criteria.scroll(ScrollMode.FORWARD_ONLY);
      int index = 0;
      while (results.next()) {
        index++;
        fullTextSession.index(results.get(0)); // index each element
        if (index % Constants.HIBERNATE_SEARCH_BATCH_SIZE == 0) {
          fullTextSession.flush();
          fullTextSession.clear(); // clear every batchSize since the queue is processed
          log.info("#0 updates complete [#1].", index, s.getLapTime());
        }
      }

      userTx.commit();
Top
 Profile  
 
emmanuel
 Post subject:
PostPosted: Thu Mar 13, 2008 2:55 pm 
Hibernate Team
Hibernate Team

Joined: Sun Sep 14, 2003 3:54 am
Posts: 7256
Location: Paris, France
Well, I've not analyzed the problem in depth, but you should keep both batch_size values synchronized. Open a JIRa issue, we probably should raise a warning if it;s not the case.

_________________
Emmanuel

Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  Page 1 of 1
  [ 5 posts ] 

Board index » Projects » Search, Validator, Shards

All times are UTC - 5 hours [ DST ]

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.