I have more than 1200000 records on a table (MySQL)
The code i use for index all this records is this:
Code:
// Extract Session from EntityManager
Long numMsgBBDD = countNews();
long numVueltas = numMsgBBDD / 5000L;
Session lclSession = (Session)entityManager.getDelegate();
System.out.println("Starting ReIndexing Routine");
// Create a FullTextSession, so we can setup a scrollable cursor
// to keep the memory footprint down while we iterate through the
// list of objects.
FullTextSession ftSession = Search.createFullTextSession(lclSession);
ftSession.setFlushMode(FlushMode.MANUAL);
int index = 0;
ftSession.setCacheMode(CacheMode.IGNORE);
Date start = new Date();
ScrollableResults results = null;
System.out.println("Getting Scrollable Cursor for Change Requests...");
int i = 0;
while(i<=numVueltas){
//Scrollable results will avoid loading too many objects in memory
results = ftSession.createCriteria( PolMensajes.class ).setFirstResult(i*5000+1).setMaxResults(5000).setFetchSize(125).scroll( ScrollMode.FORWARD_ONLY );
System.out.println("Cursor retrieved... Start Indexing...Vuelta " +i + " de "+ numVueltas);
while( results.next() ) {
index++;
ftSession.index( results.get(0) ); //index each element
// The "25" matches the value established in the persistence.xml used to
// control batch sizes for Hibernate Search.
/**
<property name="hibernate.search.worker.batch_size" value="25"/>
*
*/
if (index % 125 == 0) {
// clear the session contents to keep memory footprint down.
ftSession.clear();
System.out.println("Records Indexed: " + index);
}
}
i++;
results.close();
}
Only i can index over 30.000 - 50.000 records and i'm getting this error
Code:
11:15:23,287 INFO [STDOUT] Records Indexed: 30000
11:15:24,066 INFO [STDOUT] Cursor retrieved... Start Indexing...Vuelta 6 de 204
11:15:24,803 INFO [STDOUT] Records Indexed: 30125
11:15:25,250 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id 7f000101:85f1:4a5af7fb:32 invoked while multiple threads active within it.
11:15:25,250 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action 7f000101:85f1:4a5af7fb:32 aborting with 1 threads active!
11:15:25,251 INFO [ConnectionManager] forcing batcher resource cleanup on transaction completion; forgot to close ScrollableResults/Iterator?
11:15:25,360 WARN [JDBCExceptionReporter] SQL Error: 0, SQLState: null
11:15:25,360 ERROR [JDBCExceptionReporter] Transaction TransactionImple < ac, BasicAction: 7f000101:85f1:4a5af7fb:32 status: ActionStatus.ABORTED > cannot proceed STATUS_ROLLEDBACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: 7f000101:85f1:4a5af7fb:32 status: ActionStatus.ABORTED > cannot proceed STATUS_ROLLEDBACK)
11:15:25,480 ERROR [application] org.hibernate.exception.GenericJDBCException: could not initialize a collection: [org.poliedrico.awwmail.entity.PolMensajes.polCfgUsrMsg#120616]
javax.faces.el.EvaluationException: org.hibernate.exception.GenericJDBCException: could not initialize a collection: [org.poliedrico.awwmail.entity.PolMensajes.polCfgUsrMsg#120616]
at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:102)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
at javax.faces.component.UICommand.broadcast(UICommand.java:387)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:475)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:755)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at com.icesoft.faces.webapp.http.core.JsfLifecycleExecutor.apply(JsfLifecycleExecutor.java:17)
at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.renderCycle(ReceiveSendUpdates.java:94)
at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.service(ReceiveSendUpdates.java:54)
at com.icesoft.faces.webapp.http.core.RequestVerifier.service(RequestVerifier.java:26)
at com.icesoft.faces.webapp.http.common.standard.PathDispatcherServer.service(PathDispatcherServer.java:24)
at com.icesoft.faces.webapp.http.servlet.MainSessionBoundServlet.service(MainSessionBoundServlet.java:160)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$1.service(SessionDispatcher.java:38)
at com.icesoft.faces.webapp.http.servlet.ThreadBlockingAdaptingServlet.service(ThreadBlockingAdaptingServlet.java:19)
at com.icesoft.faces.webapp.http.servlet.EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:63)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:55)
at com.icesoft.faces.webapp.http.servlet.SessionVerifier.service(SessionVerifier.java:19)
at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:23)
at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:91)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:56)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.HotDeployFilter.doFilter(HotDeployFilter.java:53)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:595)
As you can see in the code, i close always the scrollableResults!
How i can index all this records? Thanks in advance!!!
Juan.