-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

View unanswered posts | View active topics


Board index » Projects » Search, Validator, Shards

All times are UTC - 5 hours [ DST ]


Forum locked This topic is locked, you cannot edit posts or make further replies.  Page 1 of 1
  [ 9 posts ] 
  Previous topic | Next topic 
Author Message
deltanov
 Post subject: Unable to run lucene indexation
PostPosted: Mon Aug 01, 2016 11:11 am 
Newbie

Joined: Mon Aug 01, 2016 11:04 am
Posts: 7
Our project is using a lucene indexation and we are facing problem running the indexation itself.

Our DB has 23 million records.

We make usage of the following configuration:

hibernate.search.default.batch.merge_factor=2
hibernate.search.default.batch.ram_buffer_size=1024

the JTA and the TIME out are set to 21600. When running the indexation we get the error:

exception clearing maxRows/queryTimeout

I found those parameters on the internet and I wonder what should be the correct settings for our case:

https://docs.jboss.org/hibernate/search ... mance.html


many thanks for your valuable input!
Top
 Profile  
 
deltanov
 Post subject: Re: Unable to run lucene indexation
PostPosted: Tue Aug 02, 2016 3:53 am 
Newbie

Joined: Mon Aug 01, 2016 11:04 am
Posts: 7
I made my settings as follow:

<prop key="hibernate.search.default.batch.merge_factor">15</prop>
<prop key="hibernate.search.default.batch.ram_buffer_size">2048</prop>
<prop key="hibernate.search.default.batch. max_buffered_docs">64</prop>


I am not sure if this makes sense, so I will be very grateful if someone can confirm that they are Ok. My understanding is that lucene is building a huge index file without flushing and I am hitting the TimOut setting (currently 28 800). With these parameters my aim is to force flushing (hibernate.search.default.batch. max_buffered_docs">64) and avoid reaching the time out (I have returned the TimeOut to 3600).

Is my approach correct?
Top
 Profile  
 
sanne.grinovero
 Post subject: Re: Unable to run lucene indexation
PostPosted: Wed Aug 10, 2016 4:18 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Hi,
your parameters look fine, assuming your JVM has enough heap to accommodate for a 2GB on heap buffer. Typically people will use 256 or 512 for ram_buffer_size.

I will need more details about your timeout. Do you have a stacktrace? Which version of Hibernate Search?

_________________
Sanne
http://in.relation.to/

Top
 Profile  
 
deltanov
 Post subject: Re: Unable to run lucene indexation
PostPosted: Fri Sep 09, 2016 4:41 am 
Newbie

Joined: Mon Aug 01, 2016 11:04 am
Posts: 7
Hi Sanne!

First of all many thanks for your reply to my request.

About the questions below we’re using the version 3.4.0.Final.
About the stacktrace - we do not have it.
The timeout as I have pointed is 3600 seconds.

Looking forward your comments.
Top
 Profile  
 
sanne.grinovero
 Post subject: Re: Unable to run lucene indexation
PostPosted: Fri Sep 09, 2016 5:46 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Quote:
About the stacktrace - we do not have it.

I need more details. Any more information coming with the error message?

If we have to guess, I'll have to advise you to update: 3.4.0 is extremely old by now and many issues with the MassIndexer have been resolved already.

_________________
Sanne
http://in.relation.to/

Top
 Profile  
 
deltanov
 Post subject: Re: Unable to run lucene indexation
PostPosted: Fri Sep 09, 2016 8:58 am 
Newbie

Joined: Mon Aug 01, 2016 11:04 am
Posts: 7
This is the las error message we got:

Quote:
[2016-09-08 13:08:54,223] ERROR [eu.europa.ec.estat.egr.is.view] Unexpected error occurred
java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at weblogic.utils.io.ChunkedOutputStream.writeTo(ChunkedOutputStream.java:284)
at weblogic.servlet.internal.ServletOutputStreamImpl.writeHeader(ServletOutputStreamImpl.java:167)
at weblogic.servlet.internal.ResponseHeaders.writeHeaders(ResponseHeaders.java:444)
at weblogic.servlet.internal.ServletResponseImpl.writeHeaders(ServletResponseImpl.java:1288)
at weblogic.servlet.internal.ServletOutputStreamImpl.sendHeaders(ServletOutputStreamImpl.java:281)
at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:424)
at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:298)
at weblogic.servlet.internal.ChunkOutput$2.checkForFlush(ChunkOutput.java:648)
at weblogic.servlet.internal.CharsetChunkOutput.write(CharsetChunkOutput.java:200)
at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:148)
at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:148)
at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.writeStaticResourceResponse(AbstractApplicationServlet.java:1373)
at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.serveStaticResourcesInVAADIN(AbstractApplicationServlet.java:1350)
at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.serveStaticResources(AbstractApplicationServlet.java:1251)
at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.service(AbstractApplicationServlet.java:423)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at eu.europa.ec.estat.egr.is.filter.MaintenanceModeFilter.doFilter(MaintenanceModeFilter.java:35)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at eu.europa.ec.estat.egr.is.filter.MDCUserServletFilter.doFilter(MDCUserServletFilter.java:36)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at eu.europa.ec.estat.egr.is.filter.WelcomePageFilter.doFilter(WelcomePageFilter.java:39)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
[2016-09-08 13:43:15,209] bbb WARN [org.hibernate.engine.loading.LoadContexts] fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@1ffe51c8<rs=weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl@792ec>
[2016-09-08 13:43:15,210] bbb WARN [org.hibernate.engine.loading.CollectionLoadContext] On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [2] entries
[2016-09-08 13:43:45,288] WARN [org.springframework.integration.channel.MessagePublishingErrorHandler] Error message was not delivered.
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 2832
at java.util.jar.Attributes.read(Attributes.java:377)
at java.util.jar.Manifest.read(Manifest.java:180)
at java.util.jar.Manifest.<init>(Manifest.java:50)
at java.util.jar.JarFile.getManifestFromReference(JarFile.java:168)
at java.util.jar.JarFile.getManifest(JarFile.java:149)
at weblogic.utils.classloaders.JarSource.getManifest(JarSource.java:21)
at weblogic.utils.classloaders.GenericClassLoader.defineClass(GenericClassLoader.java:319)
at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:302)
at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:270)
at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:64)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:179)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:43)
at org.springframework.integration.util.MessagingMethodInvokerHelper.findClosestMatch(MessagingMethodInvokerHelper.java:387)
at org.springframework.integration.util.MessagingMethodInvokerHelper.findHandlerMethodsForParameters(MessagingMethodInvokerHelper.java:376)
at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:220)
at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:127)
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:73)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:67)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.BroadcastingDispatcher.invokeHandler(BroadcastingDispatcher.java:121)
at org.springframework.integration.dispatcher.BroadcastingDispatcher.dispatch(BroadcastingDispatcher.java:112)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.MessagePublishingErrorHandler.handleError(MessagePublishingErrorHandler.java:83)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:231)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
[2016-09-08 13:43:47,077] ERROR [org.springframework.integration.channel.
Top
 Profile  
 
deltanov
 Post subject: Re: Unable to run lucene indexation
PostPosted: Fri Sep 09, 2016 8:59 am 
Newbie

Joined: Mon Aug 01, 2016 11:04 am
Posts: 7
This is before that:

Quote:
ERROR [org.hibernate.search.batchindexing.EntityConsumerLuceneworkProducer] error during batch indexing:
org.hibernate.TransactionException: JTA commit failed:
at org.hibernate.transaction.JTATransaction.commit(JTATransaction.java:161)
at org.hibernate.search.batchindexing.EntityConsumerLuceneworkProducer.run(EntityConsumerLuceneworkProducer.java:100)
at org.hibernate.search.batchindexing.OptionallyWrapInJTATransaction.run(OptionallyWrapInJTATransaction.java:107)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: weblogic.transaction.RollbackException: Transaction timed out after 3599 seconds
BEA1-2597EC5D80C5FA5158C3
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1884)
at weblogic.transaction.internal.TransactionImpl.checkIfCommitPossible(TransactionImpl.java:1774)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:292)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:268)
at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:308)
at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:302)
at org.hibernate.transaction.JTATransaction.commit(JTATransaction.java:154)
... 4 more
Caused by: weblogic.transaction.internal.TimedOutException: Transaction timed out after 3599 seconds
BEA1-2597EC5D80C5FA5158C3
Top
 Profile  
 
sanne.grinovero
 Post subject: Re: Unable to run lucene indexation
PostPosted: Tue Sep 20, 2016 12:23 pm 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
so it looks like you're running with container managed transactions, specifically managed by WebLogic.
But you're also running a quite old version of Hibernate Search: 3.4.0.Final. That version is unable to control the timeout of the container, so it will eventually timeout when indexing large datasets.

I would suggest to use a more recent version, as e.g. version 5.5.x is able to keep the container managed transaction alive for as long as it needs - although we tested this only on WildFly, I don't have a WebLogic instance to test it but it's using a standard API to accomplish this so I expect it to work equally well.

As an alternative to upgrading the dependencies you might want to reconfigure your transaction timeout to be large enough for the indexing to be able to complete; this would affect all your transactions though.

In the first stack trace you posted there also is an OutOfMemory exception. You probably want to monitor the JVM for memory usage: when it is running low on memory the indexing process will take significantly longer.

_________________
Sanne
http://in.relation.to/

Top
 Profile  
 
deltanov
 Post subject: Re: Unable to run lucene indexation
PostPosted: Tue Sep 20, 2016 3:51 pm 
Newbie

Joined: Mon Aug 01, 2016 11:04 am
Posts: 7
Hi Sanne,

Many thanks indeed for the reply!

Do you know if we can use version 4.X? Because we have WL 10.3.6 and it does not support version 5.X.
Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  Page 1 of 1
  [ 9 posts ] 

Board index » Projects » Search, Validator, Shards

All times are UTC - 5 hours [ DST ]

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.