-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 
Author Message
 Post subject: HS : MassIndexer hot spot error
PostPosted: Mon Aug 02, 2010 9:34 am 
Regular
Regular

Joined: Thu Oct 08, 2009 10:34 am
Posts: 55
Hi Guys,

I have changed my apps functionality from using the old way of indexing to the new way using MassIndexer functionality. I have a hit a bit of a problem with using the new mass indexing approach.

I get a hotspot error.

I set up the server with the new settings in our test environment, set the settings of the mass indexer to the following

batch.indexer.batch.size=50
batch.indexer.fetch.threads=2
batch.indexer.load.objects=1

This took 10 hours to complete versus 4 hours the old way. No errors were generated.


I then changed the configuration of the mass indexer to the following to see if i could improve performance.

batch.indexer.batch.size=50
batch.indexer.fetch.threads=4
batch.indexer.load.objects=2

30% of the overall index was indexed and then a hotspot error was generated.

It looks like the following

Quote:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002b09dfa1ec00, pid=11168, tid=1140840768
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x501c00]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x00002aab000cd000): JavaThread "Lucene Merge Thread #8" daemon [_thread_in_vm, id=5850, stack(0x0000000043efd000,0x0000000043ffe000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000002aaaef9859

Registers:
RAX=0x0000000000000005, RBX=0x0000002aaaef9851, RCX=0x00002aaae58a6100, RDX=0x00002b09dfe2eb70
RSP=0x0000000043ffc2b0, RBP=0x0000000043ffc2c0, RSI=0x00002aab000cd000, RDI=0x0000002aaaef9851
R8 =0x0000000051763900, R9 =0x00002b09dfe41f28, R10=0x00002b09dfb6ab20, R11=0x00002aaae58a6183
R12=0x00002aaae58a6181, R13=0x00002b09dfe48470, R14=0x00002aaaaf4a02a8, R15=0x00002aaae58a61f1
RIP=0x00002b09dfa1ec00, EFL=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x0000000043ffc2b0)
0x0000000043ffc2b0: 00002aaaaf4a02a8 00002aab000cd000
0x0000000043ffc2c0: 0000000043ffc350 00002b09dfbb7419
0x0000000043ffc2d0: 0000000043ffc380 00002b09df73e365
0x0000000043ffc2e0: 00002aaa00001000 0000000051763928
0x0000000043ffc2f0: 0000000000000005 00002aab000cd000
0x0000000043ffc300: 00002aaaeea46901 00002aaaab471fb4
0x0000000043ffc310: 00002aaaaf587c00 00002aaaaf591c88
0x0000000043ffc320: 000000000000000a 0000000043ffc470
0x0000000043ffc330: 0000000051763928 00002aab000cd000
0x0000000043ffc340: 00002aaaaf4a02a8 00002aab000cd000
0x0000000043ffc350: 0000000043ffc380 00002b09dfbb52bb
0x0000000043ffc360: 00002aaaaf4ffc60 0000000051763928
0x0000000043ffc370: 00002aab000cd000 0000000043ffc470
0x0000000043ffc380: 0000000043ffc3b0 00002b09dfbb51b4
0x0000000043ffc390: 0000000051098ba0 00002aab000cd000
0x0000000043ffc3a0: 00002aaaaf559f58 0000000043ffc470
0x0000000043ffc3b0: 0000000043ffc410 00002b09dfb6aba9
0x0000000043ffc3c0: 0000000000000001 0000000051763928
0x0000000043ffc3d0: 00002aab000cd000 0000000000000000
0x0000000043ffc3e0: 00002aab000cd000 00002b09df6fda28
0x0000000043ffc3f0: 0000000000000010 0000000000000000
0x0000000043ffc400: 00002aab000cd000 00002aaaaf4c0e38
0x0000000043ffc410: 0000000000000000 00002aaaab193f3e
0x0000000043ffc420: 0000000000000000 00002aaaab7a9628
0x0000000043ffc430: 00002aaaaf4a02a8 00002aaaaf4c0e38
0x0000000043ffc440: 0000000000000000 00002aaaaf559f58
0x0000000043ffc450: 00002aaaaf4a02a8 00002aaaab1c122d
0x0000000043ffc460: 000000000000000c 0000000043ffc4a0
0x0000000043ffc470: 0000000000000003 00002aaaab160f4e
0x0000000043ffc480: 00002aaaaf4a02a8 00002aaaab446840
0x0000000043ffc490: 00002aaac7027200 0000000000000001
0x0000000043ffc4a0: 00002aab000cd000 00002aab00081170

Instructions: (pc=0x00002b09dfa1ec00)
0x00002b09dfa1ebf0: 48 8b 53 08 eb 93 48 8b 53 08 e9 9a fe ff ff 90
0x00002b09dfa1ec00: 48 8b 57 08 e9 59 fe ff ff 66 66 66 90 66 66 66

Stack: [0x0000000043efd000,0x0000000043ffe000], sp=0x0000000043ffc2b0, free space=3fc0000000000000018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x501c00]
V [libjvm.so+0x69a419]
V [libjvm.so+0x6982bb]
V [libjvm.so+0x6981b4]
V [libjvm.so+0x64dba9]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v ~RuntimeStub::_complete_monitor_locking_Java
J org.apache.lucene.index.FormatPostingsTermsWriter.addTerm([CI)Lorg/apache/lucene/index/FormatPostingsDocsConsumer;
J org.apache.lucene.index.SegmentMerger.appendPostings(Lorg/apache/lucene/index/FormatPostingsTermsConsumer;[Lorg/apache/lucene/index/SegmentMergeInfo;I)I
j org.apache.lucene.index.SegmentMerger.mergeTermInfos(Lorg/apache/lucene/index/FormatPostingsFieldsConsumer;)V+383
j org.apache.lucene.index.SegmentMerger.mergeTerms()V+62
j org.apache.lucene.index.SegmentMerger.merge(Z)I+14
j org.apache.lucene.index.IndexWriter.mergeMiddle(Lorg/apache/lucene/index/MergePolicy$OneMerge;)I+423
j org.apache.lucene.index.IndexWriter.merge(Lorg/apache/lucene/index/MergePolicy$OneMerge;)V+70
j org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(Lorg/apache/lucene/index/MergePolicy$OneMerge;)V+5
j org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run()V+34
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
=>0x00002aab000cd000 JavaThread "Lucene Merge Thread #8" daemon [_thread_in_vm, id=5850, stack(0x0000000043efd000,0x0000000043ffe000)]
0x0000000053403000 JavaThread "Hibernate Search: indexwriter-2" [_thread_blocked, id=6988, stack(0x0000000046927000,0x0000000046a28000)]
0x000000005200f000 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6938, stack(0x000000004702e000,0x000000004712f000)]
0x0000000052b23800 JavaThread "Hibernate Search: indexwriter-2" [_thread_blocked, id=6381, stack(0x0000000046b29000,0x0000000046c2a000)]
0x00000000516df000 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6375, stack(0x0000000046826000,0x0000000046927000)]
0x0000000052f9e800 JavaThread "Hibernate Search: indexwriter-2" [_thread_blocked, id=6244, stack(0x0000000046725000,0x0000000046826000)]
0x00000000516db000 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6233, stack(0x0000000046a28000,0x0000000046b29000)]
0x0000000051806000 JavaThread "Hibernate Search: indexwriter-2" [_thread_blocked, id=6089, stack(0x0000000044705000,0x0000000044806000)]
0x00000000534ae000 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6077, stack(0x0000000044806000,0x0000000044907000)]
0x0000000052ecc000 JavaThread "Hibernate Search: indexwriter-2" [_thread_blocked, id=6076, stack(0x0000000044907000,0x0000000044a08000)]
0x0000000051720000 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6075, stack(0x0000000043bfa000,0x0000000043cfb000)]
0x000000005213a800 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6074, stack(0x0000000044200000,0x0000000044301000)]
0x00000000524a9000 JavaThread "Hibernate Search: indexwriter-2" [_thread_blocked, id=6073, stack(0x0000000044e0c000,0x0000000044f0d000)]
0x0000000051383800 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6072, stack(0x0000000044c0a000,0x0000000044d0b000)]
0x0000000053799800 JavaThread "Hibernate Search: indexwriter-2" [_thread_blocked, id=6071, stack(0x0000000044b09000,0x0000000044c0a000)]
0x0000000051983800 JavaThread "Hibernate Search: indexwriter-1" [_thread_blocked, id=6070, stack(0x0000000044a08000,0x0000000044b09000)]
0x0000000051a90800 JavaThread "Hibernate Search: identifierloader-1" [_thread_blocked, id=6063, stack(0x0000000047432000,0x0000000047533000)]
0x0000000052012800 JavaThread "Hibernate Search: entityloader-2" [_thread_blocked, id=6061, stack(0x0000000047230000,0x0000000047331000)]
0x0000000051f1c800 JavaThread "Hibernate Search: entityloader-1" [_thread_blocked, id=6034, stack(0x0000000045917000,0x0000000045a18000)]
0x0000000051f1a000 JavaThread "Hibernate Search: collectionsloader-4" [_thread_blocked, id=6033, stack(0x0000000045816000,0x0000000045917000)]
0x0000000051f18000 JavaThread "Hibernate Search: collectionsloader-3" [_thread_blocked, id=6021, stack(0x0000000045715000,0x0000000045816000)]
0x0000000051f16000 JavaThread "Hibernate Search: collectionsloader-2" [_thread_blocked, id=6019, stack(0x0000000045614000,0x0000000045715000)]
0x0000000051f13800 JavaThread "Hibernate Search: collectionsloader-1" [_thread_blocked, id=6018, stack(0x0000000045513000,0x0000000045614000)]
0x0000000051763000 JavaThread "Hibernate Search: BatchIndexingWorkspace-3" [_thread_blocked, id=5993, stack(0x0000000043cfb000,0x0000000043dfc000)]
0x0000000051575800 JavaThread "pool-8-thread-1" [_thread_blocked, id=6346, stack(0x00000000439f8000,0x0000000043af9000)]
0x00000000514c8000 JavaThread "pool-7-thread-1" [_thread_blocked, id=6345, stack(0x00000000438f7000,0x00000000439f8000)]
0x0000000052e9f800 JavaThread "pool-6-thread-1" [_thread_blocked, id=6344, stack(0x00000000437f6000,0x00000000438f7000)]
0x000000005147e000 JavaThread "pool-5-thread-1" [_thread_blocked, id=6343, stack(0x00000000436f5000,0x00000000437f6000)]
0x000000005147d000 JavaThread "pool-4-thread-1" [_thread_blocked, id=6342, stack(0x00000000435f4000,0x00000000436f5000)]
0x0000000052401800 JavaThread "pool-3-thread-1" [_thread_blocked, id=6341, stack(0x00000000434f3000,0x00000000435f4000)]
0x0000000051e4f000 JavaThread "pool-2-thread-1" [_thread_blocked, id=6340, stack(0x00000000433f2000,0x00000000434f3000)]
0x0000000051710800 JavaThread "pool-1-thread-1" [_thread_blocked, id=6339, stack(0x00000000432f1000,0x00000000433f2000)]
0x0000000051785000 JavaThread "http-9080-Acceptor-0" daemon [_thread_in_native, id=11498, stack(0x00000000431f0000,0x00000000432f1000)]
0x0000000051716000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=11497, stack(0x00000000430ef000,0x00000000431f0000)]
0x00000000523c7800 JavaThread "quartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=11484, stack(0x0000000042fee000,0x00000000430ef000)]
0x0000000052f93000 JavaThread "quartzScheduler_Worker-10" [_thread_blocked, id=11383, stack(0x0000000042eed000,0x0000000042fee000)]
0x0000000052827800 JavaThread "quartzScheduler_Worker-9" [_thread_blocked, id=11382, stack(0x0000000042dec000,0x0000000042eed000)]
0x0000000051bb5800 JavaThread "quartzScheduler_Worker-8" [_thread_blocked, id=11381, stack(0x0000000042ceb000,0x0000000042dec000)]
0x00000000529e4800 JavaThread "quartzScheduler_Worker-7" [_thread_blocked, id=11380, stack(0x0000000042bea000,0x0000000042ceb000)]
0x0000000052607000 JavaThread "quartzScheduler_Worker-6" [_thread_blocked, id=11379, stack(0x0000000042ae9000,0x0000000042bea000)]
0x0000000052b29000 JavaThread "quartzScheduler_Worker-5" [_thread_blocked, id=11375, stack(0x00000000429e8000,0x0000000042ae9000)]
0x0000000053563000 JavaThread "quartzScheduler_Worker-4" [_thread_blocked, id=11373, stack(0x00000000428e7000,0x00000000429e8000)]
0x00000000539c5800 JavaThread "quartzScheduler_Worker-3" [_thread_blocked, id=11370, stack(0x00000000427e6000,0x00000000428e7000)]
0x000000005297e800 JavaThread "quartzScheduler_Worker-2" [_thread_blocked, id=11358, stack(0x00000000426e5000,0x00000000427e6000)]
0x0000000052219800 JavaThread "quartzScheduler_Worker-1" [_thread_blocked, id=11357, stack(0x0000000041253000,0x0000000041354000)]
0x00000000516d3800 JavaThread "Timer-8" daemon [_thread_blocked, id=11204, stack(0x00000000425e4000,0x00000000426e5000)]
0x00000000527d9800 JavaThread "Timer-7" daemon [_thread_blocked, id=11203, stack(0x00000000424e3000,0x00000000425e4000)]
0x00000000515a2800 JavaThread "Timer-6" daemon [_thread_blocked, id=11202, stack(0x00000000423e2000,0x00000000424e3000)]
0x00000000515a1000 JavaThread "Timer-5" daemon [_thread_blocked, id=11201, stack(0x00000000422e1000,0x00000000423e2000)]
0x0000000051f39800 JavaThread "Timer-4" daemon [_thread_blocked, id=11200, stack(0x00000000421e0000,0x00000000422e1000)]
0x0000000051f79000 JavaThread "Timer-3" daemon [_thread_blocked, id=11199, stack(0x0000000041152000,0x0000000041253000)]
0x0000000051f33800 JavaThread "Timer-2" daemon [_thread_blocked, id=11198, stack(0x0000000041051000,0x0000000041152000)]
0x0000000051b18000 JavaThread "Timer-1" daemon [_thread_blocked, id=11197, stack(0x0000000040f50000,0x0000000041051000)]
0x0000000051260000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=11192, stack(0x000000004190a000,0x0000000041a0b000)]
0x00000000524f9800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=11191, stack(0x0000000041809000,0x000000004190a000)]
0x00000000524f8800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=11190, stack(0x0000000041708000,0x0000000041809000)]
0x000000005156d800 JavaThread "Timer-0" daemon [_thread_blocked, id=11189, stack(0x00000000414e5000,0x00000000415e6000)]
0x00000000517a3800 JavaThread "GC Daemon" daemon [_thread_blocked, id=11181, stack(0x0000000040b98000,0x0000000040c99000)]
0x000000005126d000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=11179, stack(0x00000000403ef000,0x00000000404f0000)]
0x0000000051142000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=11178, stack(0x00000000420df000,0x00000000421e0000)]
0x000000005113f800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=11177, stack(0x0000000041fde000,0x00000000420df000)]
0x000000005113b000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11176, stack(0x0000000041d89000,0x0000000041e8a000)]
0x0000000051139000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11175, stack(0x0000000041a57000,0x0000000041b58000)]
0x0000000051136800 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=11174, stack(0x00000000409ca000,0x0000000040acb000)]
0x0000000051112000 JavaThread "Finalizer" daemon [_thread_blocked, id=11173, stack(0x0000000041edd000,0x0000000041fde000)]
0x0000000051110000 JavaThread "Reference Handler" daemon [_thread_blocked, id=11172, stack(0x0000000041c24000,0x0000000041d25000)]
0x000000005109a000 JavaThread "main" [_thread_in_native, id=11169, stack(0x000000004022e000,0x000000004032f000)]

Other Threads:
0x000000005110b800 VMThread [stack: 0x00000000408c9000,0x00000000409ca000] [id=11171]
0x0000000051299800 WatcherThread [stack: 0x0000000041607000,0x0000000041708000] [id=11180]

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x0000000051097da0] Threads_lock - owner thread: 0x000000005110b800
[0x0000000051098320] Heap_lock - owner thread: 0x0000000051f16000

Heap
def new generation total 19136K, used 17660K [0x00002aaaae1f0000, 0x00002aaaaf6b0000, 0x00002aaaaf6b0000)
eden space 17024K, 100% used [0x00002aaaae1f0000, 0x00002aaaaf290000, 0x00002aaaaf290000)
from space 2112K, 30% used [0x00002aaaaf290000, 0x00002aaaaf32f1b0, 0x00002aaaaf4a0000)
to space 2112K, 0% used [0x00002aaaaf4a0000, 0x00002aaaaf4a0000, 0x00002aaaaf6b0000)
concurrent mark-sweep generation total 1002752K, used 325344K [0x00002aaaaf6b0000, 0x00002aaaec9f0000, 0x00002aaaec9f0000)
concurrent-mark-sweep perm gen total 83372K, used 49984K [0x00002aaaec9f0000, 0x00002aaaf1b5b000, 0x00002aaaf1df0000)

Dynamic libraries:
40000000-40009000 r-xp 00000000 fd:01 7308972 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/bin/java
40108000-4010a000 rwxp 00008000 fd:01 7308972 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/bin/java
4022e000-40231000 ---p 4022e000 00:00 0
40231000-4032f000 rwxp 40231000 00:00 0
403ef000-403f2000 ---p 403ef000 00:00 0
403f2000-404f0000 rwxp 403f2000 00:00 0
408c9000-408ca000 ---p 408c9000 00:00 0
408ca000-409ca000 rwxp 408ca000 00:00 0
409ca000-409cd000 ---p 409ca000 00:00 0
409cd000-40acb000 rwxp 409cd000 00:00 0
40b98000-40b9b000 ---p 40b98000 00:00 0
40b9b000-40c99000 rwxp 40b9b000 00:00 0
40f50000-40f53000 ---p 40f50000 00:00 0
40f53000-41051000 rwxp 40f53000 00:00 0
41051000-41054000 ---p 41051000 00:00 0
41054000-41152000 rwxp 41054000 00:00 0
41152000-41155000 ---p 41152000 00:00 0
41155000-41253000 rwxp 41155000 00:00 0
41253000-41256000 ---p 41253000 00:00 0
41256000-41354000 rwxp 41256000 00:00 0
413e4000-413e5000 ---p 413e4000 00:00 0
413e5000-414e5000 rwxp 413e5000 00:00 0
414e5000-414e8000 ---p 414e5000 00:00 0
414e8000-415e6000 rwxp 414e8000 00:00 0
41607000-41608000 ---p 41607000 00:00 0
41608000-41708000 rwxp 41608000 00:00 0
41708000-4170b000 ---p 41708000 00:00 0
4170b000-41809000 rwxp 4170b000 00:00 0
41809000-4180c000 ---p 41809000 00:00 0
4180c000-4190a000 rwxp 4180c000 00:00 0
4190a000-4190d000 ---p 4190a000 00:00 0
4190d000-41a0b000 rwxp 4190d000 00:00 0
41a57000-41a5a000 ---p 41a57000 00:00 0
41a5a000-41b58000 rwxp 41a5a000 00:00 0
41c24000-41c27000 ---p 41c24000 00:00 0
41c27000-41d25000 rwxp 41c27000 00:00 0
41d89000-41d8c000 ---p 41d89000 00:00 0
41d8c000-41e8a000 rwxp 41d8c000 00:00 0
41edd000-41ee0000 ---p 41edd000 00:00 0
41ee0000-41fde000 rwxp 41ee0000 00:00 0
41fde000-41fe1000 ---p 41fde000 00:00 0
41fe1000-420df000 rwxp 41fe1000 00:00 0
420df000-420e2000 ---p 420df000 00:00 0
420e2000-421e0000 rwxp 420e2000 00:00 0
421e0000-421e3000 ---p 421e0000 00:00 0
421e3000-422e1000 rwxp 421e3000 00:00 0
422e1000-422e4000 ---p 422e1000 00:00 0
422e4000-423e2000 rwxp 422e4000 00:00 0
423e2000-423e5000 ---p 423e2000 00:00 0
423e5000-424e3000 rwxp 423e5000 00:00 0
424e3000-424e6000 ---p 424e3000 00:00 0
424e6000-425e4000 rwxp 424e6000 00:00 0
425e4000-425e7000 ---p 425e4000 00:00 0
425e7000-426e5000 rwxp 425e7000 00:00 0
426e5000-426e8000 ---p 426e5000 00:00 0
426e8000-427e6000 rwxp 426e8000 00:00 0
427e6000-427e9000 ---p 427e6000 00:00 0
427e9000-428e7000 rwxp 427e9000 00:00 0
428e7000-428ea000 ---p 428e7000 00:00 0
428ea000-429e8000 rwxp 428ea000 00:00 0
429e8000-429eb000 ---p 429e8000 00:00 0
429eb000-42ae9000 rwxp 429eb000 00:00 0
42ae9000-42aec000 ---p 42ae9000 00:00 0
42aec000-42bea000 rwxp 42aec000 00:00 0
42bea000-42bed000 ---p 42bea000 00:00 0
42bed000-42ceb000 rwxp 42bed000 00:00 0
42ceb000-42cee000 ---p 42ceb000 00:00 0
42cee000-42dec000 rwxp 42cee000 00:00 0
42dec000-42def000 ---p 42dec000 00:00 0
42def000-42eed000 rwxp 42def000 00:00 0
42eed000-42ef0000 ---p 42eed000 00:00 0
42ef0000-42fee000 rwxp 42ef0000 00:00 0
42fee000-42ff1000 ---p 42fee000 00:00 0
42ff1000-430ef000 rwxp 42ff1000 00:00 0
430ef000-430f2000 ---p 430ef000 00:00 0
430f2000-431f0000 rwxp 430f2000 00:00 0
431f0000-431f3000 ---p 431f0000 00:00 0
431f3000-432f1000 rwxp 431f3000 00:00 0
432f1000-432f4000 ---p 432f1000 00:00 0
432f4000-433f2000 rwxp 432f4000 00:00 0
433f2000-433f5000 ---p 433f2000 00:00 0
433f5000-434f3000 rwxp 433f5000 00:00 0
434f3000-434f6000 ---p 434f3000 00:00 0
434f6000-435f4000 rwxp 434f6000 00:00 0
435f4000-435f7000 ---p 435f4000 00:00 0
435f7000-436f5000 rwxp 435f7000 00:00 0
436f5000-436f8000 ---p 436f5000 00:00 0
436f8000-437f6000 rwxp 436f8000 00:00 0
437f6000-437f9000 ---p 437f6000 00:00 0
437f9000-438f7000 rwxp 437f9000 00:00 0
438f7000-438fa000 ---p 438f7000 00:00 0
438fa000-439f8000 rwxp 438fa000 00:00 0
439f8000-439fb000 ---p 439f8000 00:00 0
439fb000-43af9000 rwxp 439fb000 00:00 0
43af9000-43afc000 ---p 43af9000 00:00 0
43afc000-43bfa000 rwxp 43afc000 00:00 0
43bfa000-43bfd000 ---p 43bfa000 00:00 0
43bfd000-43cfb000 rwxp 43bfd000 00:00 0
43cfb000-43cfe000 ---p 43cfb000 00:00 0
43cfe000-43dfc000 rwxp 43cfe000 00:00 0
43dfc000-43dff000 ---p 43dfc000 00:00 0
43dff000-43efd000 rwxp 43dff000 00:00 0
43efd000-43f00000 ---p 43efd000 00:00 0
43f00000-43ffe000 rwxp 43f00000 00:00 0
43ffe000-44001000 ---p 43ffe000 00:00 0
44001000-440ff000 rwxp 44001000 00:00 0
440ff000-44102000 ---p 440ff000 00:00 0
44102000-44200000 rwxp 44102000 00:00 0
44200000-44203000 ---p 44200000 00:00 0
44203000-44301000 rwxp 44203000 00:00 0
44301000-44304000 ---p 44301000 00:00 0
44304000-44402000 rwxp 44304000 00:00 0
44402000-44405000 ---p 44402000 00:00 0
44405000-44503000 rwxp 44405000 00:00 0
44503000-44506000 ---p 44503000 00:00 0
44506000-44604000 rwxp 44506000 00:00 0
44604000-44607000 ---p 44604000 00:00 0
44607000-44705000 rwxp 44607000 00:00 0
44705000-44708000 ---p 44705000 00:00 0
44708000-44806000 rwxp 44708000 00:00 0
44806000-44809000 ---p 44806000 00:00 0
44809000-44907000 rwxp 44809000 00:00 0
44907000-4490a000 ---p 44907000 00:00 0
4490a000-44a08000 rwxp 4490a000 00:00 0
44a08000-44a0b000 ---p 44a08000 00:00 0
44a0b000-44b09000 rwxp 44a0b000 00:00 0
44b09000-44b0c000 ---p 44b09000 00:00 0
44b0c000-44c0a000 rwxp 44b0c000 00:00 0
44c0a000-44c0d000 ---p 44c0a000 00:00 0
44c0d000-44d0b000 rwxp 44c0d000 00:00 0
44d0b000-44d0e000 ---p 44d0b000 00:00 0
44d0e000-44e0c000 rwxp 44d0e000 00:00 0
44e0c000-44e0f000 ---p 44e0c000 00:00 0
44e0f000-44f0d000 rwxp 44e0f000 00:00 0
44f0d000-44f10000 ---p 44f0d000 00:00 0
44f10000-4500e000 rwxp 44f10000 00:00 0
4500e000-45011000 ---p 4500e000 00:00 0
45011000-4510f000 rwxp 45011000 00:00 0
4510f000-45112000 ---p 4510f000 00:00 0
45112000-45210000 rwxp 45112000 00:00 0
45210000-45213000 ---p 45210000 00:00 0
45213000-45311000 rwxp 45213000 00:00 0
45311000-45314000 ---p 45311000 00:00 0
45314000-45412000 rwxp 45314000 00:00 0
45412000-45415000 ---p 45412000 00:00 0
45415000-45513000 rwxp 45415000 00:00 0
45513000-45516000 ---p 45513000 00:00 0
45516000-45614000 rwxp 45516000 00:00 0
45614000-45617000 ---p 45614000 00:00 0
45617000-45715000 rwxp 45617000 00:00 0
45715000-45718000 ---p 45715000 00:00 0
45718000-45816000 rwxp 45718000 00:00 0
45816000-45819000 ---p 45816000 00:00 0
45819000-45917000 rwxp 45819000 00:00 0
45917000-4591a000 ---p 45917000 00:00 0
4591a000-45a18000 rwxp 4591a000 00:00 0
45a18000-45a1b000 ---p 45a18000 00:00 0
45a1b000-45b19000 rwxp 45a1b000 00:00 0
45b19000-45b1c000 ---p 45b19000 00:00 0
45b1c000-45c1a000 rwxp 45b1c000 00:00 0
45c1a000-45c1d000 ---p 45c1a000 00:00 0
45c1d000-45d1b000 rwxp 45c1d000 00:00 0
45d1b000-45d1e000 ---p 45d1b000 00:00 0
45d1e000-45e1c000 rwxp 45d1e000 00:00 0
45e1c000-45e1f000 ---p 45e1c000 00:00 0
45e1f000-45f1d000 rwxp 45e1f000 00:00 0
45f1d000-45f20000 ---p 45f1d000 00:00 0
45f20000-4601e000 rwxp 45f20000 00:00 0
4601e000-46021000 ---p 4601e000 00:00 0
46021000-4611f000 rwxp 46021000 00:00 0
4611f000-46122000 ---p 4611f000 00:00 0
46122000-46220000 rwxp 46122000 00:00 0
46220000-46223000 ---p 46220000 00:00 0
46223000-46321000 rwxp 46223000 00:00 0
46321000-46324000 ---p 46321000 00:00 0
46324000-46422000 rwxp 46324000 00:00 0
46422000-46425000 ---p 46422000 00:00 0
46425000-46523000 rwxp 46425000 00:00 0
46523000-46526000 ---p 46523000 00:00 0
46526000-46624000 rwxp 46526000 00:00 0
46624000-46627000 ---p 46624000 00:00 0
46627000-46725000 rwxp 46627000 00:00 0
46725000-46728000 ---p 46725000 00:00 0
46728000-46826000 rwxp 46728000 00:00 0
46826000-46829000 ---p 46826000 00:00 0
46829000-46927000 rwxp 46829000 00:00 0
46927000-4692a000 ---p 46927000 00:00 0
4692a000-46a28000 rwxp 4692a000 00:00 0
46a28000-46a2b000 ---p 46a28000 00:00 0
46a2b000-46b29000 rwxp 46a2b000 00:00 0
46b29000-46b2c000 ---p 46b29000 00:00 0
46b2c000-46c2a000 rwxp 46b2c000 00:00 0
46c2a000-46c2d000 ---p 46c2a000 00:00 0
46c2d000-46d2b000 rwxp 46c2d000 00:00 0
46d2b000-46d2e000 ---p 46d2b000 00:00 0
46d2e000-46e2c000 rwxp 46d2e000 00:00 0
46e2c000-46e2f000 ---p 46e2c000 00:00 0
46e2f000-46f2d000 rwxp 46e2f000 00:00 0
46f2d000-46f30000 ---p 46f2d000 00:00 0
46f30000-4702e000 rwxp 46f30000 00:00 0
4702e000-47031000 ---p 4702e000 00:00 0
47031000-4712f000 rwxp 47031000 00:00 0
4712f000-47132000 ---p 4712f000 00:00 0
47132000-47230000 rwxp 47132000 00:00 0
47230000-47233000 ---p 47230000 00:00 0
47233000-47331000 rwxp 47233000 00:00 0
47331000-47334000 ---p 47331000 00:00 0
47334000-47432000 rwxp 47334000 00:00 0
47432000-47435000 ---p 47432000 00:00 0
47435000-47533000 rwxp 47435000 00:00 0
47735000-47738000 ---p 47735000 00:00 0
47738000-47836000 rwxp 47738000 00:00 0
47836000-47839000 ---p 47836000 00:00 0
47839000-47937000 rwxp 47839000 00:00 0
47937000-4793a000 ---p 47937000 00:00 0
4793a000-47a38000 rwxp 4793a000 00:00 0
47a38000-47a3b000 ---p 47a38000 00:00 0
47a3b000-47b39000 rwxp 47a3b000 00:00 0
51091000-54c16000 rwxp 51091000 00:00 0 [heap]
3fe1a00000-3fe1a1c000 r-xp 00000000 fd:00 884946 /lib64/ld-2.5.so
3fe1c1b000-3fe1c1c000 r-xp 0001b000 fd:00 884946 /lib64/ld-2.5.so
3fe1c1c000-3fe1c1d000 rwxp 0001c000 fd:00 884946 /lib64/ld-2.5.so
3fe1e00000-3fe1f4d000 r-xp 00000000 fd:00 885033 /lib64/libc-2.5.so
3fe1f4d000-3fe214d000 ---p 0014d000 fd:00 885033 /lib64/libc-2.5.so
3fe214d000-3fe2151000 r-xp 0014d000 fd:00 885033 /lib64/libc-2.5.so
3fe2151000-3fe2152000 rwxp 00151000 fd:00 885033 /lib64/libc-2.5.so
3fe2152000-3fe2157000 rwxp 3fe2152000 00:00 0
3fe2200000-3fe2202000 r-xp 00000000 fd:00 885034 /lib64/libdl-2.5.so
3fe2202000-3fe2402000 ---p 00002000 fd:00 885034 /lib64/libdl-2.5.so
3fe2402000-3fe2403000 r-xp 00002000 fd:00 885034 /lib64/libdl-2.5.so
3fe2403000-3fe2404000 rwxp 00003000 fd:00 885034 /lib64/libdl-2.5.so
3fe2600000-3fe2616000 r-xp 00000000 fd:00 885038 /lib64/libpthread-2.5.so
3fe2616000-3fe2815000 ---p 00016000 fd:00 885038 /lib64/libpthread-2.5.so
3fe2815000-3fe2816000 r-xp 00015000 fd:00 885038 /lib64/libpthread-2.5.so
3fe2816000-3fe2817000 rwxp 00016000 fd:00 885038 /lib64/libpthread-2.5.so
3fe2817000-3fe281b000 rwxp 3fe2817000 00:00 0
3fe2a00000-3fe2a82000 r-xp 00000000 fd:00 885039 /lib64/libm-2.5.so
3fe2a82000-3fe2c81000 ---p 00082000 fd:00 885039 /lib64/libm-2.5.so
3fe2c81000-3fe2c82000 r-xp 00081000 fd:00 885039 /lib64/libm-2.5.so
3fe2c82000-3fe2c83000 rwxp 00082000 fd:00 885039 /lib64/libm-2.5.so
3fe2e00000-3fe2e07000 r-xp 00000000 fd:00 885040 /lib64/librt-2.5.so
3fe2e07000-3fe3007000 ---p 00007000 fd:00 885040 /lib64/librt-2.5.so
3fe3007000-3fe3008000 r-xp 00007000 fd:00 885040 /lib64/librt-2.5.so
3fe3008000-3fe3009000 rwxp 00008000 fd:00 885040 /lib64/librt-2.5.so
3fe5600000-3fe5615000 r-xp 00000000 fd:00 884927 /lib64/libnsl-2.5.so
3fe5615000-3fe5814000 ---p 00015000 fd:00 884927 /lib64/libnsl-2.5.so
3fe5814000-3fe5815000 r-xp 00014000 fd:00 884927 /lib64/libnsl-2.5.so
3fe5815000-3fe5816000 rwxp 00015000 fd:00 884927 /lib64/libnsl-2.5.so
3fe5816000-3fe5818000 rwxp 3fe5816000 00:00 0
3fe8600000-3fe8611000 r-xp 00000000 fd:00 885050 /lib64/libresolv-2.5.so
3fe8611000-3fe8811000 ---p 00011000 fd:00 885050 /lib64/libresolv-2.5.so
3fe8811000-3fe8812000 r-xp 00011000 fd:00 885050 /lib64/libresolv-2.5.so
3fe8812000-3fe8813000 rwxp 00012000 fd:00 885050 /lib64/libresolv-2.5.so
3fe8813000-3fe8815000 rwxp 3fe8813000 00:00 0
2aaaaaaab000-2aaaaaaad000 r-xs 00004000 fd:01 8814704 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/bin/bootstrap.jar
2aaaaaaad000-2aaaaaaaf000 r-xs 00001000 fd:01 8814707 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/bin/commons-daemon.jar
2aaaaaaaf000-2aaaaaab0000 r-xs 00006000 fd:01 8814714 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/bin/tomcat-juli.jar
2aaaaaab0000-2aaaaaab1000 rwxp 2aaaaaab0000 00:00 0
2aaaaaab1000-2aaaaaab4000 r-xs 00b4a000 fd:01 5996651 /app/fulltextsearch/fts-V20100723/fulltextsearch.war
2aaaaaab6000-2aaaaaac3000 r-xp 00000000 fd:01 7309611 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libverify.so
2aaaaaac3000-2aaaaabc2000 ---p 0000d000 fd:01 7309611 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libverify.so
2aaaaabc2000-2aaaaabc5000 rwxp 0000c000 fd:01 7309611 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libverify.so
2aaaaabc5000-2aaaaabee000 r-xp 00000000 fd:01 7309629 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libjava.so
2aaaaabee000-2aaaaaced000 ---p 00029000 fd:01 7309629 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libjava.so
2aaaaaced000-2aaaaacf4000 rwxp 00028000 fd:01 7309629 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libjava.so
2aaaaacf4000-2aaaaacf5000 ---p 2aaaaacf4000 00:00 0
2aaaaacf5000-2aaaaacf6000 rwxp 2aaaaacf5000 00:00 0
2aaaaacf6000-2aaaaacfd000 r-xp 00000000 fd:01 7309618 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/native_threads/libhpi.so
2aaaaacfd000-2aaaaadfe000 ---p 00007000 fd:01 7309618 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/native_threads/libhpi.so
2aaaaadfe000-2aaaaae00000 rwxp 00008000 fd:01 7309618 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/native_threads/libhpi.so
2aaaaae00000-2aaaaae01000 rwxp 2aaaaae00000 00:00 0
2aaaaae01000-2aaaaae09000 rwxs 00000000 fd:03 393218 /tmp/hsperfdata_fulltextsearch/11168
2aaaaae0b000-2aaaaae15000 r-xp 00000000 fd:00 884761 /lib64/libnss_files-2.5.so
2aaaaae15000-2aaaab014000 ---p 0000a000 fd:00 884761 /lib64/libnss_files-2.5.so
2aaaab014000-2aaaab015000 r-xp 00009000 fd:00 884761 /lib64/libnss_files-2.5.so
2aaaab015000-2aaaab016000 rwxp 0000a000 fd:00 884761 /lib64/libnss_files-2.5.so
2aaaab016000-2aaaab024000 r-xp 00000000 fd:01 7309616 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libzip.so
2aaaab024000-2aaaab126000 ---p 0000e000 fd:01 7309616 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libzip.so
2aaaab126000-2aaaab129000 rwxp 00010000 fd:01 7309616 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libzip.so
2aaaab129000-2aaaab83a000 rwxp 2aaaab129000 00:00 0
2aaaab83a000-2aaaae12a000 rwxp 2aaaab83a000 00:00 0
2aaaae12a000-2aaaae147000 rwxp 2aaaae12a000 00:00 0
2aaaae147000-2aaaae1ea000 rwxp 2aaaae147000 00:00 0
2aaaae1f0000-2aaaf1b5b000 rwxp 2aaaae1f0000 00:00 0
2aaaf1b5b000-2aaaf1df0000 rwxp 2aaaf1b5b000 00:00 0
2aaaf1df0000-2aaaf200d000 rwxp 2aaaf1df0000 00:00 0
2aaaf200d000-2aaaf200e000 rwxp 2aaaf200d000 00:00 0
2aaaf200e000-2aaaf2222000 rwxp 2aaaf200e000 00:00 0
2aaaf2222000-2aaaf2224000 rwxp 2aaaf2222000 00:00 0
2aaaf2224000-2aaaf5b2d000 rwxp 2aaaf2224000 00:00 0
2aaaf5b2d000-2aaaf5cc3000 r-xs 02fd0000 fd:01 7309711 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/rt.jar
2aaaf5cc3000-2aaaf5cf6000 rwxp 2aaaf5cc3000 00:00 0
2aaaf5cf6000-2aaaf92c6000 r-xp 00000000 fd:00 364580 /usr/lib/locale/locale-archive
2aaaf92c6000-2aaaf92d9000 r-xp 00000000 fd:01 7309605 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libnet.so
2aaaf92d9000-2aaaf93da000 ---p 00013000 fd:01 7309605 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libnet.so
2aaaf93da000-2aaaf93dd000 rwxp 00014000 fd:01 7309605 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libnet.so
2aaaf93dd000-2aaaf93e3000 r-xp 00000000 fd:01 7309652 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libmanagement.so
2aaaf93e3000-2aaaf94e2000 ---p 00006000 fd:01 7309652 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libmanagement.so
2aaaf94e2000-2aaaf94e4000 rwxp 00005000 fd:01 7309652 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libmanagement.so
2aaaf94e4000-2aaaf94e6000 r-xs 0000c000 fd:01 8814719 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/catalina-ant.jar
2aaaf94e6000-2aaaf94eb000 r-xs 00035000 fd:01 8814721 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/catalina-tribes.jar
2aaaf94eb000-2aaaf94ee000 r-xs 00018000 fd:01 8814724 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/jasper-el.jar
2aaaf94ee000-2aaaf94f2000 r-xs 00035000 fd:01 8814730 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/tomcat-dbcp.jar
2aaaf94f2000-2aaaf9501000 r-xs 0016d000 fd:00 592651 /usr/lib64/oracle/ojdbc14.jar
2aaaf9501000-2aaaf9503000 r-xs 0000b000 fd:01 8814732 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/tomcat-i18n-fr.jar
2aaaf9503000-2aaaf9505000 r-xs 0000c000 fd:01 8814733 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/tomcat-i18n-ja.jar
2aaaf9505000-2aaaf9506000 r-xs 00007000 fd:01 8814723 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/el-api.jar
2aaaf9506000-2aaaf9509000 r-xs 0000f000 fd:01 8814731 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/tomcat-i18n-es.jar
2aaaf9509000-2aaaf9511000 r-xs 00079000 fd:01 8814726 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/jasper.jar
2aaaf9511000-2aaaf9513000 r-xs 00014000 fd:01 8814728 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/servlet-api.jar
2aaaf9513000-2aaaf951a000 r-xs 00053000 fd:00 694723 /usr/share/java/log4j-1.2.14.jar
2aaaf951a000-2aaaf9529000 r-xs 00113000 fd:01 8814722 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/catalina.jar
2aaaf9529000-2aaaf952a000 r-xs 00003000 fd:01 8814718 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/annotations-api.jar
2aaaf952a000-2aaaf952c000 r-xs 00011000 fd:01 8814727 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/jsp-api.jar
2aaaf952c000-2aaaf9538000 r-xs 0014c000 fd:01 8814725 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/jasper-jdt.jar
2aaaf9538000-2aaaf9543000 r-xs 000af000 fd:01 8814729 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/tomcat-coyote.jar
2aaaf9543000-2aaaf9546000 r-xs 0001d000 fd:01 8814720 /app/fulltextsearch/ecet/sw/apache-tomcat-6.0.26/lib/catalina-ha.jar
2aaaf9546000-2aaaf9549000 r-xs 000cb000 fd:01 7309686 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/ext/localedata.jar
2aaaf963b000-2aaaf9642000 r-xs 00111000 fd:01 7309687 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/resources.jar
2aaaf9642000-2aaaf9662000 r-xp 00000000 fd:00 592650 /usr/lib64/oracle/libocijdbc10.so
2aaaf9662000-2aaaf9761000 ---p 00020000 fd:00 592650 /usr/lib64/oracle/libocijdbc10.so
2aaaf9761000-2aaaf9763000 rwxp 0001f000 fd:00 592650 /usr/lib64/oracle/libocijdbc10.so
2aaaf9763000-2aaafa96e000 r-xp 00000000 fd:00 592646 /usr/lib64/oracle/libclntsh.so.10.1
2aaafa96e000-2aaafaa6d000 ---p 0120b000 fd:00 592646 /usr/lib64/oracle/libclntsh.so.10.1
2aaafaa6d000-2aaafab11000 rwxp 0120a000 fd:00 592646 /usr/lib64/oracle/libclntsh.so.10.1
2aaafab11000-2aaafab2d000 rwxp 2aaafab11000 00:00 0
2aaafab2d000-2aaafae1c000 r-xp 00000000 fd:00 592647 /usr/lib64/oracle/libnnz10.so
2aaafae1c000-2aaafaf1b000 ---p 002ef000 fd:00 592647 /usr/lib64/oracle/libnnz10.so
2aaafaf1b000-2aaafafcc000 rwxp 002ee000 fd:00 592647 /usr/lib64/oracle/libnnz10.so
2aaafafcc000-2aaafafcd000 rwxp 2aaafafcc000 00:00 0
2aaafafcd000-2aaaff516000 r-xp 00000000 fd:00 592649 /usr/lib64/oracle/libociei.so
2aaaff516000-2aaaff615000 ---p 04549000 fd:00 592649 /usr/lib64/oracle/libociei.so
2aaaff615000-2aaaff617000 rwxp 04548000 fd:00 592649 /usr/lib64/oracle/libociei.so
2aaaff621000-2aaaff625000 r-xp 00000000 fd:00 884759 /lib64/libnss_dns-2.5.so
2aaaff625000-2aaaff824000 ---p 00004000 fd:00 884759 /lib64/libnss_dns-2.5.so
2aaaff824000-2aaaff825000 r-xp 00003000 fd:00 884759 /lib64/libnss_dns-2.5.so
2aaaff825000-2aaaff826000 rwxp 00004000 fd:00 884759 /lib64/libnss_dns-2.5.so
2aaaff826000-2aaaff82d000 r-xp 00000000 fd:01 7309606 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libnio.so
2aaaff82d000-2aaaff92c000 ---p 00007000 fd:01 7309606 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libnio.so
2aaaff92c000-2aaaff92e000 rwxp 00006000 fd:01 7309606 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/libnio.so
2aab00000000-2aab00597000 rwxp 2aab00000000 00:00 0
2aab00597000-2aab04000000 ---p 2aab00597000 00:00 0
2aab04000000-2aab044e3000 rwxp 2aab04000000 00:00 0
2aab044e3000-2aab08000000 ---p 2aab044e3000 00:00 0
2b09df202000-2b09df203000 rwxp 2b09df202000 00:00 0
2b09df20e000-2b09df20f000 rwxp 2b09df20e000 00:00 0
2b09df20f000-2b09df210000 r-xp 00000000 fd:00 884738 /lib64/libkeepalive.so
2b09df210000-2b09df40f000 ---p 00001000 fd:00 884738 /lib64/libkeepalive.so
2b09df40f000-2b09df410000 rwxp 00000000 fd:00 884738 /lib64/libkeepalive.so
2b09df410000-2b09df417000 r-xp 00000000 fd:01 7309601 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/jli/libjli.so
2b09df417000-2b09df518000 ---p 00007000 fd:01 7309601 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/jli/libjli.so
2b09df518000-2b09df51a000 rwxp 00008000 fd:01 7309601 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/jli/libjli.so
2b09df51a000-2b09df51d000 rwxp 2b09df51a000 00:00 0
2b09df51d000-2b09dfcd2000 r-xp 00000000 fd:01 7309636 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/server/libjvm.so
2b09dfcd2000-2b09dfdd1000 ---p 007b5000 fd:01 7309636 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/server/libjvm.so
2b09dfdd1000-2b09dff5c000 rwxp 007b4000 fd:01 7309636 /app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/server/libjvm.so
2b09dff5c000-2b09dff95000 rwxp 2b09dff5c000 00:00 0
7fff5208f000-7fff520a4000 rwxp 7ffffffea000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]

VM Arguments:
jvm_args: -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DFULL_TEXT_SEARCH_CONF_DIR=/app/fulltextsearch/conf/conf -Xloggc:/app/fulltextsearch/ecet/log/gc-20100729-122520.log -verbose:gc -XX:-TraceClassUnloading -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -Xmx1000M -Xms1000M -XX:CMSInitiatingOccupancyFraction=85 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.endorsed.dirs=/app/fulltextsearch/tomcatFTS/endorsed -Dcatalina.base=/app/fulltextsearch/tomcatFTS -Dcatalina.home=/app/fulltextsearch/tomcatFTS -Djava.io.tmpdir=/app/fulltextsearch/tomcatFTS/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/app/fulltextsearch/ecet/sw/java
PATH=/usr/lib64/oracle/bin:/app/fulltextsearch/ecet/sw/java/bin:/usr/lib64/oracle/bin:/app/fulltextsearch/ecet/sw/java/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/app/fulltextsearch/ecet/bin:/app/fulltextsearch/ecet/bin
LD_LIBRARY_PATH=/app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64/server:/app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/lib/amd64:/app/fulltextsearch/ecet/sw/jdk1.6.0_20/jre/../lib/amd64:/usr/lib64/oracle
SHELL=/bin/bash
DISPLAY=localhost:11.0

Signal Handlers:
SIGSEGV: [libjvm.so+0x70ffd0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x70ffd0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x5d8cf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: SIG_IGN, sa_mask[0]=0x3e7bf217, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x5d8cf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x5d8cf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x5db510], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x5db260], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x5db260], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x5db260], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


--------------- S Y S T E M ---------------

OS:Red Hat Enterprise Linux Server release 5.4 (Tikanga)

uname:Linux 2.6.18-164.2.1.el5 #1 SMP Mon Sep 21 04:37:42 EDT 2009 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 16384, NOFILE 1024, AS infinity
load average:13.68 13.75 13.59

CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 23 stepping 8, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 2059624k(10076k free), swap 2097144k(1151320k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (16.3-b01) for linux-amd64 JRE (1.6.0_20-b02), built on Apr 12 2010 13:57:11 by "java_re" with gcc 3.2.2 (SuSE Linux)

time: Fri Jul 30 03:42:22 2010
elapsed time: 55021 seconds



Nearest i could get in relation to the problem was
https://issues.apache.org/jira/browse/L ... l-tabpanel

but i dont see a resolution to the problem in that thread and the ticket is marked as resolved but when i look here
http://wiki.apache.org/lucene-java/SunJavaBugs

I see there is no resolution for it.

I do see a comment in the ticket that could be a factor in this

Quote:
<Sun can't ignore a HotSpot compiler bug, can they?
They are safely ignoring CMS collector bugs on 64bit archs.


Will stop using the CMS collector and see if it still occurs but doubt it, never had this problem the old indexing way on the same environment . Will keep you posted on it.

Have you guys ever seen this happen with HS before?? Any comments on it??

Cheers guys,
LL


Top
 Profile  
 
 Post subject: Re: HS : MassIndexer hot spot error
PostPosted: Tue Aug 03, 2010 6:42 am 
Hibernate Team
Hibernate Team

Joined: Fri Oct 05, 2007 4:47 pm
Posts: 2536
Location: Third rock from the Sun
Quote:
Have you guys ever seen this happen with HS before??

No, never.
Quote:
Any comments on it??

OMG
Quote:
Will keep you posted on it.

thank you, this doesn't look like very good.

Quote:
This took 10 hours to complete versus 4 hours the old way.

that's disappointing, the MassIndexer should provide a noticeable improvement - you really need to change settings.
I'd try a larger eden size (remove CMSInitiatingOccupancyFraction option) and use a smaller batch size (start at 10, but really depends on the complexity of queries used to load your entities).

_________________
Sanne
http://in.relation.to/


Top
 Profile  
 
 Post subject: Re: HS : MassIndexer hot spot error
PostPosted: Tue Aug 03, 2010 8:07 am 
Regular
Regular

Joined: Thu Oct 08, 2009 10:34 am
Posts: 55
Did a run with the same configuration and no problems last night with.

Indexing time was 8hrs 42mins so improvement on my first run but still no where near as good as the old indexing way.

Quote:
you really need to change settings.
I'd try a larger eden size (remove CMSInitiatingOccupancyFraction option) and use a smaller batch size (start at 10, but really depends on the complexity of queries used to load your entities).


Duely noted, currently playing with the settings. Hopefully i can find the "sweet spot". Tracking my settings and results internally.


Top
 Profile  
 
 Post subject: Re: HS : MassIndexer hot spot error
PostPosted: Tue Aug 03, 2010 8:53 am 
Contributor
Contributor

Joined: Tue Aug 03, 2010 8:45 am
Posts: 1
Some Hotspot version have quite some bugs that we are not even able to reproduce. I had a similar bug with Lucene 3.0.2 a few weeks ago on a server running 1.6.9_19 or _20, not sure. Version _21 is running constantly without crash/restart since weeks now.

We are using CMS collector, too. If you can reproduce the bug with the latest JVM version, please open also an issue @ Lucene Java (https://issues.apache.org/jira/browse/LUCENE), so we can eventually track this down. If the bug only appears in older versions of the JVM we see them as "solved".


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 4 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.