Hi Guys,

I believe im experiencing deadlock with my mass indexer. Recently we have changed our mass indexer config to see if we can increase our throughput time in finishing indexing quicker. This has had a side effect in which our app hangs.

We currently have 8 classes which are indexed. (So 8 indexes in total). Our mass indexer config is as follows

batch.indexer.batch.size=100
batch.indexer.fetch.threads=5
batch.indexer.load.objects=2

When configured like this, the indexer never gets started.......i see no log outputs. A thread dump reveals the following to me
I have
40 threads of type "Hibernate Search: collectionsloader-4" - Of which 23 are looking for a DB connection (5 * 8 = 40)
16 threads of type "Hibernate Search: entityloader-2" Of which 12 are looking for a DB connection. (8 * 2 = 16)
8 threads of type "Hibernate Search: identifierloader-1" - All 8 are awaiting a connection

All the above threads are stuck waiting for "awaitAvailable" to retrieve a connection.



So i have 43 threads all waiting for a DB connection!!

My c3po config is standard enough



When i seen this my first reaction is deadlock.......when i dont use such a high "fetch" and "load" config i dont see this issue. Have you guys a test case for running the mass indexer when the number of threads looking for connections exceeds the pool size??

Also came across this in my searching for a solution
https://jira.atlassian.com/browse/BAM-5670 different product i know but the symptoms are similar to what im experiencing.

Any thoughts on this??

LL

Hi,

hard to say where the problem is. Here are some tips which maybe help you to narrow down the problem.
Have you turned on debug/trace level logging? If so, is there anything in the logs? Which database are you using and are there any limits on how many connections it accepts? Have you tried switching to dbcp for example to see whether there is a problem with cp3o?

--Hardy

No we don't have one; if you could create one that would be awesome as we have HSEARCH-598 : I'd be glad to look into the issue but have not had time to create a test for it.

The math you did is correct:


So that's not going to work as unfortunately with the current pipeline design it needs all components of the pipeline "alive" to be able to start. My problem in changing the pipeline design is that this one is the only one which is really fast ;)

So when you're in this situation you have two possible workarounds:
1) use the other indexing method using the scrollable results
2) Instead of using the MassIndexer on all objects, use it on each entity once:


becomes

So you won't have 43 threads started at the same time, but "only" 7 per type, which is below the 20 threshold.

Unfortunately there's no way to predict from the datasource what is the maximum amount of connection that I'm allowed to ask for, so I don't think we can log a warning about this but we might want to introduce a monitoring thread to check for deadlocks - but even so this would likely need to abort the process.

Thanks for the info Sanne.


Option 1 would possibly be too slow.

Option 2 is a better option. Need to see how this performs.

Interesting read https://hibernate.onjira.com/browse/HSEARCH-598

Whats not clear to me from the thread is does HS leak threads over time?? So if i in theory kept my pool size greater than the need that HS needs, would this over time deadlock??

So example, lets say i configure my mass indexer like so
batch.indexer.batch.size=100
batch.indexer.fetch.threads=2
batch.indexer.load.objects=2

If have 8 indexes and if the math is right
collectionsloader - 2 * 8 = 16
entityloader - 2 * 8 = 16
identifierloader - 1 * 8 = 8

So a possible need of 40 connections at any given time. Now if i set my pool size to 45 with the above config and no other threads use the connection pool, would i run into deadlock over a period time?? (So thats with indexing everything in parallel) Ive cheated with my test environment, the above config works for me but im just worried that over a period of time i will get deadlock. I will do option 2 but for now, would i run into this deadlock issue over a period of time with the above config??


Cheers,
JB

No I'm not aware of it leaking threads, nor have I seen anything which would make me think that's possible.