My project uses hibernate inside jboss. Most of the time things work ok, but now it turns out that jboss takes a huge amount of time (5 minutes or so) to process queries returning 2000 or more objects.

Why do I say jboss, rather than hibernate, is taking so long on these queries? Because I've set things up so that I can run the application outside of jboss as well as inside. And the same query that takes 5 minutes inside jboss takes 6 seconds outside of it.

I first wrote to the JBoss Performance Tuning forum, since this really doesn't seem to be a hibernate problem. But I didn't have any luck there. So I wonder if anyone here has had the same problem, and maybe even come up with an answer.

I've gone down a few blind alleys in trying to track this down. At first I thought that running outside jboss is faster simply because then I'm running a single jvm-- thus there's no need for the application to spend time serializing all those objects to my thick client. But that's not the case. The delay is all during the invocation of net.sf.hibernate.Query's list(), and, to repeat, only when that method is invoked inside jboss.

I've tried changing jboss versions (from jboss-3.2.2RC4 to 3.2.2) and I tried not deploying hibernate as an mbean in a sar. But neither of these changes made much difference. I also tried fiddling around with memory allocation and garbage collection paramaters in jboss/bin/run, but nothing I did made any difference. In fact, a heap profiler showed that jboss's heap allocation doesn't change very much while its turgidly slogging away at the query.

We use Hibernate inside JBoss and have never had this kind of problems.

Have your checked if the cache is configured properly, or compared the running SQL queries inside and outside of JBoss.

We experienced the same problem few months ago (with 2.0 version of Hibernate and 3.2.x version of Jboss).
We had a factor of 4 between outside and inside JBoss. We never went deeper into the problem as we decided not to use JBoss. But I guessed it could be some interference between the logging mechanisms of both products (not sure to well remember the problem but Hibernate seems not to find its log config file).

seb

How are you using JBoss to expose the return data to your client?

Are you using the wsdl exposure system and a SOAP protocol?



This indicates you are doing some sort of serialisation. Basically SOAP is dead slow when serialising data sets of a few hundred objects or so.

Table with 7 columns and 1100ish rows took 2 seconds to execute hibernate query and serialise/deserialise the data to a client using SOAP.

Roughly 250-300ms of that was in hibernate doing the query and generating the objects, the rest was SOAPing up and down, 1500ms+

The speed is generally linear so I would expect 2000 rows of similar size to take 3.5-4 seconds.

One BIG problem I found is the log4j system by default spews just about everything into a log file. I'm not entirely sure of the directory ( /server/<deploy_config/log/some_log_file I think ). If you haven't configured log4j.properties anywhere check in here for a massive log file. By default JBoss logs EVERYTHING including the content of all the SOAP traffic which imposes a huge overhead. This had the effect of making 2 second queries take 8-9 seconds before we found the problem.

So, more info would be handy, but I have had similar problems with JBoss appearing to have a massive speed impact on hibernate so I'd check for that log file being huge and if so scratch up a log4j.properties file with TRACE/DEBUG output disabled.

Posting your architecture, what JBoss is doing to the hibernate objects after they are created and the queries you are using plus table/schema info, would be helpful.

Good luck.

Myk.

P.S. I discarded JBoss in the end as 2 seconds for ~1000 records is too slow for my purposes, although it's the SOAPing that costs the time and JBoss does it as efficiently as other proprietory solutions I tried out.

Thanks to wdrai, scesbron, and Myk for taking the time to think about my problem.

I might yet come across performance problems due to logging, but it's starting to look like wdrai hit it on the head when he asked about caching. Hibernate' s initialization printouts are identical inside and outside jboss except for that running outside I see these lines:

Constructed cache with name: com.bioinformatics.appl.samplebank.datamodel.HibSample
Constructed cache with name: com.bioinformatics.appl.samplebank.datamodel.HibAgent
Constructed cache with name: com.bioinformatics.appl.samplebank.datamodel.HibChosenSnomedAbbreviation
etc.

So maybe I haven't really configured the cache correctly inside jboss.

But that's odd, because jboss has some knowledge of the cache. I see lines such as:
"initialized LRUMemoryCache for com.bioinformatics.appl.samplebank.datamodel.HibSample"
and
"Could not instantiate eAttr named 'jcs.region.com.bioinformatics.appl.samplebank.datamodel.HibAgent.elementattributes', using defaults."

running either inside or outside of JBoss.

And I do deploy cache.ccf as part of a sar. The sar has jars inside it, including a jar containing all the hbm.xml files. That's the jar where I put cache.ccf. And when I delete cache.ccf, jboss complains about it:

15:32:08,742 INFO [Configuration] creating JCS cache region: com.bioinformatics.appl.samplebank.datamodel.HibAgent, usage: read-write
15:32:08,820 ERROR [Configuration] Could not configure datastore from input stream
java.lang.NullPointerException
at java.io.Reader.<init>(Reader.java:61)
at java.io.InputStreamReader.<init>(InputStreamReader

So the fact that jboss doesn't print out "Constructed cache with name: ", makes me think that the cache is not configured, but the fact that jboss sseems to be using cache.ccf makes me think that it's caching after all, but just not printing out that line for some perverse reason.


I've seen a reference on the hibernate site about deploying in jboss, but it didn't explicitly mention how to deploy cache.ccf. Any ideas about where I can find such a reference, or even how best to deploy that file?


(In answer to Myk, I'm not working with a webservice. The objects are serialized to an applet or java main application, depending on how we decide to run it. We can tell that serialization is not the issue-- printlns reveal that the delay is all in the call to hibernateQuery.list().)