Prefix searches like the following don't seem to work except for small data sets:
ar*
a*

In our test environment with a very limited amount of data, the above searches work as expected. But when you increase the data set to anything but a very small size (thousands of records) we get the following exception:

Are prefix searches not really allowed with Hibernate Search or Lucene except for test environments? Or am I missing a flag somewhere? I'm guessing by the error that Hibernate Search or Lucene basically queries first for terms matching the prefix expression and then uses those terms in the query for documents? So if a* matches more than 1024 terms, then Lucene is going to throw an exception? Are we expected to raise that limit dramatically?

It's very important that I allow wildcard searching to our users. Hibernate Search and Lucene have to allow this kind of searching, right? Even if it's severely limited, I'd love to check first and tell the user, "sorry our search mechanism isn't capable of search a* because you need to be more specific, try another letter after a" rather than "sorry, there was a problem."

hi mueller,
yes that's a strong limitation of all information retrieval systems - not limited to Lucene only.
As you have guessed it's building a boolean query containing all matching terms from the prefix, the maxClauseCount is set to a default of 1024 and you could set that higher but it won't solve the problem for all cases - and possibly make the system perform badly.

The usual solution which is implemented in these cases is exactly what you suggest:

That's right, so you should catch the TooManyClauses exception and show a message to the user to politely ask to refine it's search terms - you might have noticed this on many online services.

Google does some complex workarounds, you might notice that the simple query "A*" does return the same as "A", while using a star in a more complex text results in somewhat unexpected results... there's no general solution; you can set the clauseLimit very high and verify how bad the performance goes.