I am using Hibernate Search to great effect, but my client has expressed some reservations about storing indexes on the local machine. Seeking possible alternatives, I came across the following:

[url]http://cagataycivici.wordpress.com/2007/03/06/full_text_search_with_hibernate/
[/url]

In particular, this seemed interesting:

"By default lucene indexes are can be saved either on file system or in memory. This can be configured by the hibernate.search.default.directory_provider parameter. Instead of these directory providers, it’s possible to hold the indexes in database using the Compass’s extension called JDBCDirectoryProvider. In addition a couple of event listeners need to be configured."

I assume people have used JDBCDirectoryProvider successfully with Lucene. Can I use it with Hibernate Search? If so, are there any configuration modifications I need to make?

By the way, I would also appreciate insight into any other alternatives for storing indexes in a database.

Thanks.

yes of course you can use it with Hibernate Search;

this is not highly recommended, as a DB is not really suited for this task;
good insight about this issues and about many different approaches are discussed in the book.

What is the concern of your client? this is the best option for performance and simplicity; also the index is not critical, as it can be rebuild from the database if you should loose it.
For clustering configuration much more options are available, such as pushing index changes to a JMS queue and have a master server manage the index for all nodes; all you need to configure this is explained in the reference documentation, and more insight you can find in the book too.

The issue has to do with automatic index updates without the entity manager.

I have not strayed from the default, so when the entity manager persists or updates something, then the index is updated as a result. The client is worried about the case where changes are made to the database directly through a SQL connection. In that case, a manual reindexing may be necessary, and the client worries, rightly or wrongly, that this could be an issue if someone "forgets" to generate the indexes again. The solution someone found is to use Compass because apparently Compass can be configured to generate the indexes anytime a row is updated.

I don't think that's true. After all, the author of the link I posted before says, "Using FullTextSession.index(Object entity) method, it’s possible to rebuild an index of an entity." Why even mention that if things were automatic?

Any insight you can provide into Compass or a way to handle automatic indexing upon row update through SQL is appreciated.

Thanks.

Err, compass is distributing a JDBCDirectory, however no DirectoryProvider.

My understanding is, that DirectoryProviders are an Hibernate Search invention, and not based on Lucene.

JavaDoc on DirectoryProvider:
http://www.hibernate.org/hib_docs/searc ... vider.html

Imho, to make use of the Compass JdbcDirectory one needs to write a JdbcDirectoryProvider ... which I tried and failed. I didn't really do much, my understanding of Lucene is rather limited.

Did any of you have better luck with this?

btw, I dug up this old post because it has a high ranking on google when searching for "JdbcDirectoryProvider".

No special Lucene knowledge is needed: a DirectoryProvider is just a factory, it receives the configuration and is supposed to return a Directory implementation; this could be a JdbcDirectory if you like. start() and stop() methods are defined for the case you need them to start expensive resources or cleanup on shutdown, but are not mandatory you could define empty methods as well.
So the only stuff that needs to be done is to define what you need configurable in a JdbcDirectory, which I never used as personally I don't like the idea.

Feel free to ask more details if you need to implement something new.

I'm trying to migrate from Compass and could not find a JdbcDirectoryProvider implementation through Google. I wrote my own, and it creates JdbcDirectories. But it fails on start-up with


So I'm getting a Directory back. And tables are being created in the DB. But then "segments* file" is not found. B^(
A Google on the "IndexNotFoundException" came up with some hits indicating IndexWriter.commit().close() needs to be called in some cases get the empty files to be created. But that seems further up the abstraction stack than I should be worrying about.

Anyone have a pointer to a working JdbcDirectoryProvider? Or should I suck it up and learn infinispan? I'd really rather not introduce that many changes and new things into the app. B^(

- Jasen.

Hi,
the "no segments*" error definitely means there is something not allright in your Directory implementation; if you want to contribute your code to Hibernate Search I can give a look to it when I'm back from travelling (1 week to go), would be nice as I have no interest in forcing people to use Infinispan.

Still the Infinispan one has some advantages:
a) maintained by me; that's relatively future proof
b) already tested specifically with Hibernate Search
c) easy to enable Infinispan's cool eviction algorithms to cache hot spots in memory, so it performs much better than a "pure" JDBC solution.
d) tested by another gazillion of users

My implementation is extremely trivial and no where near worth contributing.


I completely agree. I'm only trying to use a JdbcDirectory because that's what my app is already using.

I followed this tutorial (http://mprabhat.wordpress.com/2012/08/13/create-lucene-index-in-database-using-jdbcdirectory/) and it worked fine. I also used my JDBCDirectoryProvider to create the Directory and that worked, too. I used the Hibernate Search 3.4.2 provided Lucene.

I'll keep digging.

- Jasen.

Right sorry I realize now that I misread your post: I had understood that you had re-implemented the *Directory* not just the DirectoryProvider.
Your solution makes more sense, and yes I agree I guess that's trivial.

Still that exception is very likely pointing to a bug in the Directory implementation: as you say, if the we needed to do something different "further up on the stack" that wouldn't work either for the other Directory implementations.
I guess that if this one implementation requires some "extra" specific initialization you could open an IndexWriter in the initialize or start methods, then commit and close it.
Two things to be careful about:
- Make sure you close the IndexWriter or the index will stay locked and HS won't be able to use it (obviously it doesn't expect a locked index)
- Make sure you're not resetting an existing index ("reset" as in "wipe out" : verify it won't delete all content from an existing index)

Let me understand your case better: you want to store the index in an RDBMS but don't need clustering, right?
I guess so as otherwise you would have looked at Infinispan first, at least for clustered caching of entities.

Thanks for taking the time to converse with me about this.


I think that's what I need to do. I found a bug in my DirectoryProvider - I was always calling Directory.create(), which was dropping the tables and recreating them. Not good.
Now I check if the tables exist and only create them if necessary. However, ONLY the tables get created, the index "files" do not, which leads to the "segments* not found" error. I read of a similar problem elsewhere - that an API change in Lucene 3.1 necessitates the steps you mention.

What I don't know, though, is how to configure the IndexWriter. I just figured out I need to use one late last night.


Will do.


We are doing simplistic clustering. We use database replication to keep two synced copies of our database (including the Lucene index). Web clients are bounced between the two servers. Index updates only happen on one server via a cron job. Using Infinispan is probably the "right" long-term solution. However, I'm trying to change things slowly - starting with swapping out Compass for Hibernate Search but leaving everything else as untouched as possible. Moving to Hibernate Search gets us a newer version of Lucene, new index mapping syntax, and based on some tests, speeds things up a bit.

- Jasen.

I think I finally have this conquered. I'll probably make a blog posting covering my JDBCDirectoryProvider implementation. But for now here's the highlights:

Compass' JdbcDirectory.create() will create the necessary table, but not populate it with any "files".
To populate the Directory I used:

The above creates "segments.gen" and "segments_1" "files" in the Directory. Now app startup does not fail with the "segments* not found" exception. My concern with the above was using a SimpleAnalyzer. Hibernate Search lets you specify the Analyzer and I was afraid the above would interfere with Hibernate Search's Analyzer detection. This does not seem to be the case. I believe when you create a MassIndexer it wipes out whatever kind of index was there before. (At least that's my theory, and things are working fine for me.)

The next tricky part was on JDBCDirectoryProvider.initialize() deciding whether to create a new Directory (calling .create() which deletes and creates the table) or if the Directory already existed and not to do anything.
There are two parts to this decision 1) Does the table exist. 2) Are there "segments" "files" in the table. Both are required. For this, I'm currently using directory.listAll(). If the table does not exist, listAll() throws an exception. If I get results back, I can look for "segments" or just assume that one of the "files" is a segments file.

Using my JDBCDirectoryProvider along with the JDBCDirectory implementation from the blog mentioned earlier, I'm able store my Hibernate Search indices in the database. Hooray!

The only issue I have is that unlike the file based indices I can't have multiple MassIndexers updating an index at the same time - I get a lock failure. With the file based index I was able to .start() several MassIndexers that all wrote to the same Directory and everything was fine. With the database indices, I have to make sure only one MassIndexer is accessing an index at a time.

Thanks for your help.

@jasenj1
It would be very helpful if you post the whole source code of the JdbcDirectoryProvider.

Thank you

Jasen,

Can you please post your implementation of JdbcDirectoryProvider?

Same as these guys ^ if you could post your solution that would be awesome.