I am trying to confirm that my index is updating properly but this does not appear to be the case.

When I make a change to a Indexed Field of an entity the change is propagated to the database via Hibernate Core but the index does not appear to update (ie I can't search by the updated field just the old field).

Here is an example of the problem: I search for a product by its name (we will call it "product 1.0") and it is returned in the results. I go into the product edit page (click on the name in the search results) and change the name (we will change it to "product 2.0"). After the make the change I save the change and it is propagated to the database (confirmed via DBVisualizer). I then try to search for the product by the new name (search for "product 2.0" or in lucene query form +productName: product +productName: 2.0) and nothing is returned but if I search by the old name ("product 1.0") then the product is returned showing the new name ("product 2.0").

I am not getting any visible errors in the log that indicate a problem (I was getting a lock obtain timed out error but have since fixed that) so I am not quite sure how to diagnose the problem. The indexing strategy I am using is an shared Infinispan index using the JDBC CacheStore for persistance. We are using a jGroups queue to ensure that the lock is not shared between multiple machines but for right now there is only one machine in the cluster (my local machine) and it is set to "master".

Any help would be appreciated!

Hi, can you do two things:

1) Try the same using a simple FSDirectory, removing both the jgroups queue and Infinispan stacks will help isolate the problem.

2) Can you post the Infinispan configuration you're using: the caches should not use a transaction manager.

Hi Sanne,

Here is my infinispan config file:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infin ... ig-4.2.xsd"
xmlns="urn:infinispan:config:4.2">

<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->

<global>

<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<globalJmxStatistics
enabled="true"
cacheManagerName="HibernateSearch"
allowDuplicateDomains="true"/>

<!-- If the transport is omitted, there is no way to create distributed or clustered
caches. There is no added cost to defining a transport but not creating a cache that uses one,
since the transport is created and initialized lazily. -->
<transport
clusterName="HibernateSearch-Infinispan-cluster"
distributedSyncTimeout="50000">
<!-- Note that the JGroups transport uses sensible defaults if no configuration
property is defined. See the JGroupsTransport javadocs for more flags -->
</transport>

<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
Hibernate Search takes care to stop the CacheManager so registering is not needed -->
<shutdown
hookBehavior="DONT_REGISTER"/>

</global>

<!-- *************************** -->
<!-- Default "template" settings -->
<!-- *************************** -->

<default>

<locking
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="500"
useLockStriping="false"/>

<lazyDeserialization
enabled="false"/>

<!-- Invocation batching is required for use with the Lucene Directory -->
<invocationBatching
enabled="true"/>

<!-- This element specifies that the cache is clustered. modes supported: distribution
(d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
Lucene indexes, but make sure you have enough memory to store the index in your heap.
Also distribution scales much better than replication on high number of nodes in the cluster. -->
<clustering
mode="replication">

<!-- Prefer loading all data at startup than later -->
<stateRetrieval
timeout="60000"
logFlushTimeout="30000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="true"/>

<!-- Network calls are synchronous by default -->
<sync
replTimeout="20000"/>
</clustering>

<jmxStatistics
enabled="true"/>

<eviction
maxEntries="-1"
strategy="NONE"/>

<expiration
maxIdle="-1"/>

</default>

<!-- ******************************************************************************* -->
<!-- Individually configured "named" caches. -->
<!-- -->
<!-- While default configuration happens to be fine with similar settings across the -->
<!-- three caches, they should generally be different in a production environment. -->
<!-- -->
<!-- Current settings could easily lead to OutOfMemory exception as a CacheStore -->
<!-- should be enabled, and maybe distribution is desired. -->
<!-- ******************************************************************************* -->

<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<namedCache name="LuceneIndexesMetadata">

<clustering mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="30000"/>
<sync replTimeout="25000"/>
</clustering>
<loaders preload="true" shared="true">
<loader class="org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore" fetchPersistentState="true">
<properties>
<property name="driverClass" value="com.mysql.jdbc.Driver"/>
<property name="connectionUrl" value="@DB.URL@/@DB.PREVIEW.NAME@"/>
<property name="userName" value="@DB.USERNAME@"/>
<property name="password" value="@DB.PASSWORD@"/>
<property name="dataColumnType" value="LONGBLOB"/>

<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.SimpleConnectionFactory"/>
<property name="bucketTableNamePrefix" value="lucene"/>
<property name="cacheName" value="FullTextIndexes"/>
<property name="idColumnType" value="VARCHAR(256)"/>
<property name="idColumnName" value="idCol"/>
<property name="dataColumnName" value="dataCol"/>

<property name="timestampColumnName" value="timestampCol"/>
<property name="timestampColumnType" value="BIGINT"/>

</properties>
</loader>
</loaders>
</namedCache>

<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<namedCache name="LuceneIndexesData">

<clustering mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="30000"/>
<sync replTimeout="25000"/>
</clustering>
<loaders preload="true" shared="true">
<loader class="org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore" fetchPersistentState="true">
<properties>
<property name="driverClass" value="com.mysql.jdbc.Driver"/>
<property name="connectionUrl" value="@DB.URL@/@DB.PREVIEW.NAME@"/>
<property name="userName" value="@DB.USERNAME@"/>
<property name="password" value="@DB.PASSWORD@"/>
<property name="dataColumnType" value="LONGBLOB"/>

<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.SimpleConnectionFactory"/>
<property name="bucketTableNamePrefix" value="lucene"/>
<property name="cacheName" value="FullTextIndexes"/>
<property name="idColumnType" value="VARCHAR(256)"/>
<property name="idColumnName" value="idCol"/>
<property name="dataColumnName" value="dataCol"/>

<property name="timestampColumnName" value="timestampCol"/>
<property name="timestampColumnType" value="BIGINT"/>

</properties>
</loader>
</loaders>
</namedCache>

<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<namedCache
name="LuceneIndexesLocking">
<clustering
mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="30000"/>
<sync replTimeout="50000"/>
</clustering>
</namedCache>

</infinispan>

I am in the process of re indexing with FSDirectory to see if it will work under those circumstances.

Thanks!

I just finished reindexing the database using FSDirectoryProvider instead of infinispan for the backend. Still the index does not apear to update when I update the database. I am confirming this with Luke now that I have access to the files in the filesystem.

I am also including my Hibernate Search Configuration to see if it raises any red flags.

When using FSDirectoryProvider
<property name="hibernate.search.default.directory_provider">org.hibernate.search.store.FSDirectoryProvider</property>
<property name="hibernate.search.default.indexBase"> /var/opt/lucene/indexes</property>

When Using Infinispan
<property name="hibernate.search.default.directory_provider">infinispan</property>
<property name="hibernate.search.infinispan.configuration_resourcename">infinispan.cfg.xml</property>

There are no other settings for Hibernate search set in my Hibernate configuration but if you need any additional information let me know and I will try to provide it.

Thanks again!

Ok thanks now that we know it doesn't depend on Infinispan we narrowed it down from many other possible issues (network, etc...).


After reindexing the index contains the correct values?


How are you updating the database? Keep in mind that Hibernate Search is not able to intercept direct SQL/HQL update statements, only changed detected by listening to events on the entities (so entities you actually loaded in memory).
Also it won't be able to detect changed performed by other systems.

Other than that, we enabled recently some optimisations in how dirty elements are detected. They are enabled by default as we considered them safe enough, but to make sure could you try disabling them by setting


(I'm assuming you're using version 3.4.1.Final ?)