Hibernate OGM/Search/Infinispan + two apps: no sync

maciej.j · **Joined:** Mon Sep 01, 2014 2:23 am **Posts:** 5

Hello,

Let me first briefly describe configuration:
- standalone Infinispan 7.0 server (wildfly) and HotRod as communication protocol // i have changed port numbers in server config file so don't be surprised when you see 8080 as HotRod port in attached code listings
- two applications using Hibernate OGM (I am using my custom Riak dialect implementation but I also tested stock CouchDB and problem is the same so I guess it's irrelevant)
- applications have defined namedCaches as remoteStores which are stored on remote standalone server (192.168.1.130)
- both applications are simple "public static void Main" console applications
- first is basically an infinite loop executing a query em.createQuery("FROM Account")... and printing result to output
* the Query is powered by Hibernate Search + Infinispan
- second application creates new Account and persists it using EntityManager

Use case:
I start app1 (looper) and then app2 (producer). App2 creates and persists one entity in the DB+Infinispan index.
And here is the problem. App1 does not see that index has changed and it prints output corresponding to the state before.
// App1 however can access this new entity directly by em.find(Account.class, id); only the index based queries are not working as they should

When I restart App1 - then it (updates the index I guess?) shows correct output - including the recently added entity by App2.
This is not acceptable for me.

I also did some other experiment:
modified a bit the App1 that in each loop iteration a whole new EntityManagerFactory is being created (moved emf = Persistence.createEntityManagerFactory("dev22"); and emf.close(); inside for). With this approach the printed output is always up to date - no need to restart application.
This however is also not acceptable for me - presented apps are just samples for issue description purpose. I am going to use Hibernate OGM + Search + Infinispan in web application with ejbs etc. where EMF is controlled by container.

Is this expected behavior? If not could you please help me with this problem? Maybe my persistence/infinispan configuration is wrong or something...

I also checked compatibility of bare Infinispan HotRod client 6.0.2 with Infinispan 7.0 server and it works as designed so it should not be a problem with Infinispan configuration itself I guess...
For instance in AppA I loop and print out number of cache keys:

Code:

BasicCacheContainer cacheContainer = new DefaultCacheManager("infinispan.xml");
BasicCache<String, String> cache = cacheContainer.getCache("LuceneIndexesMetadata");
for (;;) {
   System.out.println(cache.keySet().size());
   Thread.sleep(2000);
}

and with AppB I add something to the index:

Code:

BasicCacheContainer cacheContainer = new DefaultCacheManager("infinispan.xml");
BasicCache<String, String> cache = cacheContainer.getCache("LuceneIndexesMetadata");
cache.put("car", "zonk");

A reacts to changes performed by B in realtime.

Config files, code snippets:

persistence.xml

Code:

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.1"
          xmlns="http://xmlns.jcp.org/xml/ns/persistence"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/persistence http://www.oracle.com/webfolder/technetwork/jsc/xml/ns/persistence/persistence_2_1.xsd">
   <persistence-unit name="dev22" transaction-type="JTA">
      <provider>org.hibernate.ogm.jpa.HibernateOgmPersistence</provider>
      <jta-data-source>java:jboss/datasources/ExampleDS</jta-data-source>

      <class>poc.ogm.search.business.indexer.entity.Account</class>

      <properties>
         <property name="hibernate.ogm.datastore.provider" value="couchdb"/>
         <property name="hibernate.ogm.datastore.host" value="localhost"/>
         <property name="hibernate.ogm.datastore.port" value="5984"/>
         <property name="hibernate.ogm.datastore.database" value="testdb"/>
         <property name="hibernate.ogm.datastore.create_database" value="true"/>

         <!-- defines which JTA Transaction we plan to use -->
         <property name="hibernate.transaction.jta.platform" value="org.hibernate.service.jta.platform.internal.JBossStandAloneJtaPlatform"/>
         <!--<property name="hibernate.transaction.jta.platform" value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform" />-->

         <property name="hibernate.search.default.directory_provider" value="infinispan"/>
         <property name="hibernate.search.default.exclusive_index_use" value="false"/>
         <property name="hibernate.search.infinispan.configuration_resourcename" value="infinispan.xml"/>
      </properties>
   </persistence-unit>
</persistence>

infinispan.xml

Code:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="urn:infinispan:config:6.0 http://www.infinispan.org/schemas/infinispan-config-6.0.xsd
      urn:infinispan:config:remote:6.0 http://www.infinispan.org/schemas/infinispan-cachestore-remote-config-6.0.xsd"
      xmlns="urn:infinispan:config:6.0"
      xmlns:remote="urn:infinispan:config:remote:6.0">

   <global>
      <globalJmxStatistics
            enabled="true"
            cacheManagerName="HibernateCacheManager"/>

      <shutdown
            hookBehavior="DONT_REGISTER" />
   </global>

   <namedCache name="LuceneIndexesMetadata">
      <persistence passivation="false">
         <remoteStore xmlns="urn:infinispan:config:remote:6.0" fetchPersistentState="false" preload="true" shared="false"
                   purgeOnStartup="false" ignoreModifications="false" remoteCacheName="default">
            <servers>
               <server host="192.168.1.130" port="8080" />
            </servers>
            <connectionPool maxActive="10" exhaustedAction="CREATE_NEW" />
         </remoteStore>
      </persistence>
   </namedCache>

   <namedCache name="LuceneIndexesData">
      <persistence passivation="false">
         <remoteStore xmlns="urn:infinispan:config:remote:6.0" fetchPersistentState="false" preload="true" shared="false"
                   purgeOnStartup="false" ignoreModifications="false" remoteCacheName="default">
            <servers>
               <server host="192.168.1.130" port="8080" />
            </servers>
            <connectionPool maxActive="10" exhaustedAction="CREATE_NEW" />
         </remoteStore>
      </persistence>
   </namedCache>

   <namedCache name="LuceneIndexesLocking">
      <persistence passivation="false">
         <remoteStore xmlns="urn:infinispan:config:remote:6.0" fetchPersistentState="false" preload="true" shared="false"
                   purgeOnStartup="false" ignoreModifications="false" remoteCacheName="default">
            <servers>
               <server host="192.168.1.130" port="8080" />
            </servers>
            <connectionPool maxActive="10" exhaustedAction="CREATE_NEW" />
         </remoteStore>
      </persistence>
   </namedCache>
</infinispan>

crucial dependencies in pom.xml:

Code:

<dependencyManagement>
      <dependencies>
         <dependency>
            <groupId>org.hibernate.ogm</groupId>
            <artifactId>hibernate-ogm-bom</artifactId>
            <version>4.1.0.Beta7</version>
            <type>pom</type>
            <scope>import</scope>
         </dependency>
      </dependencies>
   </dependencyManagement>
...
    <dependencies>
      <dependency>
         <groupId>org.hibernate.ogm</groupId>
         <artifactId>hibernate-ogm-couchdb</artifactId>
         <version>4.1.0.Beta7</version>
      </dependency>
      <dependency>
         <groupId>org.jboss.spec.javax.transaction</groupId>
         <artifactId>jboss-transaction-api_1.2_spec</artifactId>
         <version>1.0.0.Final</version>
      </dependency>
      <dependency>
         <groupId>org.hibernate.javax.persistence</groupId>
         <artifactId>hibernate-jpa-2.1-api</artifactId>
         <version>1.0.0.Final</version>
      </dependency>
      <dependency>
         <groupId>org.jboss.jbossts</groupId>
         <artifactId>jbossjta</artifactId>
         <version>4.16.4.Final</version>
      </dependency>
      <dependency>
         <groupId>org.hibernate</groupId>
         <artifactId>hibernate-search-orm</artifactId>
         <version>4.5.1.Final</version>
         <exclusions>
            <exclusion>
               <artifactId>hibernate-search-analyzers</artifactId>
               <groupId>org.hibernate</groupId>
            </exclusion>
            <exclusion>
               <artifactId>slf4j-api</artifactId>
               <groupId>org.slf4j</groupId>
            </exclusion>
         </exclusions>
      </dependency>
      <dependency>
         <groupId>org.hibernate</groupId>
         <artifactId>hibernate-search-engine</artifactId>
         <version>4.5.1.Final</version>
         <exclusions>
            <exclusion>
               <artifactId>hibernate-search-analyzers</artifactId>
               <groupId>org.hibernate</groupId>
            </exclusion>
            <exclusion>
               <artifactId>slf4j-api</artifactId>
               <groupId>org.slf4j</groupId>
            </exclusion>
         </exclusions>
      </dependency>
      <dependency>
         <groupId>org.hibernate</groupId>
         <artifactId>hibernate-search-infinispan</artifactId>
         <version>4.5.1.Final</version>
      </dependency>

      <dependency>
         <groupId>org.infinispan</groupId>
         <artifactId>infinispan-client-hotrod</artifactId>
         <version>6.0.2.Final</version>
         <scope>compile</scope>
      </dependency>
      <dependency>
         <groupId>org.infinispan</groupId>
         <artifactId>infinispan-cachestore-remote</artifactId>
         <version>6.0.2.Final</version>
         <scope>compile</scope>
      </dependency>
...

app1 (looper):

Code:

package poc.ogm.search.business.indexer.control;

import poc.ogm.search.business.indexer.entity.Account;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
import javax.transaction.TransactionManager;
import java.lang.reflect.InvocationTargetException;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;

public class Viewer {
   private static final String JBOSS_TM_CLASS_NAME = "com.arjuna.ats.jta.TransactionManager";
   private static final Logger log = Logger.getLogger(Viewer.class.getName());

   public static void main(String[] args) {
      TransactionManager tm = getTransactionManager();
      boolean active = true;

      //build the EntityManagerFactory as you would build in in Hibernate Core
      EntityManagerFactory emf;

      try {
         EntityManager em;

         //Retrieve your entities the way you are used to in plain JPA
         log.info("About to retrieve accounts");
         emf = Persistence.createEntityManagerFactory("dev22");
         for (;;) {
            tm.begin();
            em = emf.createEntityManager();
            List<Account> accounts = em.createQuery( "FROM Account", Account.class ).getResultList();
            log.info("no. of accounts: " + accounts.size());
            for (Account account : accounts) {
               log.log( Level.INFO, account.toString() );
            }
            Thread.sleep(2000);
            em.close();
            tm.commit();

            if (!active)
               break;
         }
         emf.close();

      } catch ( Exception e ) {
         e.printStackTrace();
      }

   }

   public static TransactionManager getTransactionManager() {
      try {
         Class<?> tmClass = Viewer.class.getClassLoader().loadClass( JBOSS_TM_CLASS_NAME );
         return (TransactionManager) tmClass.getMethod( "transactionManager" ).invoke( null );
      } catch ( ClassNotFoundException e ) {
         e.printStackTrace();
      } catch ( InvocationTargetException e ) {
         e.printStackTrace();
      } catch ( NoSuchMethodException e ) {
         e.printStackTrace();
      } catch ( IllegalAccessException e ) {
         e.printStackTrace();
      }
      return null;
   }
}

app2 (producer):

Code:

package poc.ogm.search.business.indexer.control;

import poc.ogm.search.business.indexer.entity.Account;
import org.hibernate.ogm.util.impl.Log;
import org.hibernate.ogm.util.impl.LoggerFactory;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
import javax.transaction.TransactionManager;
import java.lang.reflect.InvocationTargetException;

public class Producer {
   private static final String JBOSS_TM_CLASS_NAME = "com.arjuna.ats.jta.TransactionManager";
   private static final Log logger = LoggerFactory.make();

   public static void main(String[] args) {
      TransactionManager tm = getTransactionManager();

      //build the EntityManagerFactory as you would build in in Hibernate Core
      EntityManagerFactory emf = Persistence.createEntityManagerFactory("dev22");

      try {
         EntityManager em;
         Account acc;

         tm.begin();
         logger.infof( "About to store account" );
         em = emf.createEntityManager();

         acc = new Account();
         acc.setName("name1");
         acc.setAccessLevel(1);

         em.persist( acc );
         String accId = acc.getId();
         em.flush();
         em.close();
         tm.commit();


         //Retrieve your entities the way you are used to in plain JPA
         logger.infof( "About to retrieve account" );
         tm.begin();
         em = emf.createEntityManager();
         acc = em.find( Account.class, accId );
         logger.infof( "Found account %s [Level=%d]", acc.getName(), acc.getAccessLevel() );
         em.flush();
         em.close();
         tm.commit();

         emf.close();
      } catch ( Exception e ) {
         e.printStackTrace();
      }

   }

   public static TransactionManager getTransactionManager() {
      try {
         Class<?> tmClass = Producer.class.getClassLoader().loadClass( JBOSS_TM_CLASS_NAME );
         return (TransactionManager) tmClass.getMethod( "transactionManager" ).invoke( null );
      } catch ( ClassNotFoundException e ) {
         e.printStackTrace();
      } catch ( InvocationTargetException e ) {
         e.printStackTrace();
      } catch ( NoSuchMethodException e ) {
         e.printStackTrace();
      } catch ( IllegalAccessException e ) {
         e.printStackTrace();
      }
      return null;
   }
}

Account:

Code:

@Indexed
@Entity
public class Account {

   @Id
   @GeneratedValue(generator = "uuid")
   @GenericGenerator(name = "uuid", strategy = "uuid2")
   private String id;
   @Field(store = Store.YES, analyze = Analyze.NO, indexNullAs = "NULL_VALUE")
   private String name;
   @Field(store = Store.YES, analyze = Analyze.NO)
   @NumericField
   private int accessLevel;

   public String getId() {
      return id;
   }

   public void setId(String id) {
      this.id = id;
   }

   public String getName() {
      return name;
   }

   public void setName(String name) {
      this.name = name;
   }

   public int getAccessLevel() {
      return accessLevel;
   }

   public void setAccessLevel(int accessLevel) {
      this.accessLevel = accessLevel;
   }

   @Override
   public String toString() {
      return "Account{" +
            "id='" + id + '\'' +
            ", name='" + name + '\'' +
            ", accessLevel=" + accessLevel +
            '}';
   }
}

Thanks,
Maciej

maciej.j · **Joined:** Mon Sep 01, 2014 2:23 am **Posts:** 5

OK, I got the point (at least I think so;) ) what's happening here and found solution for my problem.
I didn't realize that master/slave architecture applies only to Hibernate Search and has nothing to do with Infinispan itself.

Now I have created two persistence units. One with <property name="hibernate.search.default.worker.backend" value="jgroupsSlave"/> (for producer) and another with <property name="hibernate.search.default.worker.backend" value="jgroupsMaster"/> property (for looper). Infinispan configuration is just like before. Now everything works like a charm.

--
Kind regards
Maciej

sanne.grinovero · **Posted:** Thu Nov 06, 2014 8:01 am

Great job!

That's correct, you need to connect the backends in Hibernate Search. We hope to automate that, to reuse the JGroups channel from the Infinispan storage layer but that's work in progress.
If you want to go one step further, you just need to make sure you use the FORK protocol of JGroups so that you can share the network stack among Infinispan and Search.. although that's just a small optimisation meant to reduce the number of network ports.

One thing that surprised me is that you are using Hot Rod. I guess just to inspect the cache content remotely?
Neither OGM storage nor the index storage (the Lucene Infinispan Directory) have been tested on Hot Rod; I know OGM wouldn't work correctly (but you store data in Riak so not to worry); the Lucene Directory could work in theory I think but we never tested that, and for sure it's not its ideal configuration.. would be very inefficient as I think it would need to download the whole index on demand each time you need to run a query.

Gunnar · **Joined:** Sat Jan 24, 2009 12:46 pm **Posts:** 388

@maciej.j,

Glad to hear that you got it working! I think I've asked before, but would you be interested in contributing your Riak dialect to the project?

--Gunnar

maciej.j · **Joined:** Mon Sep 01, 2014 2:23 am **Posts:** 5

Gunnar wrote:

I think I've asked before, but would you be interested in contributing your Riak dialect to the project?

Yes. I'm getting closer to moment I could share the code. I have already synced with OGM 4.1.Beta7.
Although I still have some issues to discuss/solve, next week probably I will publish it on github.
For instance there is no embedded Riak server like in MongoDB case so unit tests would require working standalone Riak server to pass. Also there is no server for Windows. Should I prepare some mock in this case?

--
Regards
Maciej

sanne.grinovero · **Posted:** Thu Nov 06, 2014 11:26 am

Great. I wouldn't expect you to code a full mock as its usefulness is quite limited (since it would be a different thing).

This is a common problem with other backends too; what we're doing is to make sure we install all needed databases on ci.hibernate.org , and the core team installs them all locally, but we wouldn't expect for non-regular contributors to do the same, so they would rely on the CI server.
Moving forward we hope to make a docker based "test your OGM" patch container.. that would probably be better thank playing with mocks.

If you could mimick what we did for other dialects that would be fantastic: essentially have the test only if a specific Maven profile is enabled, and the profile get enabled by setting environment properties which configure things like the domain name / port to connect to (we'd need that to be interepreted by tests to be able to run it on CI as well).

Gunnar · **Joined:** Sat Jan 24, 2009 12:46 pm **Posts:** 388

Hi,

Awesome, looking forward to that contribution!

As Sanne is saying, Docker images will likely be the way forward for setting up the required testing infra, but we're not there yet. For the time being, you may take the CouchDB module as base of inspiration. It is always compiled by default (so we'd detect breaking API changes done in core), but its test suite is only executed if the environment variable COUCHDB_HOST is present (in which case that host is used as test datastore).

Hth,

--Gunnar