MassIndexer : Corrupt Record issues.

lovelyliatroim · **Joined:** Thu Oct 08, 2009 10:34 am **Posts:** 55

Hi Guys,

I have a bit of an issue and just looking for some advice. Im currently using the mass indexer to carry out indexing for me. However i have some issues with it.

Long story short, we have some dates in our tables which are invalid in format as defined by our JPA model, they get into our tables from foreign entities.

Now the issue i have with mass indexing is that when HS tries to load a row which has an invalid format for a date our custom user type for dates will throw a hibernate exception

Example

Code:

@Override
   public Object nullSafeGet(ResultSet rs, String[] names, Object owner) throws HibernateException, SQLException {
      String timestamp = rs.getString(names[0]);
      try {
         return DateUtils.parseDateTime(timestamp);
      } catch (ParseException ex) {
         throw new HibernateException("could not parse time stamp: " + timestamp + " owner: " + owner);
      }
   }

So when we have an invalid date we throw a HibernateException. This in turn has the effect that when I index and come across a record with an invalid date the whole batch process for that Entity now stops, so just because of one corrupt record the whole indexing stops.

Sample of my complete stack trace looks like so

Quote:

15:11:47,737 [Hibernate Search: entityloader-1] ERROR batchindexing.IdentifierConsumerEntityProducer - error during batch indexing:
org.hibernate.HibernateException: could not parse time stamp: 20080123184467464 owner: EntityInstance{Details}
at com.mypackage.TimestampType.nullSafeGet(TimestampType.java:40)
at org.hibernate.type.CustomType.nullSafeGet(CustomType.java:109)
at org.hibernate.type.AbstractType.hydrate(AbstractType.java:104)
at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:2265)
at org.hibernate.loader.Loader.loadFromResultSet(Loader.java:1527)
at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1455)
at org.hibernate.loader.Loader.getRow(Loader.java:1355)
at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:611)
at org.hibernate.loader.Loader.doQuery(Loader.java:829)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1716)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.loadList(IdentifierConsumerEntityProducer.java:141)
at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.loadAllFromQueue(IdentifierConsumerEntityProducer.java:110)
at org.hibernate.search.batchindexing.IdentifierConsumerEntityProducer.run(IdentifierConsumerEntityProducer.java:87)
at org.hibernate.search.batchindexing.OptionallyWrapInJTATransaction.run(OptionallyWrapInJTATransaction.java:107)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

My question to you guys is........should the mass indexer stop indexing because of one invalid record?? (I dont think it should ;-) ) Any workarounds??

Thanks guys,
LL

sanne.grinovero · **Posted:** Wed Feb 09, 2011 9:32 am

sorry the MassIndexer expects the data in your tables to be healthy. I'd highly recommend to fix the external source, or as a workaround send some SQL statement before the MassIndexer to fix invalid records.
It's not currently possible to have the MassIndexer continue safely from such a corrupted state, but there are plans to try to recover something; please comment on HSEARCH-601 about your use case as I still need to clarify this.