Problems with indexing spatial entity

AndBro · **Joined:** Mon Nov 26, 2012 4:08 am **Posts:** 24

Hi!
We are upgrading a system from java 1.6 to java 1.8 and also upgrading Hibernate Search from version 4.5.3.Final to 5.5.1.Final.
After upgrade, Hibernate Search won't index certain entities.
We have an entity Utbildning with an indexed list Utbildningstillfalle. Utbildningstillfalle has an indexed entity, Adress. This entity has spatial annotations.

The problem we have is that if the list of Utbildningstillfalle in Utbildning contains more than one object, Utbildning won¨t be indexed.

We get the following error code :
ERROR 2016-04-15 15:57:22,589 org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: HSEARCH000183: Unable to index instance of type Utbildning while batch indexing: Utbildning{namn='utbNamn', forkunskaper='forkunskaper', beskrivning='cccc dddd aaaa anna-lisa', nyckelord='null', startDatum=2011-03-02, slutDatum=2019-03-02, externtSokbar=true, internInformation='intern info', utbildningId=1}
java.lang.IllegalArgumentException: DocValuesField "utbildningstillfalleLista.adress.location_HSSI_Latitude" appears more than once in this document (only one value is allowed per field)
at org.apache.lucene.index.NumericDocValuesWriter.addValue(NumericDocValuesWriter.java:54)
at org.apache.lucene.index.DefaultIndexingChain.indexDocValue(DefaultIndexingChain.java:421)
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:376)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1475)
at org.hibernate.search.backend.impl.lucene.IndexWriterDelegate.updateDocument(IndexWriterDelegate.java:74)
at org.hibernate.search.backend.impl.lucene.IndexWriterDelegate.addDocument(IndexWriterDelegate.java:53)
at org.hibernate.search.backend.impl.lucene.works.AddWorkExecutor.performWork(AddWorkExecutor.java:54)
at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:52)
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.applyStreamWork(LuceneBackendQueueProcessor.java:76)
at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:125)
at org.hibernate.search.backend.impl.StreamingOperationExecutorSelector$AddSelectionExecutor.performStreamOperation(StreamingOperationExecutorSelector.java:106)
at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.sendWorkToShards(DefaultBatchBackend.java:62)
at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.enqueueAsyncWork(DefaultBatchBackend.java:48)
at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.index(IdentifierConsumerDocumentProducer.java:292)
at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.indexAllQueue(IdentifierConsumerDocumentProducer.java:223)
at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.loadList(IdentifierConsumerDocumentProducer.java:177)
at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.loadAllFromQueue(IdentifierConsumerDocumentProducer.java:140)
at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.run(IdentifierConsumerDocumentProducer.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Has anyone encountered this problem before? Sollutions?
If I remove the spatial annotations in Adress and index the fields as an ordinary Double, there are no problems so it has to do with spatial annotations.

Here are som code snippets:

Code:

@Entity
@FullTextFilterDefs({
        @FullTextFilterDef(name = "utbildningInternSokningFilterFactory", impl = UtbildningInternSokningFilterFactory.class, cache = FilterCacheModeType.NONE),
        @FullTextFilterDef(name = "utbildningExternSokningFilterFactory", impl = UtbildningExternSokningFilterFactory.class, cache = FilterCacheModeType.NONE),
        @FullTextFilterDef(name = "utbildningUtanTillfallenInternSokningFilterFactory", impl = UtbildningUtanTillfallenInternSokningFilterFactory.class, cache = FilterCacheModeType.NONE)
})
@AnalyzerDef(name = "commaalyzer",
        charFilters = {@CharFilterDef(factory = HTMLStripCharFilterFactory.class)},
        tokenizer = @TokenizerDef(factory = PatternTokenizerFactory.class,
                params = {
                        @org.hibernate.search.annotations.Parameter(name = "pattern", value = ",")
                }),
        filters = {
                @TokenFilterDef(factory = LowerCaseFilterFactory.class)
        })
@DynamicUpdate(value = true)
@Table(name = "UTBILDNING")
@Indexed(interceptor = IndexWhenStatusActiveInterceptor.class)
public class Utbildning {
    @DocumentId
    @Id
    @Column(name = "UTBILDNING_ID")
    @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "UTBILDNING_SEQ_GEN")
    @SequenceGenerator(name = "UTBILDNING_SEQ_GEN", sequenceName = "SEQ_UTBILDNING", allocationSize = 1)
    private Long utbildningId;

    @IndexedEmbedded(depth = 4, includeEmbeddedObjectId = true)
    @OneToMany(mappedBy = "utbildning", fetch = FetchType.LAZY, cascade = CascadeType.REFRESH)
    private Set<DUtbildningstillfalle> utbildningstillfalleLista = new HashSet<DUtbildningstillfalle>();


    public Utbildning() {
    }
    public Long getUtbildningId() {
        return utbildningId;
    }
    public void setUtbildningId(Long utbildningId) {
        this.utbildningId = utbildningId;
    }
    public Set<Utbildningstillfalle> getUtbildningstillfalleLista() {
        return utbildningstillfalleLista;
    }
    public void setUtbildningstillfalleLista(
            Set<Utbildningstillfalle> utbildningstillfalleLista) {
        this.utbildningstillfalleLista = utbildningstillfalleLista;
    }
    /* Removed irrelevant fields and equals/hashcode methods */

}

Code:

@Entity
@FullTextFilterDefs({
        @FullTextFilterDef(name = "utbildningstillfalleExternSokningFilterFactory", impl = UtbildningstillfalleExternSokningFilterFactory.class),
        @FullTextFilterDef(name = "utbildningstillfalleInternSokningFilterFactory", impl = UtbildningstillfalleInternSokningFilterFactory.class)
})
@DynamicUpdate(value = true)
@Table(name = "UTBILDNINGSTILLFALLE")
@NamedQueries({@NamedQuery(name = DUtbildningstillfalle.NQ_HAMTA_ALLA, query = "SELECT u FROM DUtbildningstillfalle u")})
@Indexed(interceptor = IndexWhenStatusActiveInterceptor.class)
public class Utbildningstillfalle {

    @DocumentId
    @Id
    @Column(name = "UTBILDNINGSTILLFALLE_ID")
    @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "UTBILDNINGSTILLFALLE_SEQ_GEN")
    @SequenceGenerator(name = "UTBILDNINGSTILLFALLE_SEQ_GEN", sequenceName = "SEQ_UTBILDNINGSTILLFALLE", allocationSize = 1)
    private Long utbildningstillfalleId;

    @ContainedIn
    @ManyToOne(fetch = FetchType.LAZY, cascade = CascadeType.MERGE, optional = false)
    @JoinColumn(name = "UTBILDNING_ID", referencedColumnName = "UTBILDNING_ID", nullable = false)
    private Utbildning utbildning;

    @IndexedEmbedded(includeEmbeddedObjectId = true)
    @ManyToOne(fetch = FetchType.LAZY, cascade = CascadeType.MERGE, optional = false)
    @JoinColumn(name = "ADRESS_ID", referencedColumnName = "ADRESS_ID", nullable = false)
    private Adress adress;

    public void setUtbildning(Utbildning utbildning) {
        this.utbildning = utbildning;
    }
    public Utbildning getUtbildning() {
        return utbildning;
    }
    public Adress getAdress() {
        return adress;
    }
   public void setAdress(Adress adress) {
        this.adress = adress;
    }
    /* Removed irrelevant fields and equals/hashcode methods */
    
}

Code:

@Spatial(name = "location", spatialMode = SpatialMode.RANGE)
@Indexed(interceptor = IndexWhenStatusActiveInterceptor.class)
@Entity
@DynamicUpdate(value = true)
@Table(name = "ADRESS")
public class Adress {

    @DocumentId
    @Id
    @Column(name = "ADRESS_ID")
    @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ADRESS_SEQ_GEN")
    @SequenceGenerator(name = "ADRESS_SEQ_GEN", sequenceName = "SEQ_ADRESS", allocationSize = 1)
    private Long adressId;

    @Latitude(of = "location")
    @Field(index = Index.YES, analyze = Analyze.YES, store = Store.YES, indexNullAs = "-1")
    @Column(name = "LATITUDE")
    private Double latitud;

    @Longitude(of = "location")
    @Field(index = Index.YES, analyze = Analyze.YES, store = Store.YES, indexNullAs = "-1")
    @Column(name = "LONGITUDE")
    private Double longitud;

    public Adress() {
    }
    public Long getAdressId() {
        return adressId;
    }
    public void setAdressId(Long adressId) {
        this.adressId = adressId;
    }
    public Double getLatitud() {
        return latitud;
    }
    public void setLatitud(Double latitud) {
        this.latitud = latitud;
    }
    public Double getLongitud() {
        return longitud;
    }
    public void setLongitud(Double longitud) {
        this.longitud = longitud;
    }
    /* Removed irrelevant fields and equals/hashcode methods */
}

sanne.grinovero · **Posted:** Mon Apr 18, 2016 4:30 pm

Hi,
your main indexed entity will produce spatial coordinates with names 'utbildningstillfalleLista.adress.location_HSSI_Latitude' and 'utbildningstillfalleLista.adress.location_HSSI_Longitude' when "flattened" as a Lucene Document.

However to have this produce valid results when performing a Spatial Query, these fields can not be repeated multiple times in the same Document.
I'm afraid that the mapping is invalid in both the older and the new version of Hibernate Search, the difference is that the older version is not detecting the problem correctly: I suspect the results you had were not always correct.

We should improve Hibernate Search to detect this mapping problem at boot time, rather than throw an exception at runtime:
- https://hibernate.atlassian.net/browse/HSEARCH-2230

Regarding your problem, you need to look at reversing the mapping to the index; for example it might be possible to change your logic to search on class "Utbildningstillfalle" rather than "Utbildning" ?

The problem is the cardinality of the "@IndexedEmbedded Set<DUtbildningstillfalle>": embedded indexed collections can only contain spatial fields when it's not a "toMany" relation.

AndBro · **Joined:** Mon Nov 26, 2012 4:08 am **Posts:** 24

sanne.grinovero wrote:

Hi,
The problem is the cardinality of the "@IndexedEmbedded Set<DUtbildningstillfalle>": embedded indexed collections can only contain spatial fields when it's not a "toMany" relation.

Hi Sanne and thank you for your answer!
This is a big drawback for us, I would have thougt this scenario should be quite common, for instance a company with a list of divisions with addresses.
Unfortunately, we can't in an easy way change the search to Utbildningstillfalle. What I have to do is remove the Spatial annotations and just index latitude and longitude as ordinary fields and do my own range check in a filter.
To bad.
Shouldn't this problem be mentioned in the documentation? I couldn't find anything about this.
Regards
Andreas

sanne.grinovero · **Posted:** Tue Apr 19, 2016 6:42 pm

Hi Andreas,
I'm sorry but even if you index latitude & longitude as "ordinary fields" you would still have a hard to solve problem: having multiple Addresses embedded, you will have multiple "latitude" and multiple "longitude" fields for the same document.
When querying that, you will not be able to filter in such a way to pair the latitude(s) and longitude(s) from the same Address: that's the problem which I mentioned would affect previous versions.

Unfortunately it seems nobody noticed this limitation in combination with Spatial fields so far, so it's not documented (!!). I agree it's a strong limitation so I look forward to discuss about a proper implementation with the team - you're very welcome to join the brainstorming and propose a solution.

I did workaround a similar situation in the past - not with Spatial - by creating a different field name for each 1..N entry. For example you could devise a custom fieldbridge for Set<Address> which would generate something like "lat_1", "long_1", "lat_2", "long_2", "lat_3", "long_3", but as you can imagine it only works when you know the maximum set cardinality, and makes it much harder to query.

sanne.grinovero · **Posted:** Tue Apr 19, 2016 6:48 pm

P.S. the short term solution is to denormalize, like I suggested in my first answer: that's what most people do, and will get you the best performance out of Lucene, probably better than any alternative that we might come up with.

AndBro · **Joined:** Mon Nov 26, 2012 4:08 am **Posts:** 24

Hi Sanne, thank you for your reply.
I think the filtering approach will work for me because I don't really have to pair the latitude/longitude to a specific address as long as each pair of latitudes and longitudes belong together, or is that random too?. It is enough that one address matches the range criteria. I don't really need to know which one.
Maybe I have to do the custom fieldbridge and individually name the fields. Should be possible to loop over all the fields and sort them and match them together. If I wanted, I could also append the Id of the Address if I want to match the coordinate pairs to a specific Address.

AndBro · **Joined:** Mon Nov 26, 2012 4:08 am **Posts:** 24

Hi again!
I read your response too quickly. I don't have a Set<Address>, it's only one address to one Utbildningstillfalle so the filter approack should work without a FieldBridge?
because each utbildningstillfalleLista.adress.latitude should belong to the next utbildningstillfalleLista.adress.longitude? I will have to do some tests to se how the document is organized.

sanne.grinovero · **Posted:** Wed Apr 20, 2016 4:38 am

Quote:

as long as each pair of latitudes and longitudes belong together, or is that random too?

That's random :-(
Sorry, that's what I was trying to explain. That's how Lucene works (since recently it's able to perform a *single* join but we didn't implement support for that yet, and isn't easily mapped to a general purpose join), so you have to work on refactoring the model to match the Lucene patterns: denormalize, and search for Address.

You can invert the relation, and have an entity "address -> indexedmbedded utbildningstillfalleLista"

AndBro · **Joined:** Mon Nov 26, 2012 4:08 am **Posts:** 24

OK, then we have a big problem. Because we search on Entity Utbildning that has a set of UtbildningTillfalle that each has an Address. We have to serach on Entity Utbildning. We also have a lot of other fields on entity Utbildningstillfalle that are indexed. Since the Entity Utbildning has a Set of Utbildningstillfalle, we will have the same problem with all the fields indexed in Utbildningstillfalle in a filter.
I think we have to rethink our model.
/Andreas

Gunnar · **Joined:** Sat Jan 24, 2009 12:46 pm **Posts:** 388

As Sanne was suggesting, you could have the index on DUtbildningstillfalle instead of Utbildning.

You already have the back reference to Utbildning, so you can add @IndexedEmbedded to this association and include fields from Utbildning so you can filter on them, too. As Utbildningstillfalle only has exactly one Address, the doc value fields for longitude/latitude will only be added once to each Document.

It's not perfect, but as good as it gets for the time being.

Hth,

--Gunnar

AndBro · **Joined:** Mon Nov 26, 2012 4:08 am **Posts:** 24

Thanks for your reply, Gunnar.
Unfortunately, we want to be able to get hits even if there are no elements in the set of Utbildningstillfalle, the user could have entered search criteria that matches fields in Utbildning and no search criteria for spatial search. We don't want to divide it up to several queries either because then we have to implement our own pagination. But perhaps we have to do something like that. I have to think about it :-)