About creation of these 'special' indexes. We have a logic (based directly on lucene) that downloads html pages and indexes their content into index files. The structure of these index files corresponds with the structure that HSearch generates as I can control it in lucene.
I see, cool. You could take advantage of the IndexWriter management and async pipelines of Hibernate Search by sending "Work" operations directly to the backend, see how org.hibernate.search.backend.impl.BatchedQueueingProcessor
is being used.. unless you already have a state of the art writer management that could buy you some extra performance and better integration with the other services (like NRT readers and exclusive writers).