Hi,

I am using a Hibernate Search full text session in order to write Entities with Hibernate Search Annotations to a Lucene Index.
I get my entities with a special bulk load query and cannot use the mass indexer.
Therefore I reattach my entities with session.save.

Unfortunately, the reattach is very slow (takes 10 times as long as reading or writing).
Also, when I do it in parallel with multiple sessions, it does not speed up by more than factor 2.

Is there another way to do mass indexing?
I could not use the StatelessSession since org.hibernate.search.Search.getFullTextSession(session)
does not work with it.

Regards
Eva

Hi,

why can you not use the mass indexer. It is by far the most efficient way of doing it.

If you provide some of your code we might be able to help you more.

--Hardy

Hi Hardy,

well, we have a bitemporal data modell and need to calculate a special view on the data.

Our data are addresses and they consist of building blocks like the zip and the street.
Each of the building blocks has a time stamp.
What we need to calculate are all the time slices of the adress, the whole tree, from the time stamps of its building blocks.
That is what is indexed.

This could be modelled in the view below, but then there is also some java code (global synonyms, calculation of valid from) changing the data.

Would it be possible to use the mass indexer with own java code and that view?
My team told me tht it was not possible to use the mass indexer unless you take the data 1:1 from the db.
But maybe that's not true...

--- Eva

The View - Note: It must be parameterized from java with additional search parameters in inner selects so it does not make the TEMP tablespace explode.


with
sf_pers_addr as (
select
pers_addr.key as padr_key,
pers_addr.valid_from as padr_valid_from,
sf_impers_addr.valid_from as impers_addr_valid_from
,kap_zip_street.valid_from as street_valid_from
,kap_street_vt.valid_from as kap_street_vt_valid_from
,kap_po_box.valid_from as kap_po_box_valid_from
,kap_pob_facility.valid_from as KAP_POB_FACILITY_valid_from
,virtual_addr.valid_from as virtual_addr_valid_from
,zipPAdr.valid_from as zipPadr_valid_from
,zipPobF.valid_from as zipPobF_valid_from
,zipStreet.valid_from as zipStreet_valid_from
from kap_pers_addr_vt pers_addr
left join sf_impers_addr on pers_addr.impers_addr_key=sf_impers_addr.impaddr_key and sf_impers_addr.valid_from is not null
left join kap_virtual_addr_vt virtual_addr ON pers_addr.key = virtual_addr.key and virtual_addr.valid_from is not null
--
left join kap_po_box ON pers_addr.key = kap_po_box.key and kap_po_box.valid_from is not null
left join KAP_POB_FACILITY on kap_po_box.po_box_facility_key = KAP_POB_FACILITY.key and KAP_POB_FACILITY.valid_from is not null
--
left join kap_zip_street_vt kap_zip_street on sf_impers_addr.zip_street_key=kap_zip_street.street_key and kap_zip_street.valid_from is not null
left join kap_street_vt ON kap_zip_street.street_key = kap_street_vt.street_key and kap_street_vt.valid_from is not null
--
left join sf_zips zipPAdr on pers_addr.zip_onrp = zipPAdr.onrp and zipPAdr.valid_from is not null
left join sf_zips zipPobF on KAP_POB_FACILITY.zip_onrp = zipPobF.onrp and zipPobF.valid_from is not null
left join sf_zips zipStreet on kap_zip_street.zip_onrp = zipStreet.onrp and zipStreet.valid_from is not null
WHERE pers_addr.valid_from is not null
-- ORA-01652: Temp-Segment kann nicht um 256 in Tablespace TEMP erweitert werden
and kap_search_bulkload.get_key_min <= pers_addr.key and pers_addr.key < kap_search_bulkload.get_key_max
),
sf_changes_null as (
-- pivot & distinct
-- todo: change type: von welcher tabelle kam change?
select padr_valid_from as validity, padr_key from sf_pers_addr
union
select impers_addr_valid_from as valid_from, padr_key from sf_pers_addr
union
select street_valid_from as valid_from, padr_key from sf_pers_addr
union
select kap_street_vt_valid_from as valid_from, padr_key from sf_pers_addr
union
select kap_po_box_valid_from as valid_from, padr_key from sf_pers_addr
union
select KAP_POB_FACILITY_valid_from as valid_from, padr_key from sf_pers_addr
union
select virtual_addr_valid_from as valid_from, padr_key from sf_pers_addr
union
select zipPadr_valid_from as valid_from, padr_key from sf_pers_addr
union
select zipPobF_valid_from as valid_from, padr_key from sf_pers_addr
union
select zipStreet_valid_from as valid_from, padr_key from sf_pers_addr
)
-- temp table ?
, sf_changes as(
select validity, padr_key as key from sf_changes_null
where validity is not null -- TODO
--order by validity_key, validity
)
select

/* HASH(sf_changes) */

sf_changes.validity,

-- keys
personaliz0_.key as pa_vt_key,
personaliz14_.key as pa_key,
nondeliver1_.key as pa_nondeladr_key,
virtualadd2_.key as pa_virualadr_key,
pobox3_.key as pa_bobox_key,
poboxfacil5_.key as pa_poboxfacil_key,
zip6_.onrp as pa_zip_onrp,
zipdescrip9_.onrp as pa_zip_desc_onrp,
zipbase10_.onrp as pa_zip_base_onrp,
alternativ11_.key as pa_zip_alternative_onrp,
domicile8_.key as pa_person_domicil_key,
postrestan12_.key as pa_postrestant_key,
impersonal15_.impaddr_key as ia_key,
zipstreet17_.zip_street_key as ia_zip_street_key,
zip19_.onrp as ia_zip_onrp,
--zip30_.onrp as ia_zip2_onrp,
zipdescrip22_.onrp as ia_zip_desc_onrp,
zipbase25_.onrp as ia_zip_base_onrp,
alternativ28_.key as ia_zip_alternative_onrp,
street32_.street_key as ia_zipStreet_street_key,
alternativ35_.key as ia_alternative_zips_key,
--mailboxfac37_.mailb_fac_key as ia_mailbox_facility_key,

personaliz0_.valid_from as pa_valid3_24_0_,
personaliz0_.invalid_from as pa_invalid2_24_0_,
personaliz0_.active_from_log_id as pa_active9_24_0_,
personaliz0_.address_add_on as pa_address4_24_0_,
personaliz0_.co_address as pa_co5_24_0_,
personaliz0_.deliverable as pa_delivera6_24_0_,
personaliz0_.impers_addr_key as pa_impers10_24_0_,
personaliz0_.mailbox_mailbox_key as mailbox11_24_0_,
personaliz0_.non_deli_foreign_addr_key as non12_24_0_,
personaliz0_.person_key as person13_24_0_,
personaliz0_.po_box_key as po14_24_0_,
personaliz0_.post_rest_key as post15_24_0_,
personaliz0_.rack_labeling as rack7_24_0_,
personaliz0_.status as status24_0_,
personaliz0_.virtual_address_key as virtual16_24_0_,
personaliz0_.zip_onrp as zip17_24_0_,

impersonal15_.valid_from as ia_valid3_11_12_,
impersonal15_.invalid_from as ia_invalid2_11_12_,
impersonal15_.active_from_log_id as active8_11_12_,
impersonal15_.house_nbr as house4_11_12_,
impersonal15_.house_nbr_appendix as house5_11_12_,
impersonal15_.official as official11_12_,
--impersonal15_.state as state11_12_,
impersonal15_.zip_street_key as zip9_11_12_,

nondeliver1_.invalid_from as invalid2_17_1_,
nondeliver1_.active_from_log_id as active11_17_1_,
nondeliver1_.valid_from as valid3_17_1_,
nondeliver1_.country_iso_abbr as country12_17_1_,
nondeliver1_.row1 as row4_17_1_,
nondeliver1_.row2 as row5_17_1_,
nondeliver1_.row3 as row6_17_1_,
nondeliver1_.row4 as row7_17_1_,
--nondeliver1_.state as state17_1_,
nondeliver1_.town as town17_1_,
nondeliver1_.zip as zip17_1_,

virtualadd2_.invalid_from as invalid2_13_2_,
virtualadd2_.active_from_log_id as active8_13_2_,
virtualadd2_.valid_from as valid3_13_2_,
virtualadd2_.row1 as row4_13_2_,
virtualadd2_.row2 as row5_13_2_,
virtualadd2_.row3 as row6_13_2_,
--virtualadd2_.state as state13_2_,
virtualadd2_.zip_onrp as zip9_13_2_/*,

pobox3_.invalid_from as invalid2_15_3_,
pobox3_.active_from_log_id as active7_15_3_,
pobox3_.valid_from as valid3_15_3_,
pobox3_.nbr as nbr15_3_,
pobox3_.po_box_facility_key as po8_15_3_,
--pobox3_.state as state15_3_,
pobox3_.type as type15_3_,

poboxfacil5_.invalid_from as invalid2_14_4_,
poboxfacil5_.active_from_log_id as active6_14_4_,
poboxfacil5_.valid_from as valid3_14_4_,
poboxfacil5_.description as descript4_14_4_,
--poboxfacil5_.state as state14_4_,
poboxfacil5_.zip_onrp as zip7_14_4_,

zip6_.invalid_from as invalid2_4_5_,
zip6_.active_from_log_id as active9_4_5_,
zip6_.valid_from as valid3_4_5_,
zip6_.language1 as language4_4_5_,
zip6_.language2 as language5_4_5_,
--zip6_.state as state4_5_,
zip6_.type as type4_5_,
zip6_.zip6 as zip8_4_5_,

zipdescrip9_.invalid_from as invalid2_5_6_,
zipdescrip9_.active_from_log_id as active8_5_6_,
zipdescrip9_.valid_from as valid3_5_6_,
zipdescrip9_.description18 as descript4_5_6_,
zipdescrip9_.description27 as descript5_5_6_,
zipdescrip9_.description39 as descript6_5_6_,
zipdescrip9_.former_assignment_onrp as former9_5_6_,
--zipdescrip9_.state as state5_6_,

zipbase10_.invalid_from as invalid2_3_7_,
zipbase10_.active_from_log_id as active5_3_7_,
zipbase10_.valid_from as valid3_3_7_,
zipbase10_.base_zip_onrp as base6_3_7_,
--zipbase10_.state as state3_7_,

alternativ11_.invalid_from as invalid3_8_8_,
alternativ11_.active_from_log_id as active10_8_8_,
alternativ11_.valid_from as valid4_8_8_,
alternativ11_.description18 as descript5_8_8_,
alternativ11_.description27 as descript6_8_8_,
alternativ11_.description39 as descript7_8_8_,
alternativ11_.language as language8_8_,
--alternativ11_.state as state8_8_,
alternativ11_.former_assignment_onrp as former11_8_8_,
alternativ11_.type as type8_8_,
alternativ11_.onrp as onrp4_0__,
alternativ11_.key as key0__,

domicile8_.invalid_from as invalid2_25_9_,
domicile8_.active_from_log_id as active5_25_9_,
domicile8_.valid_from as valid3_25_9_,
domicile8_.address_information_allowed as address4_25_9_,
domicile8_.person_key as person6_25_9_,
domicile8_.pers_addr_key as pers7_25_9_,

postrestan12_.invalid_from as invalid2_26_10_,
postrestan12_.active_from_log_id as active5_26_10_,
postrestan12_.valid_from as valid3_26_10_,
--postrestan12_.state as state26_10_,

personaliz14_.invalid_from as invalid2_24_11_,
personaliz14_.active_from_log_id as active9_24_11_,
personaliz14_.valid_from as valid3_24_11_,
personaliz14_.address_add_on as address4_24_11_,
personaliz14_.co_address as co5_24_11_,
personaliz14_.deliverable as delivera6_24_11_,
personaliz14_.impers_addr_key as impers10_24_11_,
personaliz14_.mailbox_mailbox_key as mailbox11_24_11_,
personaliz14_.non_deli_foreign_addr_key as non12_24_11_,
personaliz14_.person_key as person13_24_11_,
personaliz14_.po_box_key as po14_24_11_,
personaliz14_.post_rest_key as post15_24_11_,
personaliz14_.rack_labeling as rack7_24_11_,
personaliz14_.status as status24_11_,
personaliz14_.virtual_address_key as virtual16_24_11_,
personaliz14_.zip_onrp as zip17_24_11_,

zipstreet17_.invalid_from as invalid2_6_13_,
zipstreet17_.active_from_log_id as active5_6_13_,
zipstreet17_.valid_from as valid3_6_13_,
--zipstreet17_.state as state6_13_,
zipstreet17_.street_key as street6_6_13_,
zipstreet17_.zip_onrp as zip7_6_13_,

zip19_.invalid_from as invalid2_4_14_,
zip19_.active_from_log_id as active9_4_14_,
zip19_.valid_from as valid3_4_14_,
zip19_.language1 as language4_4_14_,
zip19_.language2 as language5_4_14_,
--zip19_.state as state4_14_,
zip19_.type as type4_14_,
zip19_.zip6 as zip8_4_14_,

zipdescrip22_.invalid_from as invalid2_5_15_,
zipdescrip22_.active_from_log_id as active8_5_15_,
zipdescrip22_.valid_from as valid3_5_15_,
zipdescrip22_.description18 as descript4_5_15_,
zipdescrip22_.description27 as descript5_5_15_,
zipdescrip22_.description39 as descript6_5_15_,
zipdescrip22_.former_assignment_onrp as former9_5_15_,
--zipdescrip22_.state as state5_15_,

zipbase25_.invalid_from as invalid2_3_16_,
zipbase25_.active_from_log_id as active5_3_16_,
zipbase25_.valid_from as valid3_3_16_,
zipbase25_.base_zip_onrp as base6_3_16_,
--zipbase25_.state as state3_16_,

alternativ28_.invalid_from as invalid3_8_17_,
alternativ28_.active_from_log_id as active10_8_17_,
alternativ28_.valid_from as valid4_8_17_,
alternativ28_.description18 as descript5_8_17_,
alternativ28_.description27 as descript6_8_17_,
alternativ28_.description39 as descript7_8_17_,
alternativ28_.language as language8_17_,
--alternativ28_.state as state8_17_,
alternativ28_.former_assignment_onrp as former11_8_17_,
alternativ28_.type as type8_17_,
alternativ28_.onrp as onrp4_1__,
alternativ28_.key as key1__,

street32_.invalid_from as invalid2_9_18_,
street32_.active_from_log_id as active9_9_18_,
street32_.valid_from as valid3_9_18_,
street32_.language as language9_18_,
street32_.name as name9_18_,
street32_.official as official9_18_,
street32_.short_name as short7_9_18_,
--street32_.state as state9_18_,

alternativ35_.invalid_from as invalid2_10_19_,
alternativ35_.active_from_log_id as active8_10_19_,
alternativ35_.valid_from as valid3_10_19_,
alternativ35_.language as language10_19_,
alternativ35_.name as name10_19_,
alternativ35_.short_name as short6_10_19_,
alternativ35_.type as type10_19_,
alternativ35_.street_key as street9_9_2__,
alternativ35_.key as key2__
*/
/*,

zip30_.invalid_from as invalid2_4_20_,
zip30_.active_from_log_id as active9_4_20_,
zip30_.valid_from as valid3_4_20_,
zip30_.language1 as language4_4_20_,
zip30_.language2 as language5_4_20_,
--zip30_.state as state4_20_,
zip30_.type as type4_20_,
zip30_.zip6 as zip8_4_20_,

alternativ29_.impaddr_key as impaddr2_11_3__,
alternativ29_.onrp as onrp3__,
mailboxfac37_.invalid_from as invalid2_20_21_,
mailboxfac37_.active_from_log_id as active4_20_21_,
mailboxfac37_.valid_from as valid3_20_21_,
mailboxfac36_.impaddr_key as impaddr2_11_4__,
mailboxfac36_.mailbox_facility_key as mailbox1_4__
*/

from sf_changes
-- es muss zu jeder validity genau eine entity geben
left join kap_pers_addr_vt personaliz0_ on (personaliz0_.key = sf_changes.key)
left outer join
kap_nondel_foreign_ad_vt nondeliver1_
on personaliz0_.non_deli_foreign_addr_key=nondeliver1_.key
and nondeliver1_.valid_from <= validity and validity < nondeliver1_.invalid_from
left outer join
kap_virtual_addr_vt virtualadd2_
on personaliz0_.virtual_address_key=virtualadd2_.key
and virtualadd2_.valid_from <= validity and validity < virtualadd2_.invalid_from
left outer join
kap_po_box_vt pobox3_
on personaliz0_.po_box_key=pobox3_.key
and pobox3_.valid_from <= validity and validity < pobox3_.invalid_from
left outer join
kap_po_box_vt pobox4_
on personaliz0_.po_box_key=pobox4_.key
and pobox4_.valid_from <= validity and validity < pobox4_.invalid_from
left outer join
kap_pob_facility_vt poboxfacil5_
on pobox4_.po_box_facility_key=poboxfacil5_.key
and poboxfacil5_.valid_from <= validity and validity < poboxfacil5_.invalid_from
left outer join
kap_zip_vt zip6_
on personaliz0_.zip_onrp=zip6_.onrp
and zip6_.valid_from <= validity and validity < zip6_.invalid_from
left outer join
kap_zip_desc_vt zipdescrip9_
on zip6_.onrp=zipdescrip9_.onrp
and zipdescrip9_.valid_from <= validity and validity < zipdescrip9_.invalid_from
left outer join
kap_zip_base_vt zipbase10_
on zip6_.onrp=zipbase10_.onrp
and zipbase10_.valid_from <= validity and validity < zipbase10_.invalid_from
left outer join
kap_zip_desc_alt_vt alternativ11_
on zip6_.onrp=alternativ11_.onrp
and alternativ11_.valid_from <= validity and validity < alternativ11_.invalid_from
left outer join
kap_person_vt person7_
on personaliz0_.person_key=person7_.person_key
and person7_.valid_from <= validity and validity < person7_.invalid_from
left outer join
kap_domicile_vt domicile8_
on person7_.person_key=domicile8_.person_key
and domicile8_.valid_from <= validity and validity < domicile8_.invalid_from
left outer join
kap_post_rest_vt postrestan12_
on personaliz0_.post_rest_key=postrestan12_.key
and postrestan12_.valid_from <= validity and validity < postrestan12_.invalid_from
left outer join
kap_post_rest_vt postrestan13_
on personaliz0_.post_rest_key=postrestan13_.key
and postrestan13_.valid_from <= validity and validity < postrestan13_.invalid_from
left outer join
kap_pers_addr_vt personaliz14_
on postrestan13_.key=personaliz14_.post_rest_key
and personaliz14_.valid_from <= validity and validity < personaliz14_.invalid_from
left outer join
kap_impers_addr_vt impersonal15_
on personaliz0_.impers_addr_key=impersonal15_.impaddr_key
and impersonal15_.valid_from <= validity and validity < impersonal15_.invalid_from
left outer join
kap_zip_street_vt zipstreet17_
on impersonal15_.zip_street_key=zipstreet17_.zip_street_key
and zipstreet17_.valid_from <= validity and validity < zipstreet17_.invalid_from
left outer join
kap_zip_street_vt zipstreet18_
on impersonal15_.zip_street_key=zipstreet18_.zip_street_key
and zipstreet18_.valid_from <= validity and validity < zipstreet18_.invalid_from
left outer join
kap_zip_vt zip19_
on zipstreet18_.zip_onrp=zip19_.onrp
and zip19_.valid_from <= validity and validity < zip19_.invalid_from
left outer join
kap_zip_vt zip21_
on zipstreet18_.zip_onrp=zip21_.onrp
and zip21_.valid_from <= validity and validity < zip21_.invalid_from
left outer join
kap_zip_desc_vt zipdescrip22_
on zip21_.onrp=zipdescrip22_.onrp
and zipdescrip22_.valid_from <= validity and validity < zipdescrip22_.invalid_from
left outer join
kap_zip_base_vt zipbase25_
on zip21_.onrp=zipbase25_.onrp
and zipbase25_.valid_from <= validity and validity < zipbase25_.invalid_from
left outer join
kap_zip_desc_alt_vt alternativ28_
on zip21_.onrp=alternativ28_.onrp
and alternativ28_.valid_from <= validity and validity < alternativ28_.invalid_from
left outer join
kap_street_vt street32_
on zipstreet18_.street_key=street32_.street_key
and street32_.valid_from <= validity and validity < street32_.invalid_from
left outer join
kap_street_vt street34_
on zipstreet18_.street_key=street34_.street_key
and street34_.valid_from <= validity and validity < street34_.invalid_from
left outer join
kap_street_desc_alt_vt alternativ35_
on street34_.street_key=alternativ35_.street_key
and alternativ35_.valid_from <= validity and validity < alternativ35_.invalid_from
-- NO VT Table --> Leiche
--left outer join
-- kap_zip_imp_addr alternativ29_
-- on impersonal15_.impaddr_key=alternativ29_.impaddr_key
-- and alternativ29_.valid_from <= validity and validity < alternativ29_.invalid_from
--left outer join
-- kap_zip_vt zip30_
-- on alternativ29_.onrp=zip30_.onrp
-- and zip30_.valid_from <= validity and validity < zip30_.invalid_from
--left outer join
-- kap_mb_fac_imp_addr_vt mailboxfac36_
-- on impersonal15_.impaddr_key=mailboxfac36_.impaddr_key
-- and mailboxfac36_.valid_from <= validity and validity < mailboxfac36_.invalid_from
--left outer join
-- kap_mailb_fac_vt mailboxfac37_
-- on mailboxfac36_.mailbox_facility_key=mailboxfac37_.mailb_fac_key
-- and mailboxfac37_.valid_from <= validity and validity < mailboxfac37_.invalid_from
where
kap_search_bulkload.get_key_min <= personaliz0_.key and personaliz0_.key < kap_search_bulkload.get_key_max
;

And the method which prepares data for indexing:

public static BaseEntity<?> indexEntityPrepare(
BaseEntity<?> entity, //
Date dateValidFrom, //
Map<Number, Set<Date>> datesWithinId, //
Map<AliasType, List<GlobalBaseAlias>> globalAliases)
{
// Entity.invalidFrom bezieht sich leider nur auf die einzelne Entity selber, nicht auf sie
// und alle ihre Nachkommen (e.g. PersonalizedAddress invalidFrom berücksichtigt keine
// Änderungen an der PLZ). Daher müssen wir dateInvalidFromOfEntityTree hier selber
// berechnen.
// Warum müssen wir ValidFrom nicht selber setzen?
// sessionContextUtils.setValidityTime(date, false);
// macht das beim laden schon, das füttern wir schon beim Laden mit den Slices
Date dateInvalidFromOfEntityTree =
findInvalidFromOfEntityTree(dateValidFrom, ((TemporalBaseEntity) entity)
.getInvalidFrom(), datesWithinId.get(entity.getKey()), entity.getClass()
+ "key=" + entity.getKey());

LuceneId luceneId = setEntityLuceneId(dateValidFrom, dateInvalidFromOfEntityTree, entity);
setLuceneIdReflection(entity, luceneId);

// wir laden alle von der DB
// wir müssen aber die globalen Aliases pro Entität filtern, dass macht die jeweiligeMethode
// setGlobalAliases auf den entsprechenden Entitäten
// TODO Test
// Only Private und Business Person haben eine Implementation
setGlobalAliasReflection(globalAliases, entity);

Object[] o = {luceneId.getId(), dateValidFrom, dateInvalidFromOfEntityTree};
logger.trace("indexed id: {}, validFrom: {}, invalidFrom: {} ", o);

return entity;
}

/**
* @param globalAliases
* @param entity
*/
public static void setGlobalAliasReflection(
Map<AliasType, List<GlobalBaseAlias>> globalAliases,
BaseEntity<?> entity)
{
if (globalAliases != null && globalAliases.size() > 0)
{
try
{
Method methodSetLuceneId =
entity.getClass().getMethod("setGlobalAlias", Map.class);
methodSetLuceneId.invoke(entity, globalAliases);
}
catch (NoSuchMethodException e)
{
logger.warn("Method setGlobalAlias is not avaiable", e.getMessage());
}
catch (Exception e)
{
logger.warn("Method setGlobalAlias not properly ended", e);
}
}
}

/**
* @param entity
* @param luceneId
*/
public static void setLuceneIdReflection(BaseEntity<?> entity, LuceneId luceneId)
{
try
{
Method methodSetLuceneId = entity.getClass().getMethod("setLuceneId", LuceneId.class);
methodSetLuceneId.invoke(entity, luceneId);
}
catch (NoSuchMethodException e)
{
logger.warn("Method setLuceneId is not avaiable", e.getMessage());
}
catch (Exception e)
{
logger.warn("Method setLuceneId not properly ended", e);
}
}

public static BaseEntity<?> indexEntityPrepare(
BaseEntity<?> entity, //
Date dateValidFrom, //
Map<Number, Set<Date>> datesWithinId, //
Map<AliasType, List<GlobalBaseAlias>> globalAliases)
{
// Entity.invalidFrom bezieht sich leider nur auf die einzelne Entity selber, nicht auf sie
// und alle ihre Nachkommen (e.g. PersonalizedAddress invalidFrom berücksichtigt keine
// Änderungen an der PLZ). Daher müssen wir dateInvalidFromOfEntityTree hier selber
// berechnen.
// Warum müssen wir ValidFrom nicht selber setzen?
// sessionContextUtils.setValidityTime(date, false);
// macht das beim laden schon, das füttern wir schon beim Laden mit den Slices
Date dateInvalidFromOfEntityTree =
findInvalidFromOfEntityTree(dateValidFrom, ((TemporalBaseEntity) entity)
.getInvalidFrom(), datesWithinId.get(entity.getKey()), entity.getClass()
+ "key=" + entity.getKey());

LuceneId luceneId = setEntityLuceneId(dateValidFrom, dateInvalidFromOfEntityTree, entity);
setLuceneIdReflection(entity, luceneId);

// wir laden alle von der DB
// wir müssen aber die globalen Aliases pro Entität filtern, dass macht die jeweiligeMethode
// setGlobalAliases auf den entsprechenden Entitäten
// TODO Test
// Only Private und Business Person haben eine Implementation
setGlobalAliasReflection(globalAliases, entity);

Object[] o = {luceneId.getId(), dateValidFrom, dateInvalidFromOfEntityTree};
logger.trace("indexed id: {}, validFrom: {}, invalidFrom: {} ", o);

return entity;
}

/**
* @param date
* @param entity
* @return
*/
public static LuceneId setEntityLuceneId(Date validFrom, Date invalidFrom, BaseEntity<?> entity)
{
LuceneId luceneId = new LuceneId();
luceneId.setId(entity.getKey().toString());
luceneId.setLuceneValidFrom(validFrom);
luceneId.setLuceneInvalidFrom(invalidFrom);
return luceneId;
}

Hi,

just some thoughts, since the above code is not very helpful without any formatting.
I am still at loss what the problem is. Obviously Hibernate Search works best when you are using Hibernate itself in a fairly standard way (which seems not to be the case for you). What I mean is, that the data you are indexing is the entity data as loaded from the database.

Have you tried the flush to index approach - http://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#search-batchindex-flushtoindexes? This approach also allows from some configuration and might give you a chance to modify the data before indexing. With a little effort you can pluralize this code as well. There is also no need for reattaching anything imo.

The next question which goes through my mind - what is your actual bottle neck? It seems there is a lot of db load. If your database is the bottle neck you can improve the java code as much as you like you won't get better results.

--Hardy

Hi Hardy,

thanks for your answer.

Sorry that I did not format the code, the easiest thing would probably be a short explanation:
The current java code
1) calculates the correct invalidFrom Date of the entity tree from its nodes.
2) calculates a luceneId which is PrimaryKey and validFrom Date
3) sets global aliases
If the MassIndexer could be customized with user java code, it would be usable for us.
Any plans?

> I am still at loss what the problem is. Obviously Hibernate Search works best when you are using Hibernate itself in a fairly standard way > (which seems not to be the case for you). What I mean is, that the data you are indexing is the entity data as loaded from the database.
We do load the data from the database through the big sql mentioned above.
That big SQL can be definded a s a view in the database in order to look like a regular table to Hibernate.
However, since the view is prone to memory problems in the temp tablespace, we need to parameterize it from java
(something like sessionContextUtils.setSessionContext(minPrimaryKey, maxPrimaryKey);.
We cannot just load one table by the usual mechanisms, but we can get something close to it.

> Have you tried the flush to index approach - http://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#search-
> batchindex-flushtoindexes? This approach also allows from some configuration and might give you a chance to modify the data before
> indexing. With a little effort you can pluralize this code as well. There is also no need for reattaching anything imo.
Yes, that is exactly the approach I am using:

FullTextEntityManager fullTextEntityManager =
Search.getFullTextEntityManager(entityManager);
// TODO Stateless Session for Batching - lesen
Session session = (Session) fullTextEntityManager.getDelegate();
session.setFlushMode(FlushMode.MANUAL);
session.setCacheMode(CacheMode.IGNORE);
session.setDefaultReadOnly(true);

FullTextSession fullTextSession =
org.hibernate.search.Search.getFullTextSession(session);
fullTextSession.setFlushMode(FlushMode.MANUAL);
fullTextSession.setCacheMode(CacheMode.IGNORE);

fullTextSession.index(personalizedAddress);

fullTextSession.flushToIndexes();
fullTextSession.clear();

I noticed that the programm does not index anything without the reattch by the session.save(entity).
I currently get a resultSet (Object[]) from my view (it could also be a Hibernate Entity defined on the view).
Therefor I build the entities myself from what I read.
That is probably the reason why I need to reattach.
Do you have any better way of doing it in store?

> The next question which goes through my mind - what is your actual bottle neck? It seems there is a lot of db load. If your database is > the bottle neck you can improve the java code as much as you like you won't get better results.
I measured the times for read and session.Save: read takes 1/10 of the time of the session.save, surprisingly.
Then I tried to get faster with n parallel sessions, but the session.save does not scale linear. The Factor you get is maximal 2 or 3.

-- Eva

I don't see this anywhere in your code (the session.save).



Why not use a Pojo/Entity and let Hibernate populate it? What is your whole point of using Hibernate if you are
not using any of its actual features. If you just want an object array you might as well use plain JDBC.
And again, indexing is a data reading operation from the database point of view. There should be no need to save
anything.

--Hardy

Hi Hardy,

Quote:
I noticed that the programm does not index anything without the reattch by the session.save(entity).
I don't see this anywhere in your code (the session.save).
Answer:
I did not explicitely send it to you. It is:
session.save(personalizedAddress.getPerson());
session.save(personalizedAddress);
before the flush.

Quote:
Why not use a Pojo/Entity and let Hibernate populate it? What is your whole point of using Hibernate if you are
not using any of its actual features. If you just want an object array you might as well use plain JDBC.
And again, indexing is a data reading operation from the database point of view. There should be no need to save
anything.
Answer:
The current code is experimental to see if there is a faster way to index, so I have not defined my view Object or
a result transformer yet. So I just look some field from the result set and put them in the right entity.
I could use a Transformer to map the selected data on the entities annotated with hibernate search annotations,
but would that be faster than the reattach with the save?
Besides, my personal Experience with Transformers and a complex object trees not been positive,
it looked to me as if only simple structures could be mapped.
But I did not try that much, so I could be wrong.
Would you recommend the transformer?

Besides, how would you map the bulk load views fields?
The example from the documentation is
sess.createSQLQuery("SELECT c.ID, NAME, BIRTHDATE, DOG_ID, D_ID, D_NAME FROM CATS c, DOGS d WHERE c.DOG_ID = d.D_ID")
.addEntity("cat", Cat.class)
.addJoin("cat.dog");
I have something like
select
padresse_field1,
padresse_field2,

person_field1,
....
from view
So how can i mark my fields in the sql so that hibernate knows to which entity they go?
Do you know a good example?

-- Eva