Hi,
I hope, that I have understood seqhilo algorithm correctly. But as far as I know hibernate should query one high, than use
max_low generated identifiers and than query the new high. But in my project, the seqhilo is spamming DB for every edentifier (as if i have used classic sequence), which is really slowing down my project (inserting about 10M entities).
Probably there is some misunderstanding in how seqhilo works or in my config.
Thanks.
Edit: as I look at it...the inserts look as batch inserts neither...
(Hibernate: 3.5.4-Final, PostgreSQL)
Inserting code
Code:
do {
for (se.kb.oai.pmh.Record r : list.asList()) {
String metadata = replaceLeader(r.getMetadataAsString());
MarcXmlReader reader = new MarcXmlReader(new ByteArrayInputStream(metadata.getBytes("utf8")));
while (reader.hasNext()) { //tohle probehne pouze jednou
Record record = reader.next();
processRecord(record, r.getHeader().getIdentifier(), r.getHeader().getDatestamp());
loadedCount++;
if (loadedCount % 50 == 0) { //flush
this.setChanged();
notifyObservers(loadedCount);
System.out.println("FLUSHING");
sessionFactory.getCurrentSession().flush();
sessionFactory.getCurrentSession().clear();
}
if(loadedCount % 300 == 0){ //commit
sessionFactory.getCurrentSession().getTransaction().commit();
sessionFactory.getCurrentSession().beginTransaction();
}
}
}
if (list.getResumptionToken() != null) {
list = server.listRecords(list.getResumptionToken());
} else {
more = false;
}
}while(more);
Session factory props
Code:
<props>
<prop key="hibernate.dialect">${hibernate.dialect}</prop>
<prop key="hibernate.cache">false</prop>
<prop key="hibernate.c3p0.hibernate.c3p0.max_statements">10</prop>
<prop key="hibernate.hbm2ddl.auto">update</prop>
<prop key="hibernate.jdbc.batch_size">50</prop>
<prop key="hibernate.order_inserts">true</prop>
<prop key="hibernate.order_updates">true</prop>
<prop key="hibernate.current_session_context_class">thread</prop>
<prop key="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</prop>
<prop key="hibernate.connection.release_mode">on_close</prop>
<prop key="hibernate.transaction.auto_close_session">false</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.use_sql_comments">true</prop>
<prop key="hibernate.generate_statistics">false</prop>
</props>
AbstractBusinessObject (every entity extends this class)
Code:
@SuppressWarnings("serial")
@MappedSuperclass
public abstract class AbstractBusinessObject implements Serializable {
@Id
@GeneratedValue(generator = "hibseq")
@GenericGenerator(name = "hibseq", strategy = "seqhilo",
parameters = {
@Parameter(name = "max_lo", value = "1000"),
@Parameter(name = "sequence", value = "hibernatesequence")
})
protected Long id;
The bombardment (output of one cycle)
Code:
Hibernate: select nextval ('hibernatesequence')
Hibernate: select nextval ('hibernatesequence')
.
.
. //many more of them
.
.
.
.
Hibernate: select nextval ('hibernatesequence')
Hibernate: select nextval ('hibernatesequence')
Hibernate: select nextval ('hibernatesequence')
Hibernate: select nextval ('hibernatesequence')
Hibernate: select nextval ('hibernatesequence')
Hibernate: select nextval ('hibernatesequence')
Hibernate: select nextval ('hibernatesequence')
FLUSHING
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Metadata */ insert into Metadata (marc21Id, marc21ProviderId, oaiDate, oaiId, id) values (?, ?, ?, ?, ?)
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Metadata */ insert into Metadata (marc21Id, marc21ProviderId, oaiDate, oaiId, id) values (?, ?, ?, ?, ?)
.
.
. Many more inserts here
.
.
.
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Book */ insert into Book (metadata_id, remainderOfTitle, title, id) values (?, ?, ?, ?)
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Book */ insert into Book (metadata_id, remainderOfTitle, title, id) values (?, ?, ?, ?)
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Book */ insert into Book (metadata_id, remainderOfTitle, title, id) values (?, ?, ?, ?)
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Book */ insert into Book (metadata_id, remainderOfTitle, title, id) values (?, ?, ?, ?)
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Book */ insert into Book (metadata_id, remainderOfTitle, title, id) values (?, ?, ?, ?)
Hibernate: /* insert cz.micka.pavel.oaintk.bo.Book */ insert into Book (metadata_id, remainderOfTitle, title, id) values (?, ?, ?, ?)
.
.
. Many more inserts here
.
.
.
Login
FAQ
Search