-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 3 posts ] 
Author Message
 Post subject: Use Hibernate to Export to Flat File?
PostPosted: Tue Sep 02, 2008 2:51 pm 
Newbie

Joined: Tue Sep 02, 2008 2:45 pm
Posts: 2
I searched the forum for this but either I used bad keywords, or else maybe there isn't a known solution for this. It seems like something that others must have wanted to do, so I'm curious if there is support for this either w/in Hibernate itself or with some extension to Hibernate.

What we'd essentially like to do is export the in-memory entity objects to flat files that could be later batch imported in an out-of-band process. The entity model we have is for a quality system -- and depending on the input data, it can generate a large number of violations. Submitting these to hibernate, even after tweaking the batch processing settings still is slower than we would like. The target database is Oracle 10g.

We'd like to keep hibernate in place for the query capability, but not use it for batch inserts. Instead, we'd like to dump the entity cache to flat-files with a specified delimiter; later we can turn around and do a bulk import into Oracle.

We can write this ourselves from scratch or I was considering using HSQL-DB to dump to files, but I was wondering if anyone else has considered doing this, and what solution you settled on?


Top
 Profile  
 
 Post subject: parsing and loading to db
PostPosted: Tue Sep 02, 2008 3:19 pm 
Beginner
Beginner

Joined: Fri May 16, 2008 3:59 pm
Posts: 30
Instead of loading to a flat file, I just loaded straight to db. The db is my intermediary location. After that, I load into SalesForce.com

If your batch is slow, are u flushing the session, do you have batch enabled?

I am using hibernate to load say 10-12 files, at least 200MB per, in just under a few mins, ending with about 36,000 rows when done.

It moves pretty quickly for me.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Sep 03, 2008 3:23 pm 
Newbie

Joined: Tue Sep 02, 2008 2:45 pm
Posts: 2
Yes, flush() and clear() are both called. Batch size is set. 2nd level cache is off. Performance is pretty bad. This is the old "N INSERT statements for one-to-many associations problem". I can't find a way around this even though apparently a bug fix is in.

http://opensource.atlassian.com/projects/hibernate/browse/HHH-1

For example if you have this:

Code:
@Entity
public class Parent {
 
  @OneToMany(mappedBy="parent", cascade={CascadeType.PERSIST})
  public Collection<Child> children = new ArrayList<Child>();
}


And you do something like this:

Code:
Parent parent = new Parent();
Collection<Child> children = new ArrayList<Child>();
for(int x=0; x<1000; x++) {
  children.add(new Child());
}
parent.setChildren(children);
EntityTransaction tx = entityManager.beginTransaction();
entityManager.persist(parent);
tx.flush();
tx.clear();
tx.commit();


The SQL executes a single

INSERT INTO CHILD ...etc...

1000 times. I want it to do a bulk insert.

I'm thinking the best way out of this is to make a stored procedure and have hibernate execute that...anyone have any better ideas?


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 3 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.