Recursive entity relationships & out of memory errors

loester · **Joined:** Tue May 08, 2007 10:55 am **Posts:** 5

The following problem concerns a web crawler.

Let's say you have the entity class "Page" to represent a Page downloaded from the internet:

Code:

@Entity
class Page
{
int id;
String pageUrl;
String pageText; //contains the whole page contents

@OneToMany(cascade = CascadeType.ALL)
List<Page> children;
} 

As you can see, a web page can link to other web pages, which are stored as the "children". Now the problem is, children can also have children ad infinitum . Currently what I do is:

Page rootPage = new Page("http://www.msn.com");
entityManager.persist(rootPage);
rootPage = crawl(rootPage);

During the crawl memory usage goes up rapidly, until and "out of memory" exception is thrown.

Any recommendations on how to limit memory usage?

waswani · **Posted:** Sun Apr 20, 2008 11:49 pm

try to make use of hibernate.jdbc.batch_size property

OR

make use of session.flush() and session.clear() at regular intervals.

loester · **Joined:** Tue May 08, 2007 10:55 am **Posts:** 5

Thanks, but the real problem is that the rootPage is growing without bounds.

The idea behind my code is that I call the entitymanager once with entityManager.persist(rootPage), and then the function crawl(rootPage) executes the statement rootPage.setChildren(List<Page> childPages) and for each child also executes child.setChildren, and so on infinitely & recursively.

Each time setChildren is called, Hibernate adds the children to the database table Page and links them to the parent in the same table...