TreeNode and Collection Performance

szaccaria · **Joined:** Fri Nov 16, 2007 9:27 pm **Posts:** 5

Hi to all,
I still try to create a TreeNode in memory, so serialized in db with Hibernate and so deserialized in memory with Hibernate, loading entire tree. This make do, but I've a performance problem. I attached the src of my test... Anybody have some idea to improve the performance? In my p4 notebook, the test used about 40s for create, serialized and deserialized the tree.

Thanks in advance

The node tree

Code:

@Entity
public class Nodo implements Serializable {
   private static final long serialVersionUID = 1L;   
   Long id;   
   private String valore;   
   private Nodo padre;   
   private List<Nodo> figli;
   @Id
   @GeneratedValue(strategy=GenerationType.AUTO)
   public Long getId() {
      return id;
   }
   public void setId(Long id) {
      this.id = id;
   }   
   public String getValore() {
      return valore;
   }
   public void setValore(String valore) {
      this.valore = valore;
   }
   @ManyToOne
   public Nodo getPadre() {
      return padre;
   }
   public void setPadre(Nodo padre) {
      this.padre = padre;
   }   
   @OneToMany(mappedBy="padre",cascade=CascadeType.ALL, fetch=FetchType.EAGER )
   @Fetch(value=FetchMode.JOIN)
   public List<Nodo> getFigli() {
      return figli;
   }
   public void setFigli(List<Nodo> figli) {
      this.figli = figli;
   }   
   @Transient
   public void addFiglio(Nodo nodo){
      if( figli == null)
         figli = new ArrayList<Nodo>();      
      figli.add(nodo);      
   }
}

The test

Code:

public class Test {   
   public static void main(String[] args) {      
      Session session = HibernateUtil.getSessionFactory().openSession();
      session.beginTransaction();        
        Nodo nodo = new Nodo();
        nodo.setValore("root");        
        for (int i = 0; i < 200; i++) {            
           Nodo nodo_f1 = new Nodo();             
           nodo_f1.setValore("Nodo 1o livello " + i);
           nodo_f1.setPadre(nodo);
           nodo.addFiglio(nodo_f1);
           for (int l = 0; l < 30; l++) {
              Nodo nodo_f2 = new Nodo();  
               nodo_f2.setValore("Nodo 2o livello " + i);
               nodo_f2.setPadre(nodo_f1);
               nodo_f1.addFiglio(nodo_f2);
           }                      
      }                 
        session.save(nodo);  
        session.getTransaction().commit();           
      HibernateUtil.getSessionFactory().close();      
      session = HibernateUtil.getSessionFactory().openSession();
        session.beginTransaction();              
        Nodo root = (Nodo)session.createCriteria(Nodo.class).add(Restrictions.isNull("padre")).uniqueResult(); 
        session.getTransaction().commit();      
      HibernateUtil.getSessionFactory().close();          
   }
}

And the utility

Code:

public class HibernateUtil {
    private static final SessionFactory sessionFactory;
    static {
        try {
            // Create the SessionFactory from hibernate.cfg.xml
            sessionFactory = new AnnotationConfiguration().configure().buildSessionFactory();
            } catch (Throwable ex) {
            // Make sure you log the exception, as it might be swallowed
            System.err.println("Initial SessionFactory creation failed." + ex);
            throw new ExceptionInInitializerError(ex);
        }
    }
    public static SessionFactory getSessionFactory() {
        return sessionFactory;
    }
}

thatmikewilliams · **Posted:** Sat Nov 17, 2007 12:43 pm

Change collection fetch method to subselect:

Code:

   @OneToMany(mappedBy = "padre", cascade = CascadeType.ALL, fetch = FetchType.EAGER)
---->   @Fetch(value = FetchMode.SUBSELECT)
   public List<Nodo> getFigli() {
      return figli;
   }

This should improve read performance.

szaccaria · **Joined:** Fri Nov 16, 2007 9:27 pm **Posts:** 5

thatmikewilliams wrote:

Change collection fetch method to subselect:

Code:

   @OneToMany(mappedBy = "padre", cascade = CascadeType.ALL, fetch = FetchType.EAGER)
---->   @Fetch(value = FetchMode.SUBSELECT)
   public List<Nodo> getFigli() {
      return figli;
   }

This should improve read performance.

Thanks Thatmikewilliams,
for your suggest... but this not had the hoped result, the performance not improved. I stil the same problem...
Have you some other idea, or a best solution to make a treenode?

thatmikewilliams · **Posted:** Sat Nov 17, 2007 4:50 pm

Hmmm that's interesting. I ran your code and the change I suggested took the test time down from 12s to 7s (improved read times).
Original write time was 5.4s
Original read time was 6.3s
New write time 5.5s
New read time 1.1s

There was huge reduction in the number of read sql statements issued. Perhaps you could turn on show_sql=true and verify that you see the same reduction?

With FetchMode.JOIN I see thousands of these:

Code:

Hibernate: select figli0_.padre_id as padre3_1_, figli0_.id as id1_, figli0_.id as id0_0_, figli0_.padre_id as padre3_0_0_, figli0_.valore as valore0_0_ from Nodo figli0_ where figli0_.padre_id=?

With FetchMode.SUBSELECT I only see these 4:

Code:

Hibernate: select this_.id as id0_1_, this_.padre_id as padre3_0_1_, this_.valore as valore0_1_, nodo2_.id as id0_0_, nodo2_.padre_id as padre3_0_0_, nodo2_.valore as valore0_0_ from Nodo this_ left outer join Nodo nodo2_ on this_.padre_id=nodo2_.id where this_.padre_id is null
Hibernate: select figli0_.padre_id as padre3_1_, figli0_.id as id1_, figli0_.id as id0_0_, figli0_.padre_id as padre3_0_0_, figli0_.valore as valore0_0_ from Nodo figli0_ where figli0_.padre_id=?
Hibernate: select figli0_.padre_id as padre3_1_, figli0_.id as id1_, figli0_.id as id0_0_, figli0_.padre_id as padre3_0_0_, figli0_.valore as valore0_0_ from Nodo figli0_ where figli0_.padre_id in (select figli0_.id from Nodo figli0_ where figli0_.padre_id=?)
Hibernate: select figli0_.padre_id as padre3_1_, figli0_.id as id1_, figli0_.id as id0_0_, figli0_.padre_id as padre3_0_0_, figli0_.valore as valore0_0_ from Nodo figli0_ where figli0_.padre_id in (select figli0_.id from Nodo figli0_ where figli0_.padre_id in (select figli0_.id from Nodo figli0_ where figli0_.padre_id=?))

Optimising the write side is apparently more difficult. Typically you have to tune database specific settings to get an improvement.

What are your typical use cases? E.g. how often are you creating a 6000 node graph?

Mike

szaccaria · **Joined:** Fri Nov 16, 2007 9:27 pm **Posts:** 5

Hi Mike,
I apologise because I made some mistake into use your suggest!
Now the fase of retriev has improved performance.
The problem was inside my original code ( that I don't had posted ).
I had put the 'data' field in a external object; so as you suggested me I take a look to the sql trace I see thousands of the sql rows.
I suppose that the over loading be to bind to the external object.
Have some comments about ?

The Original Node

Code:

@Entity
public class Node<T> implements Serializable {
   private static final long serialVersionUID = 1L;
   private Long id;
   public T data;
   private Node<T> parent;
   public List<Node<T>> children;

   @Id
   @GeneratedValue(strategy = GenerationType.AUTO)
   public Long getId() {
      return id;
   }
   public void setId(Long id) {
      this.id = id;
   }
   public Node() {
      super();
   }
   public Node(T data) {
      this();
      setData(data);
   }
   @OneToMany(mappedBy = "parent", targetEntity = Node.class, cascade = CascadeType.ALL, fetch = FetchType.EAGER)
   @Fetch(value = FetchMode.SUBSELECT) 
   public List<Node<T>> getChildren() {
      if (this.children == null) {
         return new ArrayList<Node<T>>();
      }
      return this.children;
   }
   public void setChildren(List<Node<T>> children) {
      this.children = children;
   }
   @Transient
   public int getNumberOfChildren() {
      if (children == null) {
         return 0;
      }
      return children.size();
   }
   @Transient
   public void addChild(Node<T> child) {
      if (children == null) {
         children = new ArrayList<Node<T>>();
      }
      children.add(child);
      child.setParent(this);
   } 
   @Transient
   public void insertChildAt(int index, Node<T> child) throws IndexOutOfBoundsException {
      if (index == getNumberOfChildren()) {         
         addChild(child);
         return;
      } else {
         children.get(index);
         children.add(index, child);
      }
   }
   @Transient
   public void removeChildAt(int index) throws IndexOutOfBoundsException {
      children.remove(index);
   }
   @OneToOne(targetEntity = Data.class, cascade = CascadeType.ALL,  fetch = FetchType.EAGER)   
   public T getData() {
      return this.data;
   }
   public void setData(T data) {
      Method metodo;
      try {
         metodo = data.getClass().getMethod("setOwner", new Class[] { this.getClass() });
         if (metodo != null)
            metodo.invoke(data, new Object[] { this });
      } catch (SecurityException e) {         
         e.printStackTrace();
      } catch (NoSuchMethodException e) {         
         e.printStackTrace();
      } catch (IllegalArgumentException e) {         
         e.printStackTrace();
      } catch (IllegalAccessException e) {         
         e.printStackTrace();
      } catch (InvocationTargetException e) {         
         e.printStackTrace();
      }
      this.data = data;
   }
   @Transient
   public String toString() {
      StringBuilder sb = new StringBuilder();
      sb.append("{").append(getData().toString()).append(",[");
      int i = 0;
      for (Node<T> e : getChildren()) {
         if (i > 0) {
            sb.append(",");
         }
         sb.append(e.getData().toString());
         i++;
      }
      sb.append("]").append("}");
      return sb.toString();
   }   
   @ManyToOne(targetEntity = Node.class,fetch=FetchType.EAGER)
   @JoinColumn(name = "pid")
   public Node<T> getParent() {
      return parent;
   }
   public void setParent(Node<T> parent) {
      this.parent = parent;
   }
}

The Data

Code:

@Entity
public class Data implements Serializable{      
   private static final long serialVersionUID = 1L;   
   private Long id;   
   private Node<Data> owner;   
   private String codice;   
   private String descri;   
   private Boolean attivaVarianti;   
   private Object obj;   
   private List<RigaH> righe;    
   public Data(){
      super();
   }   
   public Data( String codice ){
      this();
      this.codice = codice;
   }
   @Id
   @GeneratedValue(strategy = GenerationType.AUTO)
   public Long getId() {
      return id;
   }
   public void setId(Long id) {
      this.id = id;
   }
   @OneToOne(mappedBy = "data")
   public Node<Data> getOwner() {
      return owner;
   }
   public void setOwner(Node<Data> owner) {
      this.owner = owner;
   }
   public String getDescri() {
      return descri;
   }
   public void setDescri(String descri) {
      this.descri = descri;
   }
   public Boolean getAttivaVarianti() {
      return attivaVarianti;
   }   
   public void setAttivaVarianti(Boolean attivaVarianti) {
      this.attivaVarianti = attivaVarianti;
   }
   public void setAttivaVarianti(String attivaVarianti) {
      this.attivaVarianti = Boolean.parseBoolean(attivaVarianti);
   }
   public String getCodice() {
      return codice;
   }
   public void setCodice(String codice) {
      this.codice = codice;
   }
   @Transient
   public List<RigaH> getRighe() {
      return righe;
   }
   public void setRighe(List<RigaH> righe) {
      this.righe = righe;
   }   
   @Transient
   public void addRiga(RigaH riga) {
      if (righe == null) {
         righe = new ArrayList<RigaH>();
      }
      righe.add(riga);
   }
   @Transient
   public Object getObj() {
      return obj;
   }
   public void setObj(Object obj) {
      this.obj = obj;
   }   
}

Now I put direct the data inside the node and the loading of the tree is more fast!

The new Node

Code:

@Entity
public class Node implements Serializable {
   private static final long serialVersionUID = 1L;
   private Long id;
   private Node parent;
   public List<Node> children;   
   private String codice;   
   private String descri;   
   private Boolean attivaVarianti;   
   private Object obj;   
   private List<RigaH> righe;   
   @Id
   @GeneratedValue(strategy = GenerationType.AUTO)
   public Long getId() {
      return id;
   }   
   public void setId(Long id) {
      this.id = id;
   }   
   public Node() {
      super();
   }   
   public Node(String codice) {
      this();
      setCodice(codice);      
   }      
   @OneToMany(mappedBy = "parent", cascade = CascadeType.ALL, fetch = FetchType.EAGER)
   @Fetch(value = FetchMode.SUBSELECT) 
   public List<Node> getChildren() {
      if (this.children == null) {
         return new ArrayList<Node>();
      }
      return this.children;
   }   
   public void setChildren(List<Node> children) {
      this.children = children;
   }   
   @Transient
   public int getNumberOfChildren() {
      if (children == null) {
         return 0;
      }
      return children.size();
   }   
   @Transient
   public void addChild(Node child) {
      if (children == null) {
         children = new ArrayList<Node>();
      }
      children.add(child);
      child.setParent(this);
   }
   @Transient
   public void insertChildAt(int index, Node child) throws IndexOutOfBoundsException {
      if (index == getNumberOfChildren()) {
         // this is really an append
         addChild(child);
         return;
      } else {
         children.get(index); 
         children.add(index, child);
      }
   }   
   @Transient
   public void removeChildAt(int index) throws IndexOutOfBoundsException {
      children.remove(index);
   }   
   @Transient
   public String toString() {
      StringBuilder sb = new StringBuilder();
      sb.append("{").append(getCodice()).append(",[");
      int i = 0;
      for (Node e : getChildren()) {
         if (i > 0) {
            sb.append(",");
         }
         sb.append(e.getCodice().toString());
         i++;
      }
      sb.append("]").append("}");
      return sb.toString();
   }      
   public String getCodice() {
      return codice;
   }
   public void setCodice(String codice) {
      this.codice = codice;
   }
   public String getDescri() {
      return descri;
   }
   public void setDescri(String descri) {
      this.descri = descri;
   }
   public Boolean getAttivaVarianti() {
      return attivaVarianti;
   }
   public void setAttivaVarianti(Boolean attivaVarianti) {
      this.attivaVarianti = attivaVarianti;
   }
   public void setAttivaVarianti(String attivaVarianti) {
      this.attivaVarianti = Boolean.parseBoolean(attivaVarianti);
   }
   @ManyToOne
   @JoinColumn(name = "pid")
   public Node getParent() {
      return parent;
   }   
   public void setParent(Node parent) {
      this.parent = parent;
   }   
   @Transient
   public List<RigaH> getRighe() {
      return righe;
   }
   public void setRighe(List<RigaH> righe) {
      this.righe = righe;
   }   
   @Transient
   public void addRiga(RigaH riga) {
      if (righe == null) {
         righe = new ArrayList<RigaH>();
      }
      righe.add(riga);
   }
   @Transient
   public Object getObj() {
      return obj;
   }
   public void setObj(Object obj) {
      this.obj = obj;
   }   
}

Ok, you are ask me, about what to do with this tree...
I want to load a tree of many artical, with group of kinds, and show to user this...
So I use this tree only in memory and I not serializated it.
Perhaps exist a method to load in memory without use the transaction ( that i think has a cost )?

Have some comments about ?

And, of course, many thanks