I have a swing application with a Jtable that displays the details of music files, album, artist ... . The data model is held in memory and unfortunately because there are 70 different columns and users a re tyrying to load their complete music collection (e.g 100,000 files) I am having memory issues.

I am now considering backing the data with a database and using Hibernate. Do you think this is feasible or would it run too slowly, has anybody done this ?

The first issue is whether rows can be retrieved quickly enough as a user scrolls down a table , I thinking that if I use hibernate in conjunction with a database Iit caching will ome into play but will that solve the issue.

Also colums are sortable, so if a user clicked on a column it would somehow need access to every row in order to sort correctly.

Although the table column can have seventy columns by default only ten are shown, however the other sixty are still stored in the tables data model using valuable memory so I think I could grab some memory back here, but then if they decide to show the other 60 columns it could blow memory in one go.

I also considered using some kind of paging mechanism so that even if there were 100,000 rows loaded , only a 1000 were loaded into a table in one go, and you get to the bottom of the table the next 1000 are loaded, is this feasible.

thanks for any advice

In my experience hibernate is very fast considering what it does for you. I only find problems with hibernate with complicated joins and often that can be solved with a little thought about the structure of the data or even just better index's in the database itself. Obviously if you write your own sql for the database using plain jdbc you'll be able to get a performance improvement, but that performance improvement in my opinion is not worth the productivity using hibernate gives you.


You would have to sort at the database level, however hibernate offers multiple ways to do this in a database neutral way. You can look at criteria or just using hql(basically these will give you a way to define the order by clause).

.
If you are using hql you can define the columns that are returned and they will come back as a list of objects not your mapped classes. For instance:

This will return a List of Lists with a String Object for firstName and a String object for lastName.

You can also cast your items to an object for instance using a map

This will return a List of Maps where you can just call map.get("FirstName") to retrieve the first name. You can use the same method to cast other types of objects also.


I would page the data in small increments. As human being I can't process even a thousand rows at a time. If I look at list I generally scroll to a certain item, look at the top ten or look at the bottom ten in a list. Never do I process the whole list. I would say a hundred is sufficient with paging.

Yes I know its generally fast but have you any experience of trying it in the context I am talking about, backing a GUI table ?


Yes you are right, i can see thast using database sort might actually be an improvement over by existing sort mechanism.

I didn't point out any specifics because it's not really relavant whether you are filling an xml document or a JTable. If you think about it the processing time of filling the JTable or anything else for that matter is relatively the same whether using Hibernate, jdbc or a custom dao. The performance difference comes when you load the data from the database.

I have filled a table structure in swing which was based on a JTable and it worked fine.

Like I said I wouldn't be worried about any performance loss and if it's really a concern for you then measure the difference yourself. Fill the jtable through jdbc then fill it through hibernate see for yourself if you really care about the difference. After all nothing really can answer a performance question better than a real test.

Hi, Im not talking about the performance difference between SQL and Hibernate but between a database solution and a memory solution. to see if its feasible The specifics are important, for example using the simplest database solution as you scroll up and down a table and news rows come into view it would have to reget the row from the database even if it has previously got it, scrolling rapidly down the list would require many sql calls. Thanks for your help but what I really would like is confirmation from someone that is has worked with a decent size Jtable.

The in memory solution is always faster. It's just not feasible to do so. Keep in mind that even though you think you have a lot of data 100,000 records is not that much. At work we have an app that on a daily basis loads over 100,000 records. We also use hibernate to load and process this data. This 100,000 records doesn't even compare to what I am sure other people out there have done.

I don't think this is really a hibernate question nor a JTable question. Seems more like a general design problem.

It is a SPECIFIC design problem and I am trying to find out if hibernate/database can solve it for me. Of course if I just loaded the Jtable with 100,000 rows from the database at iniitlization it would perform fine, but it would use too much memory and wouldnt solve my problem.

What I am proposing is creating a Jtable with a only skeleton data stored for each row (i.e rowNo and database primary key) , but then retrieving the row data as required because it is being rendered on screen, and removing old data as not required in order to keep wihin memory limits. Now with the simplest SQL this would perform sql queries for one row at a time as needed, but I could do something cleverer to grab batches of records, hibernate may make this easier with its cache I dont know.

But this is alot of work to do so I want to find out if anybody has achieved something simailr before embarking on this.