OR mappers are usually better than datasets for large complicated systems. But expect to have problems with binding objects to windows forms, it's not that easy as with datasets. NHibernate supports both forms of concurrency - optimistic (timestamps) and pessimistic (locking). It doesn't support multiple databases well - you can use several databases, but you can't have associations across databases.
As for bugs, there are some (see
http://jira.nhibernate.org/), we're going to fix most of them for the 0.9 release. NHibernate is already being used by many people in quite large applications.
About OO databases - the only free one I have heard of is db4o, and I don't know how well it scales. There's also an OO database by Versant, but it's quite expensive.