So, your problem is about memory usage; not performance !?!
tj wrote:
it grows by many Megabytes for only a small object.
Are you sure that this megabytes are used by your "small object" ?
The creation of the Configuration and the SessionFactory can take a good amount of memory; and don't forget that .NET takes far more memory than native applications (at the loading)
Try to load 100 objects then 1000 and finally 10000; the memory usage should grow (almost) like native applications (that is sizeof(YourObject) * Number of instances)