The formula element/attribute uses plain SQL, not HQL, so your first example won't work.
one-to-one uses primary key (id) properties, and as you can't use that, you must use many-to-one unique="true".
Are X and Y mapped? If they're just legacy tables that aren't being mapped, then you can just use a straightforward many-to-one unique="true" property-ref="xid" or whatever (though you'll need to "merge" the xid and yid columns into one hibernate property using the properties element; see section 5.1.14 of the ref docs).
If X and Y are mapped, then you don't need to model the relationship between A and B. Use queries to get corresponding items:
Code:
select a, b from A a join a.X x join x.Y y join y.B b
Obviously that query assumes that all relationships are bidirectional, but equivalents can be found if any of the relationships are unidirectional.