You need to specify a property-ref on the many-to-one and key of the set in order for it to look at something other than the PK. i.e.
passenger.hbm.xml
Code:
<many-to-one
name="car"
class="hibernatetest.car"
cascade="all"
property-ref="vin"
not-null="true"
/>
car.hbm.xmlCode:
<set
name="passengers"
lazy="false"
inverse="true"
cascade="all"
sort="unsorted"
fetch="select"
>
<key column="vin" not-null="true" property-ref="vin" inverse="true"/>
<one-to-many class="hibernatetest.passenger" />
</set>
Note also that with a bi-directional association like you've used you need to specify inverse="true" on the set.