I have a question in one to many relation

in the entity class when to use Set and when to use List

and what is the difference and effect on hibernate performance

Set per definition cannot hold duplicate entries
furthermore it does not preserve the order as its elements are not indexed.

List per definition can hold duplicate entries (which for mapping relations has poor sense: associate a certain couple of objects twice over the same relation) and preserves the order as its elements are indexed.

On hibernate performance I would say that there is no significat performance difference between them (beside the fact that list needs an extra index field on schema)
On Java-level set.contains(obj) is sure much more performant as list.contains(obj) because ladder must do a sequential scan.

Hibernate documentation generally suggests Set.
In my opinion Set reflects more that what under UML is intended as relation.
Use List if you need that the order in which the elements were associated must be preserved or if you wish
to access the associated elements per index (list.getAt(index))