You cannot get a collection back in that way. If doctor.getContactSet isn't good enough for you, the best you can do is Object[] { doctor, contact }.
Your observation that getting the doctor/contact pair is faster than getting the doctor then getting the set of contacts is not correct. When you get the doctor/contact pair, you are also getting the set of contacts in the doctor entity (or at least, you're getting the proxied reference to the set). If your issue is the n+1 selects that would result from the simple "from Doctor" case, use "join fetch":
Code:
select doctor
from Doctor doctor
join fetch doctor.contactdoctorSet c
where doctor.specialization in (:specialization)
This will eagerly fetch all the contacts.