It seems that the suitors wish to avoid going to the house belonging to Penelope's father, Icarius, because they do not want him to be the one to choose which of them she would marry.  (They assume she is a widow since her husband, Odysseus, has been gone for twenty years and missing for almost ten years since the end of the Trojan War).  Telemachus says,

"Suitors plague my mother — against her will —
sons of the very men who are your finest here!
They'd sooner die than approach her father's house
so Icarius himself might see to his daughter's bridal,
hand her to whom he likes, whoever meets his fancy."

The suitors do not want Penelope's father to be in charge of the proceedings because then he will choose her husband from among them, not her.  Perhaps they each feel that they will be unable to impress him, and if he denies them her hand, they will have little recourse.  Further, the suitors have it really good at Odysseus's house: Penelope is powerless to affect their behavior, and Telemachus is too because he is so young.  They can basically drink up all Odysseus's wine, eat up all his livestock, and so forth, and such a happy situation is not likely to meet them at Icarius's house.