Holling first refers to his house as "perfect" in the first chapter, "September." According to his father, the house is perfectly in the middle of town. It's not on the north side with Jewish people, or on the south side with Catholic people. It's perfectly in the middle with the Hoodhood family, who are Presbyterian.
Later in the same chapter, Holling continues his discussion of the perfect house.
I walked back to the Perfect House slowly that afternoon. I could always tell when I got there without looking up, because the sidewalk changed. Suddenly, all the cement squares were perfectly white, and none of them had a single crack. Not one. This was also true of the cement squares of the walkway leading up to the Perfect House, which were bordered by perfectly matching azalea bushes, all the same height...The cement squares and azaleas stopped at the perfect stoop ... It was like every other house on the block, except neater, because my father had it painted perfectly white every other year...and the aluminum screen door, which gleamed dully and never, ever squeaked when you opened it.
Despite the perfection of the house, not all is perfect in the Hoodhood household. Our first hint is Holling's mother, smoking by the open window, hiding her smoking, unsuccessfully, from her family.
This relates to Holling's father because Holling's father, an architect, is intent on presenting the perfect face of a perfect family living in a perfect house to the outside world. As the story unfolds, the reader begins to learn all the ways in which Holling's home is less than the perfection Mr. Hoodhood wants everyone to think it is.