Literary criticism of As For Me and My House in both the Cyclopedia of Literary Characters and Masterplots II: British and Commonwealth Fiction Series indicates that literary critics and analysts accept that author Sinclair Ross intends us to understand that Philip Bentley did in fact have an affair with Judith. The clues that Mrs. Bentley reports, like the laughter that one night, lead her and us to the suspicion that there must have been an affair. The suspicion is confirmed when shortly afterward, Judith reveals that she is pregnant. In the era and in the town setting of the story, promiscuity was not an everyday occurrence as it is in present day America. Single women who were respected and of good standing in their towns and churches did not have multiple partners on casual terms. When Judith flees without disclosing the name of her lover, confirmed suspicions become solid conclusions: she and Philip had an affair. This means that Mr. and Mrs. Bentley adopt Philip's illegitimate child as their own and cement their marriage around his infidelity, which has a bit of a ring of a Biblical allusion to the story of Abraham and Sarah.
Based upon the structure of the novel, this question is almost impossible to answer with any degree of accuracy. The fact that Mrs. Bentley writes using a first-person narrative makes it impossible to verify the accuracy of her claims. Although she heard Judith laughing in the woodshed and witnessed her pregnancy afterwards, there is no actual proof that Phillip was involved. Instead, his changing attitude could have been a result of Mrs. Bentley's intensifying relationship with Paul.
The fact that Mrs. Bentley suspects an affair generally causes the reader to have the same impression. The guilt she sees in Judith and Phillip, however, could be what she wants to see rather than what actually occurred.
Hope this helps =)