That is kind of a tough question to pinpoint an answer for. It really depends on how far back you are willing to look. Perhaps they "went wrong" because of a lack of forgiveness and communication about John's affair with Abigail. If you go further back than that, then it went wrong when John had the affair. If you go back further than that, then it would be the hiring of Abigail as their servant girl.
Personally, I think it is a combination of the first two. John and his wife, Elizabeth, would likely have less marital tension if John hadn't committed adultery. John feels horrible about it and his guilt eats him up. John longs for his wife's forgiveness and to be able to forgive himself. This line sums it up nicely. John says to Abigail, "But I will cut off my hand before I’ll ever reach for you again. Wipe it out of mind."
I do think that John's healing process and the restoration of John and Elizabeth's marriage could have been helped along by Elizabeth communicating her forgiveness a bit more. Elizabeth is an honest woman and a good Puritan. But she can be cold and distant. That might have driven John away in the first place, but that's not confirmed. What her cold distance now does to John though is deepen his guilt. It can be evidenced when John tells Elizabeth that her "justice would freeze beer." He's colorfully illustrating how cold she is, since alcohol has a lower freezing point than water.
In summary, John and Elizabeth did not go wrong with a single event or word. Their troubles were an accumulation of each other's actions and attitudes toward their partner.