In Act One, Abigail and John are left alone in the room where Betty lies ill. The conversation that they have is very enlightening. Before this conversation, Miller gave us some background information on John Proctor; we learn he is a farmer in his thirties who tends to speak things as they are, and who people tend to respect because of it. He holds himself with confidence, but, deep down, as you mentioned above, he knows himself a sinner, and as a result, feels that he is a fraud.
Left alone with Abigail, the true nature of his sin is revealed. Abigail speaks warmly to him, referring to a past relationship. Apparently, they had an affair. Given the details that come out as they speak, apparently John's wife Elizabeth discovered the affair and fired Abby because of it. Abby hates Elizabeth as a result, and wonders how John can let such a "sickly wife" boss him around. Abigail is still deeply in love with John; she "waits for" him every night, hoping he'll drop by and resume the affair. However, John remains stoic; he stopped the affair months ago, and is firm about it. He insists to Abigail that "we never touched," and refuses to give in to her flirting appeals.
It is because of this affair that John feels he is a sinner; rightly so, because in his society, adultery was a forbidden act, and one looked upon with great abhorance. So, he lives his days out, feeling like a fraud, but doing the best he can to move on and make amends for what he has done. I hope those thoughts helped; good luck!