In this passage the Lord speaks to Cain like a kindly father, who is concerned that Cain's anger over the fact that his offering has been rejected by God will lead to his eternal disgrace and ruin.
His advice to Cain is twofold: 1) Do what is right and you will receive the respect that you want so badly, and 2) Be the master of your own emotions instead of allowing them to be your master.
The sin, therefore, that God warns Cain against is nurturing the anger that he feels in his heart against Abel.
The best way to understand any particular verse in the Bible is to look at the context. This particular passage occurs after God accepted Abel's sacrifice (the best part of the firstborn lambs in his flock) and rejected Cain's offering (some of his crop.) Cain was angry and God called him on it:
Then the LORD said to Cain, "Why are you angry? Why is your face downcast? If you do what is right, will you not be accepted? But if you do not do what is right, sin is crouching at your door; it desires to have you, but you must master it."
God's warning evidences the attitude of Cain at his offering. Abel brought what was best with a heart of gratitude for God's provision. Cain, on the other hand, brought "some" of what he had, an indication that his gift was given out of duty rather than devotion. In 1 Samuel, the writer reminds the reader, "Man looks on the outward appearance, but the LORD looks at the heart."
The warning, then, is not toward a specific sin, but toward an attitude that makes it easy to sin. To do what is right with the right attitude leads to acceptance. To do what is right with selfish motives is a step toward doing what is wrong.