The concept of instinct versus reason comes up a few times in the story, but is specifically stated by General Zaroff when he is trying to explain his new hunting philosophy to Rainsford. Zaroff believes himself to be superior to other humans because of his strength and intelligence, and while hunting animals he discovers that he is too smart to get a thrill from the hunt.
"No animal had a chance with me any more. That is no boast; it is a mathematical certainty. The animal had nothing but his legs and his instinct. Instinct is no match for reason."
(Connell, "The Most Dangerous Game," classicreader.com)
This shows the specific philosophy behind Zaroff's actions; he thinks that since he is intelligent, and since animals are not, he is too superior to them for the hunt to be challenging. Therefore, since he moves logically from that superiority to humans, he believes it is moral to hunt humans. However, Rainsford proves to be a more instinctual prey animal than either man had thought; after Rainsford's traps -- all based on his personal hunting experience -- fail to stop Zaroff, he trusts his instincts and leaps into the ocean, not knowing if he will survive. Zaroff cannot rationalize the decision, which proves to be his undoing.