This is a great question. Clearly the answer lies somewhere in the middle. However, if I had to choose only one, I would say that reason is what wins out in the end. This is intimated in the middle of the story and confirmed at the end of the story.
In a conversation between Rainsford and General Zaroff, Zaroff says that he does not find delight in hunting animals. Animals are too easy to hunt. So, Zaroff hunts humans. Humans present a greater challenge, because they can reason. Here is what Zaroff says:
"Precisely," said the general. "That is why I use them. It gives me pleasure. They can reason, after a fashion. So they are dangerous."
The assumption here is that reason is more important than instinct.
Towards the end of the story, Rainsford comes to the same conclusion. He decides to attack Zaroff. He comes to this conclusion through reason. The text says:
Rainsford knew he could do one of two things. He could stay where he was and wait. That was suicide. He could flee. That was postponing the inevitable. For a moment he stood there, thinking. An idea that held a wild chance came to him, and, tightening his belt, he headed away from the swamp.
This is when the story changes, and Rainsford is waiting for Zaroff at his house. Reason made this decision not instinct.