Zaroff invents a new kind of hunting game for the same reason that a lot kids make up new games or new rules to games.  He got bored with his previous game. 

Zaroff is an excellent hunter.  Of course the reader has to take him at his word though.  While Zaroff has heard of Rainsford's greatness as a hunter, Rainsford doesn't recognize Zaroff at all.  We assume that he is a good hunter, because Rainsford seems to believe that Zaroff is a good hunter based on all of the mounted heads in his dining room.  

"You have some wonderful heads here," said Rainsford as he ate a particularly well-cooked filet mignon. "That Cape buffalo is the largest I ever saw."

Zaroff then takes the opportunity to brag about how awesome of a hunter he is. Through his narration, the reader learns that Zaroff has grown tired of the normal game that he hunts.  

The general smiled. "No," he said. "Hunting tigers ceased to interest me some years ago. I exhausted their possibilities, you see. No thrill left in tigers, no real danger. I live for danger, Mr. Rainsford."

Zaroff then hints that he has found the most dangerous of all game, and that spikes Rainsford's interest.  Rainsford keeps probing with different animal suggestions, and finally Zaroff says this: 

"Hunting was beginning to bore me! And hunting, remember, had been my life. I have heard that in America businessmen often go to pieces when they give up the business that has been their life. . . Simply this: hunting had ceased to be what you call `a sporting proposition.' It had become too easy. I always got my quarry. Always. There is no greater bore than perfection."

In order to combat his boredom with normal hunting, Zaroff had to invent a new prey.  That new prey is humans.  Humans don't bore him, because humans have wits, cunning, and courage.  Hunting people is the ultimate challenge and ultimate high for Zaroff.  

"I wanted the ideal animal to hunt," explained the general. "So I said, 'What are the attributes of an ideal quarry?' And the answer was, of course, 'It must have courage, cunning, and, above all, it must be able to reason.'"

"But no animal can reason," objected Rainsford.

"My dear fellow," said the general, "there is one that can."

"But you can't mean--" gasped Rainsford.

"And why not?"

"I can't believe you are serious, General Zaroff. This is a grisly joke."

"Why should I not be serious? I am speaking of hunting."

"Hunting? Great Guns, General Zaroff, what you speak of is murder."

Zaroff gives a lengthy exposition explaining his reasons for "inventing" the game of hunting people instead of animals.

Zaroff has always been a hunter - it is as much a career to him as he perceives a businessman would think of his own. The problem is that Zaroff was already an excellent hunter in Russia, and the political overthrow that country experienced obliged him to leave, which simply opened him up to new hunting opportunities all over the world. However, the thrill of hunting gradually began to diminish as he came to realize that he always won - there was no animal that could match him, because he had a rifle and his wits about him. Worst of all, Zaroff came to realize that he was getting bored with hunting. He takes this as a deeply personal challenge, because if he defines his life by hunting, then being bored with hunting implies he is bored with life itself, and he has no desire to lose his mind.

Thus, Zaroff sees a fulfilling hunt as a fulfillment of his personal character; it is essentially what he was born to do. If it becomes too easy, then his purpose for existing is called into question.

Zaroff chooses humans as his new targets because they are capable of reason, which no other animal can do. He presumes that this will create a challenging hunt that will actually put his abilities to the test. He justifies this by saying that he hunts the "scum of the earth", but this is also the unfortunate reason for none of his hunts being very exciting; these low-class working types tend to lose their wits while hunted and Zaroff is disappointed that they revert to little more than animals. This is why he thinks Rainsford will be able to give him a challenge - because Rainsford is accustomed to the conditions of hunting, and might be able to put aside his terror and use his reasoning.