A bit of context is important first. Rainsford is running away from Zaroff. We are in the middle of the hunt. As Zaroff is making gains, Rainsford decides to hide in a tree.
At first, Rainsford believes that he is lucky. The general is close by, but he does not lift his eyes up to look higher. Were he to do this, he would have seen Rainsford. However, as Rainsford thinks about it, he has another thought, something terrifying. He thinks to himself, "Why was the general smiling?"
The conclusion is that the Zaroff actually did see him, but wanted to hunt another day. In other words, Zaroff was playing a game of cat and mouse. When Rainsford realizes this, terror truly grips him. Here is what the text says:
[Rainsford's] first thought made him feel sick and numb. The general could follow a trail through the woods at night;... he must have uncanny powers; only by the merest chance had the Cossack failed to see his quarry.
Rainsford's second thought was even more terrible. It sent a shudder of cold horror through his whole being. Why had the general smiled? Why had he turned back?
Rainsford did not want to believe what his reason told him was true, but the truth was as evident as the sun .... The general was playing with him! The general was saving him for another day's sport! ... Rainsford knew the full meaning of terror.
In conclusion, the smile of Zaroff is the smile of a madman, who likes to play with the lives of people.