This is a great question. Rainsford asks to be excused from seeing the heads of game in the library, because Rainsford is disgusted with the conversation that he is having with general Zaroff. Some context is important.
At first, Rainsford is pleasently surprised. Not only does he find a congenial man in Zaroff, but Zaroff is also cultured. Zaroff even read Rainsford's books on hunting. In view of this, Rainsford might have even been flattered.
As the conversation continues, general Zaroff says that he created a new type of animal to hunt. Rainsford is shocked to hear this. When Rainsford realizes that this new animal is man, he is disgusted, as this is not only barbaric, but also murder.
Moreover to the point, the heads that are hanging in the library are probably human heads. The text does not explicitly say this, but it is strongly suggested by the context. In light of this, we cannot blame Rainsford for not wanting to go into the library.