I believe that when Rainsford calls himself a "beast at bay", he is referring to his self control. Rainsford acts like a perfectly polite, well mannered gentleman, but that doesn't mean he is always a perfectly polite, well mannered gentleman.
Think of it like a police dog. They are some of the most well trained and behaved dogs that I can think of. They pose zero danger to anybody that they come in contact with, unless their handler gives a specific command. Then the beast that was being kept at bay through training is let loose, and that dog becomes a potent threat to a "bad guy." And when that threat is eliminated, that dog goes back to being a perfectly mild mannered animal.
Rainsford is like that dog. He acts the part of a gentleman, but when he is put under certain conditions, he has no qualms about acting like a rampaging beast. Rainsford says the line at the end of the story right before he kills Zaroff. What Rainsford is saying is that he is going to revert to his beast at bay calm self just as soon as the threat that Zaroff poses is eliminated.