The answer to your question is a very sad story. The author uses the word “vermin” in chapter 2 of the book Bud, Not Buddy when he writes about Mrs. Amos in reference to Bud.
Vermin are wild animals that carry disease and are generally harmful. Rats and mice are often referred to as vermin and, as such, are often exterminated. During the incident in question, Mrs. Amos is upset because she thinks that Bud has beaten her son, Todd. In reality, Bud is sleeping while Todd sticks a pencil up Bud’s nose. Being startled awake, Bud smacks Todd across the face as a knee-jerk reaction. Todd retaliates by beating Bud severely. Mrs. Amos witnesses the beating; however, she does not stop it because she sees a red mark (that matches Bud’s palm) on the side of Todd’s face. Mrs. Amos proceeds to reprimand Bud for being one of the “members of our race who do not want to be uplifted.” It is also here that Mrs. Amos uses the word “vermin” to emphasize her disdain for Bud:
I do not know if I shall ever be able to help another child in need. I do know I shall not allow vermin to attack my poor baby in his own house.
Mrs. Amos is referring to Bud as “vermin” here and is referring to Todd as her “poor baby.” As a result of this altercation, Bud is forced to spend the night in the Amos’ shed, has an encounter with a hornet’s nest, and escapes to freedom. This is where Bud’s real adventure begins.