We learn as the book opens that at age four, James Henry Trotter has a perfect life with his parents in a house by the sea, with many children near by to play with. However, everything changes when his parents travel to London without him. They are killed by a giant rhino that has escaped from the London zoo. The angry animal eats them up on a crowded street in broad daylight.

But as the narrator points out, while the parents' problems were over in a "jiffy," James Henry's had just begin. The house by the beach was sold, and he was sent to live with his two aunts, Aunt Sponge and Aunt Spiker. The two are "selfish and lazy and cruel." They beat James Henry and don't give him any toys or books.

The two aunts live in an isolated old house at the top of a hill. Henry James can see the woods and beautiful scenery below, as well as the ocean in the distance, but he is never allowed to leave the yard. If he does, his aunts tell him, they will lock him in the cellar with the rats. So he has to play alone in a barren yard with only a peach tree for company. He becomes sad and lonesome.