The answer to this question can be found in Chapter Six. In this chapter, that focuses on the character of Pearl for the first time in the novel, she is explicitly contrasted with the other children of the town who, unlike her, come from "good" Puritan families. Note the following quote that explores the differences between Pearl and these children:
Pearl was born outcast of the infantile world. An imp of evil, emblem and product of sin, she had no right among christened infants. Nothing was more remarkable than the instinct, as it seemed, with which the child comprehended her loneliness; the destiny that had drawn an inviolable circle round about her; the whole peculiarity, in short, of her position in respect to other children.
Pearl therefore is marked by her birth as being tainted by the same evil that is associated with the scarlet letter that her mother so proudly wears on her breast. It is this sense of being different and profoundly separate that keeps her detached from the other children and causes her to not engage in the same games and play that they indulge in.