We could argue that there are two disparate types of childhood experience presented in Spender's poem, but the perspective of the speaker only allows us full understanding of one of those. The speaker in the poem, possibly Spender himself, is evidently under the protection of his parents, who strive to "keep [him] away from children who were rough." If the other children are "rough," then the speaker is socially superior to them, but it is evident that he fears them—their every action toward him is imbued with violence, as shown in the use of the simile, "they threw verbs like stones." This is juxtaposed with the comment that they "wore torn clothes" as if the two things aggrieved the speaker equally, being equal threats to his "world."
The behavior of these children, in some ways, sounds natural to childhood: there is a certain idyll implied in the comment that they "climbed cliffs and stripped by the country streams." This is a world outside of that of the speaker, a more sheltered child. In being separated from them, he fears them more, specifically their "hands and their knees tight on my arms" and their mockery of his "lisp." A child who is different fears the commentary of those who are not allowed to be his peers.
In the final stanza, however, there is an indication that the speaker, although he fears the other children, secretly longs to be one of them. The world of the speaker, "my world," is distinct from theirs and isolated; in the cases when they attacked him, he would "pretend . . . to smile," and he says that "I longed to forgive them, but they never smiled." The two types of children have been artificially separated by the concern of the parents, who fear how they might interact. The result, unfortunately, is that the ragged children seem to regard the speaker with as much distrust as he regards them. All of them are curious about one other, but their two worlds are never able to meet.