The main theme in "We Are Seven" is the subject of life and death. The speaker of the poem meets a rural child and asks her how many siblings she has. She claims, "Seven in all," implying that she is one of seven children. Then the child elaborates and explains that two of her siblings are dead. "Two of us in the churchyard lie," (21). This is where the theme of life and death gets more specific. The theme becomes more about the different ways children and adults interpret death.
The speaker challenges the young girl's interpretation:
If two are in the church-yard laid,
Then ye are only five. (35-36)
Here, the speaker tells the girl that, since two of her siblings are dead, she is one of only five children ("ye" refers to the girl and her living siblings). The girl counters the speaker's interpretation of death by explaining how she knits, eats dinner, and sings songs beside her dead siblings' graves, which happen to be a mere 12 steps from the door to her house.
The adult speaker is insistent that the two siblings are in heaven and therefore, the girl is no longer one of seven. The girl is equally obstinate that those two have lived on in some way.
This discrepancy between the speaker's and the young girl's interpretations of death can be interpreted in a number of ways. One could argue that this poem presents a theory that children are unable to fully grasp what death means; that is, for adults, death represents a more distinct finality. One could also argue that the young girl expresses an idea that the deceased live on in the memories of the living; it's just that the girl conflates that "place" where the memory lies with the actual place (grave) where her siblings lie.
Given Wordsworth's praise of the rural life and the innocence of youth, it is logical to suppose that Wordsworth is that adult speaker in the poem and he wants to interpret death the way the young girl does. If this is the case, then the speaker is not frustrated with the girl's view that her two dead siblings are still alive in some way. Rather, the speaker is frustrated with his own jaded, maybe even morbid, overly rational view of death as a finality.