Perhaps, in his usage of "you" Frost addresses an adult audience in his poem "The Pasture"; he may be inviting his readers to recall the beauty of nature as they peruse his poem. For, as he describes his own actions, there is the almost parenthetical afterthought, "Why do you not come along, too?"
And, Frost further encourages his audience to reflect upon their childhood memories of farm life as he describes what he is going to do. Furthermore, his encouragement of "I shan't be gone long" lets his readers know that he will not take much of their time for them to engage in their happy reflections of nature such as watching the water sweep away the fallen leaves of fall and winter, and the sweet and touching innocence of the little calf that "totters" as it nurses from its mother's udder--images that will cheer the mind of the reader who is stressed by urban life.
There's not much to summarize in a poem that is only eight lines long. The speaker is going to go out and do a couple of little chores. The speaker needs to clear out a spring and "fetch" a calf.
I think that "you" in the poem refers to a child. The speaker is a parent or some other figure like that. The parent seems to be comforting the child, telling them they can come with the parent when the parent goes to do these short chores.
To me, the setting reinforces this image -- it is pastoral and comforting, with leaves, and springs, and baby cows.