I think that you can argue this both ways. If you take this poem at face value, it's a nice little peaceful and pastoral poem. It's restful and reassuring because it's about common things that need to be done on a farm (a typically peaceful setting). Nothing terrifying there.
But if you want to, you can argue that the speaker is talking about death and the afterlife. When he goes out for a little time, you could say he's talking about dying, sort of like how Frost might be talking about death in "After Apple Picking." If you look at the poem like this, it is terrifying because it suggests that death is lurking everywhere, even in such idyllic settings as the one seen in this poem.
Well, I guess this rather depends upon your reading of this poem. Interestingly, it was this poem that Frost himself chose to act as an introduction to his Collected Poems, and in a sense, it acts as an invitation. Note the way in which both stanzas of this simple poem end with an invitation to the reader to join Frost on his amble into nature: "You come too." However, we need to ask what the nature of this invitation is. Throughout Frost's poems, he manages to explore serious human issues and states through the setting of nature. Personally, I don't think of Frost as a "terrifying poet" in the sense that Lionel Trilling referred to, but I definitely think of him as an unsettling poet in the way that he uses everyday scenes from nature to evaluate and reassess such massive concepts as work, death, glory, remembrance, friendship, loneliness and fame. Frost's poems to me are deceptive: they appear to be about one thing, but when we understand them, they are actually commenting on something else that is quite different. "The Pasture" can be shown to be an example of this in the way that the speaker tells us what he is going to do and to see and invites us to look at the sights with him. Can we see the "raking of the leaves" as a symbol of finding hidden meanings in such natural and normal sights? I guess that is open to opinion.