Wordsworth is thinking of the materialistic and self-centered attitudes and activities of people in the "world" that he refers to in the opening of his poem. He is making the point that people miss too much because they are obsessed with "getting and spending."
He is frustrated because people don't understand and appreciate, or even notice, the beauties and glories of nature all around them. "This Sea that bares her soul to the moon; The winds that will be howling at all hours...It moves us not." Wordsworth sees people as being blind to the wonders all around them because they are concentrating on the wrong things, the things of "the world" instead of the things created by God in "Nature".
At the end of the poem, he announces he would rather be pagan, outside of the relentless driven spirit of "the world," but capable of appreciating the natural wonders around him than to continue in the blind rush surrounding him.