There is more suffering in this world
Your question seems to be referring to another poem by Yeats, "The Stolen Child," which has the following refrain:
Come away, O human child!
To the waters and the wild
With a faery, hand in hand,
For the world's more full of weeping than you can understand.
Both "The Lake Isle of Innisfree" and "The Stolen Child" have some similarity. They suggest that escape from the real world is the best thing to do. There is a similar theme in Yeats' "Sailing to Byzantium." In that poem the speaker feels uncomfortable and alienated in his present surroundings and wishes to escape into the distance and into the past--or perhaps he only means that he wishes to escape mentally by losing himself in creative work.
It is easy for the modern urban dweller to understand and relate to these sentiments. Big cities at first seem attractive and exciting, but eventually they come to fell like expensive prisons full of lonely people who have been uprotted from nature and are leading lives of what Thoreau called "quiet desperation."