An understanding of this aspect of Anglo-Saxon society is absolutely fundamental to appreciating the plight of the speaker in this poem. Taken out of context, the idea of being a "wanderer" doesn't sound like such a terrible thing, but it is clear that the speaker in this poem is actually in isolation. In a society focused around vassalage, kinsmen and the loyalty between a warrior and his lord, this man is in an extremely unenviable position because he is alone.
A lord would have provided many things to his loyal retainer. The speaker describes many of them here: he feels that he has nobody to whom he can confide his innermost thoughts, and he explains precisely why. Years ago, he "hid" his lord ("goldwine minne") in the "darkness of the earth." What the Wanderer is saying here is that he has outlived his lord, and that is a very terrible thing for a warrior. A warrior is expected to die in defense of his lord. A warrior who does not, who remains alive when all his kinsmen are dead, is an exile, dishonorable, without a hall to belong to or a "giver of treasure" to provide him with material wealth or emotional sustenance. The strength of the emotional bond between warrior and lord is clear in the Wanderer's remembrances of how he would "clyppe ond cysse," hold and kiss his lord and lay his head on his lord's knees. The Wanderer, in his lonely exile, seems to miss most keenly not the material wealth he enjoyed when he was part of a clan and attached to a lord, but the emotional wealth this afforded him. He is now friendless, a warrior without a lord, effectively a man without any reason to live.