Color always plays a very important part in most poetry. Many poets rely on the hope that a reader knows about the implications made regarding color and why color is used in the way it is in the poem. The use of the color grey (British spelling) is no different in Walter De La Mare's "The Listeners."
The color grey has many implications. The use of the color typically illustrates balance given the fact that it fails to depict a charged emotion. More realistically, it is seen as conservative.
Other uses of the color grey denote mourning, formality, and the sophisticated. In some instances, grey is used as a lighter black--meaning moody or bad, but not too moody or bad.
Grey has also been used to denote wisdom or age (as in graying hair or a move towards death). In other circumstances, it has been used to denote mystery and knowledge when used in the eyes.
As for the implications the "grey eyes" have in the poem "The Listeners," the color could represent a few different things. First, it could represent the tiredness of the traveler. Or, it could represent the ambiguities presented in the poem by illustrating the mystery and unknown--given grey is such an ambiguous color itself.
In reality, readers can only simply assume why the poet used the color grey to describe the traveller's eyes.