I think the narrator is referring, in part, to the darkness of the past, the darkness of maintaining a tradition that no longer seems relevant. The neighbor who insists on keeping the wall claims that "'Good fences make good neighbors,'" but the narrator would like to "put a notion in his head": he wants to question why good fences make good neighbors. There are no cows to prevent from wandering onto a neighbor's property, so what is it exactly that the wall is meant to keep in or keep out?
The narrator wants to question the tradition of maintaining the wall, despite the fact that he works to maintain it as well, because each man only grows trees, and the trees will not bother one another. The origins of the wall seem to have been forgotten; these neighbors just keep on maintaining it, though the one really has no idea why. The wall's origins are dark; the reasons to maintain it are dark -- no light illuminates the reasoning used by the neighbor who so wants to maintain the wall. His ideas seem antiquated, alienating, and without logic.