In answering this question, one should remember that the speaker is addressing his wife at nighttime ("The sea is calm tonight"). Although the moon is shining, the lights on the French coast have gone out, casting a pall of darkness upon the beach. The loss of light inevitably means that the speaker puts more stress on sounds than images.
And what sounds! The loud, "grating roar" of the waves drawing back over the pebbles symbolizes the gradual retreat of faith in mid-Victorian-era Britain. No wonder, then, that Arnold, who laments this development, refers to the insistent, back-and-forth movement of the waves as bringing in "the eternal note of sadness." Again, in the absence of light, sounds step into the breach to convey the poem's central theme.
Ever since the time of the ancient Greek tragedian Sophocles, the sound of the waves has brought to mind the "ebb and flow / Of human misery." In the daylight, the sight of the constant movement of the waves would not have had anything like the same effect. But the sound of it in the dark conjures up all kinds of feelings, none of which are positive and all of which are foreboding. Shrouded in darkness and assailed by the "melancholy, long, withdrawing roar" of the Sea of Faith, all that the speaker and his wife can do is be true to one another.