If we assume the speaker and the poet to be one and the same, then we can infer that the poet feels grief-stricken and hopeless. The recurring image throughout the poem, described by the speaker, is of the waves of the sea as they "Break, break, break" upon the "cold gray stones ... At the foot of crags." This image connotes hopelessness, because the waves each time come to nothing and are seemingly defeated by the lifeless rocks upon which they crash. The image serves as a metaphor for the speaker's feelings of love for the loved one he has lost. His love, now that his loved one is gone, has come to nothing, much like the waves crashing against the rocks. Indeed, at the end of the poem, the speaker laments that "the touch of a vanish'd hand ... Will never come back" to him.
In the second stanza, the speaker contrasts his own feelings with those of "the fisherman's boy," who "shouts with his sister at play," and "the sailor lad," who "sings in his boat on the bay." The fact that the children are playing and the sailor is singing suggests that all three feel happy and carefree. Although the speaker longs to feel as they do, their feelings in fact emphasize, by contrast, his own feelings of melancholy and despair.