In line 20 the poet identifies the saddest word that the ear of man has ever heard. That word is "alone." Indeed, line 19 says it is a "bitter and dreary word." "Bitter" implies not only that it is not sweet, but also that it leaves one hurt or resentful because of the way one has been treated. It is a dreary word because it is depressing and dull; when one is alone, life is boring and it is hard to have hope for the future.
The entire poem builds up the sadness of the word "alone" even before it reaches line 20. The end of each stanza is the doleful refrain, "Alone, alone, ah woe! alone!" The repetition of the word within the line and the repetition of the line at the end of each stanza reinforces the ongoing, relentless sadness of the state of loneliness. The poet uses assonance, repetition of vowel sounds in adjacent words, to further emphasize the sadness of the word. The long /o/ sound in the words "alone" and "woe" is the sound of someone wailing or moaning, a sound of sadness, hopelessness, and sorrow. The word "woe" means great sorrow or distress; its repetition and placement next to the word "alone" enhances the sadness of the word. The poem doesn't just state that "alone" is the saddest word; it allows the reader to experience that feeling through the poetic devices it employs.