In the poem, the speaker claims to be "half in love" with death. He thinks that death will release him from his mental torment. Indeed, he's portrayed death in a soft light in "many a mused rhyme" before, and now he thoroughly believes that he should die while the nightingale sings in ecstasy.
The speaker believes that death will release him from the "weariness, the fever, and the fret" of earthly life. Sometimes, he longs to imbibe enough alcohol to numb him to the pain of living. Being in a state of chemical oblivion would allow him to "leave the world unseen," and the nightingale to "fade away into the forest dim." The speaker is basically discouraged enough about life to hope for death.
He wants to go to a place where young people don't grow old and where men don't "sit and hear each other groan." Alas, the speaker never sees his hopes materialize. When the nightingale eventually flies away, he can't even decide whether the nightingale's music is "a vision, or a waking dream."