The speaker first tries to rationalize the raven's answer. The word "nevermore" seemed to answer the question the narrator had just asked, which is "what is your name"? The way the narrator explains this coincidence is to assume that the raven is repeating what a previous owner said. Here are the lines from the poem:
`Doubtless,' said I, `what it utters is its only stock and store,
Caught from some unhappy master whom unmerciful disaster
Followed fast and followed faster till his songs one burden bore
The narrator also assumes that the raven will fly away the next morning. He is wrong in both assumptions.