“The Fall of the House of Usher” is a very gothic story, full of madness and death, and fantasy masquerading as reality. By joining the House of Usher, even as a visitor, the narrator becomes a part of the insanity.
The House of Usher is full of darkness and gloom.
I know not how it was—but, with the first glimpse of the building, a sense of insufferable gloom pervaded my spirit. (p. 4)
Roderick calls the narrator “madman” because he is himself losing his mind with dread and grief. He actually put his living sister into the tomb, burying her alive.
The narrator is trying to calm an increasingly erratic Roderick at this point near the end of the story.
“Not hear it?—yes, I hear it, and have heard it. Long—long—long—many minutes, many hours, many days, have I heard it—yet I dared not—oh, pity me, miserable wretch that I am!—I dared not—I dared not speak! We have put her living in the tomb! (p. 13)
The emphasis is placed here because Roderick’s sister actuallywas put living into the tomb, and the narrator and Roderick both realize this. By spending time in the House of Usher, the narrator himself joins the unreal world. He manages to escape though, as they both die.