It is the house, which represents the slow deterioration of reality and sanity, what sets the mode for what the story tries to represent and furthermore adds the Gothic atmosphere of despair, horror, super natural, darkness, and hopelessness that Poe adds to his works as his signature style for his works.
The traditional gothic storytelling evokes feelings of coldness and terror in the reader. It is because of it that we have the "goth" movement represented as a fashion as well.
As Freud implies in his essay The Uncanny, there is an integral relationship between space and horror in Gothic literature. If one looks at the abundance of Gothic castles, Manors, Halls in the literature produced by Walpole and Radcliffe, the horror is always something that stems from the enigma of a particular place.
Poe's story The Fall of The House of Usher is a self-consciously Gothic story and though it subtly subverts the form too, the use of the House and its surroundings is definitely Gothic.
The house is like a psychic space where the paranoid games of schizophrenia (the Roderick-Madeline dyad and the narrator adding up to a trio) are played out. from the beginning, it is anthropomorphized as its 'eye-like windows' are described. The house may well be seen as a cursed place which operates the menace that seems to be lurking over the entire Usher lineage and is hell bent upon making it extinct. Both Roderick and the house are in a state of illness.
In a classic Freudian interpretation, the house has been associated with the Ushers' dead mother and seen in this way, its fall is like a collapsing of the whole structure into its inverse-image as reflected in the pool. The fall, in this light, is like a return to the desire of the dead mother where she takes her kids back through a strange spectacle of death.
The surrounding landscape is also steeped in darkness, gloom, colourlessness and a certain kind of deserted roughness, that characterizes Usher too. The decayed trees, the 'iciness' all around--all fit in. The 'fissure' that goes in a zig-zag curve down the hall is an early indication of the destruction that is waited in the final moment when the tarn, like a killing-caring mother, closes upon the remaining fragments of the house.