You can actually start with the title of the story, which alludes to the absolute chaos that lurks within the mansion. This "broken" house is "falling" in every way. Genetically speaking, the Ushers clearly sound like they are the product of generations of inbreeding, or else generations of careless mating. They are a sickly clan, vulnerable, and weak.
He suffered much from a morbid acuteness of the senses; the most insipid food was alone endurable; he could wear only garments of certain texture; the odours of all flowers were oppressive; his eyes were tortured by even a faint light; and there were but peculiar sounds, and these from stringed instruments, which did not inspire him with horror.
The Ushers' penchant for isolation also places them in a somewhat supernatural realm where they exist with the world, but not "in" the same world as everyone. It is as if "the house of Usher" is the only niche where they can actually exist...and now, it can no longer resist it: it is falling apart.
The house, as well as its resident, is described:
with an utter depression of soul which I can compare to no earthly sensation more properly than to the after-dream of the reveller upon opium--the bitter lapse into everyday life--the hideous dropping off of the veil
All of this is symbolic of the mental, spiritual, and physical state of this rare family, which (as it is debated) could even represent a fragment of the narrator's own state of mind. As "a house", the "house of Usher" really encompasses the entire generation of Ushers. They are a dying race; they are disappearing and the house is falling from its foundation.