Setting, as your question identifies, is absolutely crucial to this excellent short story, as the setting helps create the foreboding atmosphere of imminent doom and terror that pervades the entire text. Note how even in the first paragraph, as the narrator comes upon the House of Usher, the mood of the setting is clearly established:
I had been passing alone, on horseback, through a singularly dreary tract of country; and at length found myself, as the shades of the evening drew on, within view of the melancholy House of Usher. I know not how it was - but, with the first glimpse of the building, a sense of insufferable gloom pervaded my spirit... I looked upon the scene before me - upon the mere house, and the simple landscape features of the domain - upon the bleak walls - upon the vacant eye-like windows - upon a few rank sedges - and upon a few white trunks of decayed trees - with an utter depression of soul which I can compare to no earthly sensation more properly than to the after-dream of the reveller upon opium - the bitter lapse into everyday life - the hideous dropping off of the veil.
This description continues, establishing the profound and horrific impact that seeing the House of Usher has upon the narrator. Note how specific words of description such as "melancholy," "mere," "vacant," "rank," and "decayed" help reinforce this atmosphere of a stagnant setting that strikes terror into the narrator's soul.
Absolutely vital to note is how this feeling of horror is repeated as the narrator sees his old friend again and then sees his sister. There is evidently a link between the heirs of the House of Usher and the house itself, that joins them in their fate. Note how Roderick and Madeline and the House itself all perish together, indicating that the curse that is upon the House of Usher will bring it down completely. It is this link that is influenced by the setting, as the effect of the House on the narrator is mirrored by his meeting with the actual heirs of the House.