This sentence is best understood in the context of the paragraph in which it appears. In this paragraph, the narrator is describing his behavior in order to argue that he is not a madman. He begins the paragraph by saying,
Now this is the point. You fancy me mad. Madmen know nothing.
The speaker then contrasts himself to how he believes a madman behaves. He does so by demonstrating, step by step, how cautious he was in entering the old man's room. Part of that caution is expressed by the sentence in question:
And every night, about midnight, I turned the latch of his door and opened it—oh, so gently!
What is significant about this sentence is that, while it does show him acting cautiously, it is also the description of a creepy and repetitive act that suggests the opposite of what the speaker wants us to believe. Who but a disturbed or obsessive individual would stalk an old man at midnight every night by silently opening his door and sneaking into his room? The caution is not the point: the creepy behavior is.
The sentence is ironic, meaning the opposite of what the speaker intends: it shows he is disturbed, not sane. Further, it builds suspense, as we can only wonder, with increasing agitation, why the speaker feels he has to enter this room. Finally, the exclamation point indicates that the speaker experiences an emotional high from his aberrant behavior, more evidence that he is disturbed.