The first reason that Montresor in "The Cask of Amontillado" by Edgar Allan Poe is an effective villain is that he is unambiguously evil. He does not just murder at the spur of the moment in a fit of passion, but in a cold and calculating fashion plots out his murder. The murder is particularly gruesome, in that it involves walling his victim Fortunato up in a cellar and letting him die a slow painful death there.
In the part of the narrative in which Montresor describes his method of leading Fortunato to the catacombs, he seems to take pride, and even joy, in his own duplicity. There is no sign of remorse in his account, and he seems to enjoy mocking Fortunato as he leads him to his doom.