We have to distinguish between "unstable" psychology, in which the subject's mental state can change from time to time, and "pathological" psychology: in this case, a mental state in which Montresor feels driven to torture Fortunato.
It seems that Montresor is stable enough, since he is still here and by implication at liberty 50 years later.
But it's apparent that he is subject to bizarre thinking, in which he becomes obsessed by anger at what he perceives are "the thousand insults of Fortunato". He also seems to have what used to be called psychopathic or sociopathic personality disorder, where he lacks empathy and any sense that another person has some right not to be harmed. He seems to regard Fortunato not as a "real" person, but rather as an annoying bug that he can punish and destroy.
Montresor is obviously able to think well enough to plan his ambush well in advance, and protect himself from retribution.
He also needs to preserve his superiority over Fortunato: Montresor's family "was great and numerous", and their arms showed a foot crushing a snake; the motto means essentially "nobody disses me". But Fortunato's name signifies "lucky", and I guess that the thousand insults were occasions when Fortunato outshone Montresor.
Montresor's bizarre thinking magnifies Fortunato's successes into mortal insults, and he convinces himself that he has to eradicate Fortunato from the Earth.
After the tomb is walled up Montresor feels sick at heart, but he is able to dismiss his human reaction to the death as something trivial induced by the dampness of the catacombs.
However, I was left with the impression he had not subsequently killed anybody else; possibly he could not accept his actions, and never dared to repeat them.