The word "motley" refers to the kind of costume typically worn by jesters or fools. Wikipedia gives the following definition:
Motley refers to the traditional costume of the court jester, or the harlequin character in commedia dell'arte. The latter wears a patchwork of red, green and blue diamonds that is still a fashion motif.
Montresor says that he encountered Fortunato on the street at the height of the carnival. Almost everyone would be wearing a costume, and most would be wearing masks. Montresor himself does not identify with the Italians and is not wearing a costume. He says he is wearing a black cloak (roquelaire) and a black mask. This will make him practically invisible, or like a shadow, beside Fortunato, who could not be more conspicuously dressed and who is even wearing the traditional cap with jingling bells.
It was about dusk, one evening during the supreme madness of the carnival season, that I encountered my friend. He accosted me with excessive warmth, for he had been drinking much. The man wore motley. He had on a tight-fitting parti-striped dress and his head was surmounted by the conical cap and bells.
Montresor wants to lure Fortunato to his palazzo and down into the catacombs. Montresor's main problem is to accomplish this without being recognized by anyone. Why should Poe have made his protagonist's task even more difficult by dressing the victim in the most conspicuous possible costume? If Fortunato is glaringly conspicuous it will detract from any notice of his companion. If the authorities or other searchers begin asking questions after Fortunato's disappearance, many people will remember seeing him, but none will remember anything about a companion.
The costume is used to characterize Fortunato, not as a fool, but as a person who likes to play jokes on others, as fools traditionally did in olden times. Montresor has undoubtedly been a victim of Fortunato's "jests." The fact that it is tight-fitting is important. It shows he could not be wearing a sword. Montresor is wearing a rapier under his cloak and will have a huge advantage when he gets Fortunato down under the palazzo.
Furthermore, the niche in which Montresor intends to entomb his victim is very small.
Within the wall thus exposed by the displacing of the bones, we perceived a still interior recess, in depth about four feet, in width three, in height six or seven.
Montresor will be able to pin Fortunato tightly against the granite. If his victim had been wearing looser clothing, or a cloak, he would have had more freedom of movement and might have been able to slip out of the short chains, or at least to interfere with the wall-building.
A moment more and I had fettered him to the granite. In its surface were two iron staples, distant from each other about two feet, horizontally. From one of these depended a short chain, from the other a padlock.
Poe had to make sure that Fortunato would be fettered so tightly that he could not touch the wall that Montresor was constructing only four feet away. Even after Montresor had finished his work and departed, Fortunato would be unable to reach the wall and try to push it down while the mortar was still wet.
The fact that Fortunato was wearing a "tight-fitting" costume shows that he could not have had anything in his pockets that would have helped him in trying to pick the padlock or file at the chain.
Poe displays his genius in choosing motley for the costume of Montreso's victim.