Although it is not fact that the Pardoner has a physical defect, our Host certainly hints at it. The Host passes judgement in this line from "The Prologue": "I judge he was a gelding, or a mare." A gelding is a castrated male horse, and a mare is a fully mature female horse; therefore, the Host is accusing the Pardoner of being either a castrated male or a female! This judgement is preceded, by the way, by a description of the Pardoner's very smooth face: one that would never need a razor. Suddenly it all makes sense. What does this physical defect say about the Pardoner? It says that the Pardoner is not only vastly effeminate but also someone to be ridiculed and laughed at from the male perspective.
As for other defects (can there be anything worse than castration?), one might mention that "he had bulging eye-balls like a hare." I laugh when I think about this being possible glaucoma, but I suppose it could be. I also suppose that I would be wrong not to mention that the Pardoner also "had the same small voice a goat has got." A defect as well? Perhaps.
Luckily, the Pardoner is commonly known as the scum of the earth in that he sells pardons from the Church ("Brimful of pardons come from Rome, all hot") and doesn't have the best hygiene in the world ("hair . . . hanging down smoothly like a hank of flax"). Anyway, don't be too concerned with the Pardoner's physical defects. If anything, find the humor in them. There are many other more honorable characters for the reader to worry about in Chaucer's The Canterbury Tales.