I don't know that I would necessarily call it a "handicap," but the problem that Silas has is that he is sort of "hunched over," if that makes sense. He was a weaver by profession and therefore spent a lot of time crouched over his loom, which left him with a crooked back. He also has kind of intense, creepy eyes, and looks older than he really is (though these aren't really "handicaps" in the true sense of the word.)
Catolepsy (the trances he falls into). This is a handicap in the sense that superstitious minds can regard him as strange or of the devil.