At the beginning of Chapter 17 in Harper Lee's To Kill a Mockingbird, Scout describes for us the prosecutor, Mr. Gilmer, and how it is that he makes people pay attention to him.
As Scout describes, Mr. Gilmer has a "slight cast in one of his eyes." The word cast when used to describe eyes can refer to the direction an eye looks or to a particular expression in the eye. The word can also refer to the eye looking like it is squinting (American Heritage Dictionary, 5th ed.). All in all, it appears that Scout is describing Mr. Gilmer as having a slightly lazy eye. A lazy eye looks in a fixed direction, even if the observer is not really looking in that direction. On top of having a lazy eye, it appears that Scout is also describing Mr. Gilmer's eye as having a slight squint to it so that it always looks deadly serious.
As Scout further narrates, Mr. Gilmer uses his squinting lazy eye to "his advantage" because, as she phrases it, "[H]e seemed to be looking at a person when he was actually doing nothing of the kind, thus he was hell on juries and witnesses." In other words, since juries and witnesses always felt like Mr. Gilmer was looking at them closely and seriously, even though he wasn't really looking in their direction, they felt like they were "under close scrutiny," meaning that they were being closely watched and examined. The feeling of being always scrutinized would be "hell" on a jury member because he or she would fear the consequences of looking as if he or she had not been paying attention to a vital point. It would be "hell" on a witness because he or she could fear that Mr. Gilmer is recognizing or suspecting lies.