Indeed, people often do fear what they do not understand. This is central to the trial of Tom Robinson, as seen within the chapters you noted. What Tom, a Black man in 1930s Alabama, is essentially on trial for is for "taking pity" on a poor, white woman (Mayella) and giving freely of his time to help her.
The idea that a Black man might have taken pity on a White woman seems to turn the social order of Maycomb on its ear. Likely, the jury hearing this testimony did not know what to make of such a statement in a town where the lines between Black and White are firmly drawn. Fear is apparent here.
Likewise, Tom's testimony against Mayella further contends (if one reads between the lines) that Mayella was romantically interested in Tom. Again, the idea of a White woman in love with a Black man is something fearful to the townspeople of Maycomb.
While Mayella is a far less reliable source of information, her argument that Tom attempted to rape her gels more with the comfort level of what the White residents of Maycomb "know" about their African-American counterparts.
Likely, there was fear on the part of the jurors if they were to side with Tom. What would this do to the social order of the town? How would they be perceived in the county, and perhaps even beyond? Deciding to side with Mayella and find Tom guilty helped jurors to assuage the fears the might have felt in contemplating what life would have been like had they found him innocent.