Primarily, the townsfolk of Maycomb don't like the Radleys because they keep themselves to themselves. In this small Southern town, where everyone knows each other's business, it's considered almost offensive for people to cut themselves off from society. The Radleys don't actually mean or do any harm to anyone; but for local people that's not enough. In this neck of the woods it doesn't pay to be anti-social.
The Radleys are also unpopular on account of the colorful legends that have attached to Boo over the years. This factor's related to the first; it's precisely because the Radleys keep themselves to themselves that the local townsfolk have been able to make up all kinds of weird stories about them. Essentially, they're motivated by ignorance, filling in the huge gaps about what they really know of the Radleys with legend, gossip, and folk-tales.