At this point, only getting through chapter one, you should see that the kids view the Radley house as a sort of haunted house or masoleum. They believe a bogeyman-type character lives there. They are afraid of what the bogeyman-type character is capable of and they believe many legends the neighborhood has told about him.
This house signifies that which shouldn't be touched. It is an unattainable, hand-off sort of symbol for them. Although they are intrigued by the house's mystery, they have a reverence and a fear of the place that keeps them from checking it out.
"Radley pecans can kill you."
You have to remember what it is like to be a kid and make stuff up about a person or a place and believe it.