Infamous essentially means to be famous for a bad reason, or to have a bad reputation.  803 Oriole Street is the home of Finsterwald, a man in the neighborhood whom no one really knows much about, but whom everyone is very afraid of.  It's kind of like a haunted house; for a long time people have been telling horrible stories about what might happen to a person if they went near the house, so now everyone believes it.  The house at 803 Oriole Street has a very bad, scary reputation - it is infamous.

No one knows for sure what will happen to a person who goes near Finsterwald's, but everyone is very afraid to do it.  The general belief is that "if some poor, raggedy, nicotine-stained wretch is seen shuffling thorugh town, word will spread that this once was a bright, happy, normal child who had the misfortune of blundering onto Finsterwald's property".  That is why everyone stays away - they don't want to be ruined by the evil forces which have become associated with the address in people's minds.

Maniac Magee, who tends to be way more sensible than the general populace in the area, is the only one who is not afraid of Finsterwald's.  When some mean older kids throw a terrified boy named Arnold Jones into Finsterwald's back yard, Maniac, who does not see why everyone is afraid, goes in and rescues the boy, who is paralyzed with fear.  Maniac then "stretch(es) out on the forbidden steps (to Finsterwald's house) and open(s) a book to read".  Everyone is amazed at Maniac's courage, and they look upon him as a hero (Chapter 5).