Outsiders see Esperanza's neighborhood as a dangerous place. They are frightened if they come into it. They are afraid the residents will attack them with "shiny knives." To outsiders, the shabby, run-down houses and the Latino people make the area seem dangerous
Esperanza says the only people who come into the neighborhood are the "stupid" people who get lost. She says that the neighborhood is not at all scary to the people who live in it. The residents know that the men who might look intimidating to outsiders are nobody to fear. They know, too, that with "brown all around," there is nothing to be worried about. For them, other neighborhoods are frightening.
Esperanza's point is that people tend to judge by stereotypes that may or may not bear any semblance of reality. The shabby externals of Mango Street are not the entirety of it and do not represent the souls of the residents who infuse the place with light and life. People are people when you get to know them, rich or poor. Through writing her book, Esperanza is helping us as readers to get to know her neighborhood and the people in it so that we don't have to be frightened of it as the outsiders are.