What is Simon referring to when he says "maybe it's only us that we're afraid of"?
This quote comes in chapter five. Simon is referring to the "beast" that has many of the boys frightened. Some of the boys, like Jack and Ralph, do not believe the monster is real. Some of the littluns do believe it is real and some even claim to have seen the monster or beast. Jack, being Jack, more or less tells everybody to not worry about it, because if there is a real beast, then he and his hunters will find it and kill it anyway.
Simon, at this meeting, says his quote about the boys only being afraid of themselves. Nobody takes his idea seriously, but it does a nice job of foreshadowing Simon's realization in chapter 8. Simon returns to his "peaceful place" only to find a sow's head impaled on a stake. Simon "hears" the head speaking to him and is told that he can never escape the Lord of the Flies because he/it is actually the potential evil that lies within the hearts of all people. That realization brings the reader right back to Simon's quote: "maybe its only us that we're afraid of."