What Simon knows, and the rest have yet to discover, is the magnitude of Jack and his gang's savagery. When he says "...maybe it's only us that we're afraid of” he means that maybe we know that there isn’t a beast, but it’s easier to fear the beast than it is to face the reality that we’re actually afraid of each other.
Simon is trying to convince the boys that they do not need to fear the beast. Simon knows that Jack created the beast to make the members of his gang fearful. By instilling fear, Jack tries to make himself out to be a better leader than Ralph by offering his protection from the beast.
When he says, "Maybe it's only us," Simon understands the concept of psychological projection, or "blame shifting," as it is commonly called.
This observation of Simon's goes to the heart of William Golding's allegory of human nature. Simon, who is intuitive and sensitive to the spirit of others, eventually recognizes the innate bestial (i.e. brutal or savage) qualities that lie within the human heart. For instance, in Chapter Four, he has observed Jack's beastly cruelty to Piggy as Jack punches Piggy in the stomach and breaks his glasses as Piggy is hit in the head.
In Chapter Five Ralph calls a meeting and outlines the important things that must be done. But, his attempts to restore order are disrupted by talk of the beast. One little boy named Percival suggests that the beast might come out of the sea because his father has told him that there could be creatures in the ocean. As the boys argue among themselves about the beast, Simon makes an effort to explain that they are trying to objectify what is actually something in themselves; they are "blame shifting." However, Simon stumbles,
"What I mean is...maybe it's only us....We could be sort of..." [Penguin edition does not contain "that we're afraid of"]
Simon tries to suggest that the beast may be something within themselves, but he
...became inarticulate in his effort to express mankind's essential illness.
When he, then, tries to give a comparative example, this, too, fails. It is not until later in the novel when Simon confronts the Lord of the Flies that he finally acquires the capacity to articulate the evil inherent in humans.
Essentially, Simon is referring to the fact that the real danger on the island is not the so called "beastie" but the innate evil that lies within the boys themselves. While the boys excuse their actions by blaming their fear on something imaginary, the real danger on the island is the boy's potential to be evil. Simon, the most spiritual and odd of the boys, gets this, while the other boys really don't.