Simon does not escape to his private little hideaway (the bower to which you refer) in chapter four; however, he does do this at the very end of chapter three of Lord of the Flies by William Golding.
Chapter three centers around the obvious conflict in thinking between Ralph and Jack. One of them, Ralph, is interested in keeping a signal fire lit and keeping everyone as safe as possible until they are rescued from the island. Jack, on the other hand, is really only concerned with himself and his blood lust for hunting. Simon has been helping Ralph work on the huts all day, but as soon as he can escape from the work and the conflict he sneaks away.
While we do not know how often Simon sneaks off, we know he is familiar with his hidden bower and we know he sometimes goes there when he assumes the boys are all sleeping (because one of the littluns has seen someone and Simon has to confess it was him so no one would think it was some kind of beast).
What is clear is that Simon is not well suited to endure the conflict which is happening more and more often on the island. In fact, in chapter four the tensions intensify when Jack allows the signal fire to go out and the boys miss an opportunity to be rescued. Golding tells us twice in chapter four that Simon is uncomfortable with the arguing and fighting between Jack and Ralph.
Simon looked now, from Ralph to Jack, as he had looked from Ralph to the horizon, and what he saw seemed to make him afraid.
Passions beat about Simon on the mountain-top with awful wings.
At the end of chapter three, Simon retreats to his private world because he just seems to need time to meditate and think--or even just be--in a place where there is no confusion or conflict.
Simon paused. He looked over his shoulder as Jack had done at the close ways behind him and glanced swiftly round to conﬁrm that he was utterly alone. For a moment his movements were almost furtive. Then he bent down and wormed his way into the center of the mat.
Once he drops the "screen of leaves into place," he is cut off from the outside world, surrounded only by the aromatic smells and the peaceful sounds of the jungle.
We know that Simon acts as a kind of Christ figure in this novel, so it is not surprising that he must find time, as Jesus did, to be alone and away from the crowds for a time throughout his ministry. This secret place takes on much greater significance as the novel progresses.