When Jerry first encounters these "big boys -- men to [him]," they're described as being "of that coast, all of them burned smooth dark brown, and speaking a language he did not understand.  To be with them, of them, was a craving that filled his whole body."  These boys are clearly practiced swimmers, who must swim every day in order to be "burned smooth dark brown" as they are.  Jerry -- younger, foreign, less practiced -- simply wants to feel accepted by them, as if this would somehow prove to him that he is nearing adulthood too. 

At first, these boys seem to accept him.  "He dived, and they watched him, and when he swam around to take his place, they made way for him."  However, once they dive and swim through the tunnel in the rock, he begins to "panic" because he cannot keep their attention now they've turned it to the rock.  When he clowns around, "They looked down gravely, frowning."  When Jerry acted like one of them, they "proceeded to forget him," and this had the appearance of acceptance to him.  Now that he is acting like a little child, however, "They swam back to the shore without a look at him."  As long as he wasn't bothering them, they tolerated him, but when he began to annoy them, they simply left him to his tears, and "he cried himself out."