At this point in the narrative, Bruno is trying to continue his "narrative."  He wished for the friendship element that is such a part of   his yearning.  When he pursues this issue with Shmuel about perhaps coming over to that side of the fence and meeting his friends and hanging out in his world, the statement becomes uttered.  For Shmuel, being able to "share" in his world is something that makes Bruno "on the wrong side of the fence."  Bruno is of a world that will be forever and fundamentally different than Shmuel's.  Bruno speaks of a world where there is chocolate and buffet cars on trains.  Shmuel speaks of a world where there are no doors on trains and no fixed time for supper.  Shmuel speaks the idea that Bruno is on the wrong side of the fence to indicate the idea that the two will never be able to bring both worlds together simply because the fence is more than a barrier.  It is a demarcation of different lives, and different narratives.  In order for their worlds to be together, Bruno lives on the wrong side of the fence.  It is here where Shmuel's words are powerful in their articulation of life during the Holocaust.