All of the prisoners clearly have spent some time together since their capture, and during that time they have come to associate Billy as somebody who is unable to sleep peacefully. When Billy tries to go to sleep, the person next to him suspects that it is Billy Pilgrim that is lying next to him and insists that he moves somewhere else. At this stage, all of the POWs refuse to have Billy Pilgrim sleep next to them:
And now there was an acrimonious madrigal, with parts sung in all quarters of the car. Nearly everybody, seemingly, had an atrocity story of something Billy Pilgrim had done to him in his sleep.
This of course strongly suggests that what Billy Pilgrim experiences as jumps in time are actually no such thing; that what Billy actually does is fall asleep and during that sleep has intense dreams of going through time and being abducted by aliens. His extreme physical actions during such dreams, which clearly disturb the men around him, highlight this possibility and the element of uncertainty about the veracity of the perception of Billy's reality.