Crane begins from the first sentence to set up the point of view of the story. It isn't exactly correct to say the point of view is limited to only what the men in the boat see and hear. The point of view also encompasses what they feel. In some places, the point of view also is shaped by what they think, such as when they debate the crew that may or may not be in a "house of refuge." This shows that their perception of the severity of their plight is shaped in part by their thoughts, not solely by their physical perceptions.
Thus it seems more accurate to say the point of view is limited to select sensory perceptions and cognitive perceptions. Another example of cognitive perceptions adding to the point of view is the incident involving the captain's head and the sea gull. The captain was relieved to see it go because of his perception of what it might do to his hair, while the crew is glad to see it go because of their perception of the encroaching sea gull as an ominous negative omen:
After it had been discouraged from the pursuit the captain breathed easier on account of his hair, and others breathed easier because the bird struck their minds at this time as being somehow grewsome and ominous.
The opening short sentence establishes the men's sight as a limiting factor to the point of view: "None of them knew the color of the sky." In the second sentence, Crane does three things: He confirms the visual point of view. He establishes the extreme tension of the circumstance of the story ("their eyes ... were fastened upon the waves ..."). He expands the point of view to suggestively include the perception of feeling by writing "the waves that swept toward them." In the next sentences, Crane reinforces the visual aspects of the point of view and dramatizes the tension and intensity of the situation:
the men knew the colors of the sea ... . waves were most wrongfully and barbarously abrupt and tall ....
Certainly the narrator's description of their experience on the waves brings their feelings into the point of view:
these problems in white water, the foam racing down from the summit of each wave, requiring a new leap, and a leap from the air.
The first suggestion of hearing being part of the point of view is when Crane's narrator describes the oiler's "thin little oar [that] seemed often ready to snap." As "snap" is an auditory experience, this introduces the aspect of sound and hearing. A significant twist to the aspect of hearing is the absence of sound during the struggle in the little dingy with the tiny oars:
There was a terrible grace in the move of the waves, and they came in silence, save for the snarling of the crests.
Later, a new sound is introduced in the breaking of the surf:
a new sound struck the ears of the men in the boat. It was the low thunder of the surf on the shore. "We'll never be able to make the light-house now," said the captain. "Swing her head a little more north, Billie," said the captain.
In this same quote, we encounter cognitive perception again ("'We'll never be able to make the lighthouse now") and the physical perception of feeling in "Swing her head a little more north, Billie."
The point-of-view in Stephen Crane's short story "The Open Boat" is established in the first paragraph. The narration is in the third-person, and the narrator (to the extent that there is one) seems able to read the minds of the four men in the small boat. This disembodied narrator never seems to leave the limited world of the men, however.
One early point for me (aside from the opening paragraph) that established that the story tells everything in terms of what can be seen and heard by the men in the boat is the comment in the first section about how each wave blocks the men's view of everything else: "As each slatey wall of water approached, it shut all else from the view of the men in the boat."
This point-of-view is consistent throughout the story. See, for example, the man on the shore (in section four) who waves his coat at the men but remains out of earshot. At this point in the story, we (as the readers) are as puzzled as the men in the boat. We're not really what the coat-waving is supposed to mean.