A lot of what the signalman does strikes the narrator as being a bit strange.  This strange behavior starts at the very beginning of the story.

At the start of the story, the narrator calls out to the signalman.  It is obvious that the voice must have come from up where the narrator is, but instead of looking up, the signalman looks "down the Line."  Later on, the signalman does not even move until the narrator is on him.  Soon after that, he gives the red light a really strange look.

All of this will make more sense later in the story, but for now, it seems unusual.