The speaker of these lines is the signalman and he is speaking to the narrator. The signalman feels certain that a "dreadful calamity" is about to happen because two accidents have recently occurred on the railway line and each of these was preceded by a ghostly apparition which warned him of the events to come.
The signalman suspects that this third ghostly sighting is designed to warn him of another impending accident but he is unable to decipher the ghost's message, prompting him to call it a "cruel haunting." He desperately wants to understand the meaning, to try and prevent an accident but is unable to do so.
The signalman is understandably grieved by these recent events and the prospect of another accident on his watch. This is supported by the narrator's observation of the man:
His pain of mind was most pitiable to see. It was the mental torture of a conscientious man, oppressed beyond endurance by an unintelligible responsibility involving life.
The narrator's response is to try and help the signalman. His initial scepticism now gone, the narrator wants to prove that he has indeed been visited by a ghost. His solution is to accompany the signalman to see the "wisest medical practitioner" in the area and take his opinion.
The narrator and the signalman thus agree to meet the next evening but the meeting never takes place. In a tragic twist of events, the signalman is killed the next morning and the narrator realises that the ghost had come to warn the signalman of his own death on the line.