Ebenezer Scrooge seems to be the most frightened and disturbed by the third spirit, the Ghost of Christmas Future. There are a two primary reasons for this: the spirit does not answer Scrooge's increasingly frantic questions about his own future and the spirit shows Scrooge his own tombstone.
In Stave Four the Ghost of Christmas Future shows Scrooge several upsetting scenes including Tiny Tim's death and people who are in debt to Scrooge and are relieved upon hearing he has died. Scrooge is most disturbed, however, when he asks about his own future and the spirit takes him to a churchyard and points to a tombstone.
“Before I draw nearer to that stone to which you point,” said Scrooge, “answer me one question. Are these the shadows of the things that Will be, or are they shadows of things that May be, only?”
Scrooge suspects it might be his tombstone, and that is why he asks the spirit this. But the spirit does not answer Scrooge and this causes him to become more and more frantic and upset. He is genuinely frightened by the spirit's lack of response. The other two spirits answered his questions. Even if he did not like the answers, there was some comfort in having a response. We see how agitated Scrooge becomes when the spirit does not reply.
“Spirit!” he cried, tight clutching at its robe, “hear me! I am not the man I was. I will not be the man I must have been but for this intercourse. Why show me this, if I am past all hope!”
The spirit returns Scrooge to his home without confirming whether Scrooge's changes will matter.