The most frightening Christmas spirit to Scrooge was the Ghost of Christmas Yet to Come. This spirit showed Scrooge a future in which he was alone, unloved, frightened, and unfulfilled in life. The spirit showed him a world in which all of Scrooge's loved ones and acquaintances had abandoned him. Indeed, it seemed to Scrooge that no one in this future world even acknowledged his existence. Scrooge found this future surprising and depressing. Even after being visited by the first two spirits, he had not yet come to realize the grave effects of miserliness and bad temperament on his own future. The Ghost of Christmas Yet to Come forced Scrooge to deal with the frightening consequences of his behavior.
While Scrooge is probably most frightened by the images of what will happen in the future by the third spirit, the spirit who actually terrifies his the most is arguably the spirit of his dead partner, Marley.
When Scrooge returns home to his "gloomy suite of rooms," he sees not the knocker of his door, but the face of Marley. Its livid color creates a sense of horror in Scrooge, but he is not so easily terrified. However, as he sits before his meager fire, Scrooge hears a clanking sound from below the floor where he sits; it is as though someone were dragging heavy chains over the casks in the wine cellar.
"It's humbug still!" said Scrooge. "I won't believe it."
But, when an apparition comes through his heavy door and moves into the room in which Scrooge stands. Scrooge blanches--"His colour changed though"--when the ghost of Marley passes through the door.
Upon its coming in, the dying flame leaped up, as though it cried, "I know him; Marley's Ghost!" and fell again.
Scrooge interrogates it in an effort to "keep down his terror," and tells the specter that he does believe in it. However, Scrooge trembles when Marley's ghost tells him that his chain was forged by himself; further, Marley's ghost informs Scrooge of the "ponderous chain" Scrooge has forged himself.
Scrooge tries to be sarcastic, but
[T]he truth is, that he tried to be smart, as a means of distracting his own attention, and keeping down his terror; for the spectre's voice disturbed the very marrow in his bones.
Clearly, Scrooge finds Marley's ghost, the spirit of his old partner, the most frightening because no other spirit disturbs to the extent that this one does.