While Scrooge is probably most frightened by the images of what will happen in the future by the third spirit, the spirit who actually terrifies his the most is arguably the spirit of his dead partner, Marley.
When Scrooge returns home to his "gloomy suite of rooms," he sees not the knocker of his door, but the face of Marley. Its livid color creates a sense of horror in Scrooge, but he is not so easily terrified. However, as he sits before his meager fire, Scrooge hears a clanking sound from below the floor where he sits; it is as though someone were dragging heavy chains over the casks in the wine cellar.
"It's humbug still!" said Scrooge. "I won't believe it."
But, when an apparition comes through his heavy door and moves into the room in which Scrooge stands. Scrooge blanches--"His colour changed though"--when the ghost of Marley passes through the door.
Upon its coming in, the dying flame leaped up, as though it cried, "I know him; Marley's Ghost!" and fell again.
Scrooge interrogates it in an effort to "keep down his terror," and tells the specter that he does believe in it. However, Scrooge trembles when Marley's ghost tells him that his chain was forged by himself; further, Marley's ghost informs Scrooge of the "ponderous chain" Scrooge has forged himself.
Scrooge tries to be sarcastic, but
[T]he truth is, that he tried to be smart, as a means of distracting his own attention, and keeping down his terror; for the spectre's voice disturbed the very marrow in his bones.
Clearly, Scrooge finds Marley's ghost, the spirit of his old partner, the most frightening because no other spirit disturbs to the extent that this one does.