For one thing, Spencer, in "The Jolly Corner," is so frightened of his alter ego because it is a ghost of a life unlived. This is a degree worse than the ghost of a life once lived. At least in that case, there was a life preceding the ghost. In Spencer's case, the ghost did not have a life that existed before it. It is a manifestation of a powerful unchosen life, so powerful that it spontaneously manifested an essence, that essence being a ghost. In fact, it is the ghost of Spencer's alter ego, his unlived life, unfulfilled opportunities and unfulfilled potentials. It would seem that encountering the ghost of something that never was--the you that never was--would be extremely frightening, rather like Scrooge encountering the Ghost of Christmas Future.
Secondly, upon returning to New York, Spencer is having regrets about his choices to embrace a life in Europe and a life of ease. He regrets not having the gumption to have embraced a life of work in business, a life that would have created a self that was powerful, accomplished, determined, focused and driven toward success. He senses that he could have been so much more than his choices allowed him to become. Then, as though to prove his point, he encounters the spectre of that which he never was. This is pretty frightening: Not only is it a specter from the unreal, it is a specter from the unreal him--the specter of the life never examined, never lived, never desired.