Scrooge seems to feel that Christmas is a time when people behave more irresponsibly with money than they do during the rest of the year. In the first stave, he says to his nephew, Fred, "What’s Christmas time to you but a time for paying bills without money; a time for finding yourself a year older, but not an hour richer; a time for balancing your books and having every item in ’em through a round dozen of months presented dead against you?" For Scrooge, acquiring money is the point of life, and all people do is spend it at Christmas, even when they do not really have the money to spend.
When the two gentlemen come collecting money for the poor, Scrooge tells them that his business partner, Marley, died seven years ago on "this very night": Christmas Eve. Marley was the only person Scrooge was really close to as an older adult, and we know that he didn't paint out Marley's name on the sign to their business after Marley died. When Marley died, this left Scrooge almost totally alone in the world, and since he died on Christmas Eve, this likely also makes the day a dark one for Scrooge.
When Scrooge's clerk, Bob Cratchit, expresses a desire for the day off on Christmas, Scrooge declares that it is neither "convenient" nor "fair" that he be expected to pay his employee for a day of no work. He claims that it is the same as "picking a man's pocket every twenty-fifth of December!" In other words, Scrooge feels cheated by the holiday.
In the second stave, we also learn that Scrooge had a pretty miserable childhood and was left alone at school, even on holidays like Christmas. When everyone else went home to be with family, he was abandoned and alone, with only his imagination and the fictional characters in books to keep him company. He recalls these characters—Ali Baba, Valentine and Orson, Robinson Crusoe—as vividly as if they had been real people, probably because they were all he had. Christmas would have been a terrible and blunt reminder of how alone Scrooge was while all his young classmates were home and enjoying their holiday.
Also during this stave, we learn that Belle, Scrooge's former fiancée, called off their engagement around Christmas. At this point, his face "had begun to wear the signs of care and avarice" that would show that his greedy "passion had taken root" already. After Belle leaves him, Scrooge has no one except Marley, but we know how that partnership ends as well.
In short, Christmas has always been a time when Scrooge has felt abandoned—as a child by his family, as a young man by Belle, as an older adult by Marley—so he seems only to associate the holiday with pain (while others feel pleasure) and wastefulness.