Well, on the one hand automatically the way in which Alec is the author of Tess's downfall through seducing her and his presentation as a n'er-do-well should make us question this statement immediately. The way in which Alec is presented as a devil-like figure is clear from the conversation he has with Tess when he is holding a pitchfork towards the beginning of the novel, and his seduction of Tess is explicitly linked to the seduction of Eve by the Serpent in the Garden of Eden. It is his seduction of Tess that ruins any chance for happiness that she might gain in life, and it is this seduction to that is the means for Angel Clare abandoning her.
However, at the same time, perhaps we could argue that Angel Clare makes Tess suffer more precisely because of his goodness and the way in which he leads Tess to believe he is the opposite of Alec. This makes her depression and her feelings of pain all the more acute when he leaves her after discovering the truth about her and Alec. We expect Alec to be bad because of the way Hardy presents him as a demon-like figure, however, we do not expect the angel-like Angel Clare to be so demonic in the way that he shuns Tess and abandons her. This would suggest that Angel Clare hurts Tess more than Alec overall.