Notice the wording that Victor uses when describing the creature just after it has come to life. He is struck by its physical horror, which he apparently did not notice up until this point. He has chosen to patch together the body parts of multiple men in order to create his new "man," but the combination of all these beautiful parts has only made for a monstrous whole. He is so overcome by horror at the ugliness of his creature, and how frightening its appearance is, that he overlooks the fact that he has just accomplished something that no other person ever has, bringing something back from the dead. As a result, Victor considers the entire experiment a dismal failure...based solely on the creature's looks.
Sadly, the physical appearance of the creature makes such a negative first impression that Victor, the very person who put together this body to begin with, wants nothing to do with him and shuns the creature. This reaction is then shared by every other person the creature encounters. They are so frightened by his physical appearance, they do not take the opportunity to get to know him.