Shelley makes a point of showing that Victor grew up in the bosom of a loving home, with parents who cared deeply about him. He had friends and a fiancee. He was physically attractive and surrounded by affection. He had the opportunity go to a university and pursue his passion for science.
In stark contrast, the monster is rejected by his "parent," Victor, who recoils from him in horror and runs off, abandoning his "child." This rejection repeats itself in the monster's life. The monster is on his own in terms of obtaining food, shelter, resources, and knowledge. He learns by spying on a loving family. Nobody offers him love, support, or kindness. Even the kindest people treat him as a horror.
The monster was wrong to kill the people Victor loved, but Victor ran away from his responsibility to care for and nurture the being he had created. This is all the more shocking considering how Victor expresses his appreciation of his own loving family.