While it can seem farfetched at times, Mary Shelley's narration in the novel is critical to answering your question. The frame of the story is Walton documenting what he is told by Victor--in many cases this is not only second-hand, but third-, fourth-, or fifth-hand information. Shelley does this not to demonstrate Victor's (and the creature's) apparently photographic memory, but to show different points of view.
Victor's account of events is rather expected. A terrible creature, over whom Victor apparently has no responsbility or control, has committed all kinds of terrible crimes, including multiple murders.
Shelley also includes, however, the creature's account of events. These differ greatly from Victor's recollections and offer insight into why the creature behaves the way that he does. Whether truly justified or not, the creature believes that he is forced to commit crimes because of the way he lived his early life. This is the classic nature-versus-nurture argument: Victor believes the creature is inherently evil, while the creature sees himself as the product of a cold, unfeeling society.
Review chapter ten, in which the creature speaks to Victor for the first time, saying "All men hate the wretched; how, then, must I be hated, whom am miserable beyond all living things!" This begins the creature's justification for his actions and his attempt to place responsibility on Victor.
Is the creature a "monster" because of his actions, or is Victor a "monster" because of his irresponsibility and neglect?
There is no question that, physically, the real monster is the creature that Victor Frankenstein creates. We are told that the monster is really horrible looking -- enough so that people are repelled by his appearance. The monster is also pretty monstrous in that he kills a number of people.
But you could also argue that Victor is the monster. He is really responsible for what the monster looks like and for its very existence. He is the one who has done the original evil deed by creating this creature. He has gone beyond what humans should do and you can say this makes him a monster.