Mary Shelley's Frankenstein is a multiple narrative (meaning readers "hear" the tale from three different perspectives: Walton's, Victor's, and the Creature's). Essentially, by limiting the narrative voice to only one, Victor, readers would lose much of the back story (mainly Walton's) and the Creature's moving and emotionally charged tale.
The novel would change most dramatically if the narrative voice only belonged to Victor because readers would fail to hear the true sorrow and desire found on the voice of the Creature. It is through the Creature's narrative that allows readers to feel the utter sadness felt by the Creature. Without this narrative voice, many readers may fail to feel the sympathy the Creature requires in order to identify him as a sympathetic being.
Also, given that Victor tends to act as both judge and jury throughout the novel (seen through the thematic relevance of justice and injustice), Victor's skewed sense of morality would be further skewed. Readers would most likely fail to see Victor's lack of honest judgement. Instead, they would see him to be making the best judgement for himself alone, not ones which would benefit those around him.
Finally, readers would not find true closure. Given that the Creature shows up after Victor dies, readers would not know what happens to the Creature or "see" him apologize to Victor. This, for some, proves to be one of the most important examples of the Creature's desire to be honestly accepted by Victor. Instead of offering this last final emotionally charged dialogue, the novel would simply end at Victor's death, offering the reader no true closure.