my mother partook of our enjoyments [...]; the voice of command was never heard amongst us; but mutual affection engaged us all to comply with and obey the slightest desire of each other.

There seems to have such a general sense of accord and love that no one fought, everyone read what they loved, and all got along without incident.  

On the other hand, it seems that Victor's imagination really did require more direction than he received. Although he was "calm and philosophical [...,] [his] temper was not so yielding" and he felt keenly a desire to "discover" the world's secrets. He began to study the work of writers like Cornelius Agrippa (writers whose work focused on the fantastic, not the realistic at all). When he told his father what he'd been reading, rather than instructing him further, his father simply told him, "do not waste your time upon this; it is sad trash."  Now, as an adult, Victor says, 

I cannot help remarking here the many opportunities instructors possess of directing the attention of their pupils to useful knowledge, which they utterly neglect [....].  If, instead of this remark, my father had taken the pains to explain to me, that the principles of Agrippa had been entirely exploded [....,] It is even possible that the train of my ideas would never have received the fatal impulse that led to my ruin.

Victor censures his father for his lack of education, and he actually blames his father for not curbing the impulses that eventually lead to his own catastrophic mistakes, ruin, and death. Thus, even though his childhood appeared ideal in many ways, it becomes obvious that adult Victor sees it as a liability rather than a blessing.

Victor Frankenstein sees his childhood as a happy one. He remembers how his parents doted on him and loved him dearly. His parents were very well off and came from a long line of distinguished people. They are very wealthy and have no problem showering their son with love and affection. From an early age, Victor saw his mother caring for the less fortunate. 

Victor remembers that he had a happy childhood and said his parents were possessed by the spirit of kindness and indulgence. When he was with his friends and saw the way their parents acted, he felt fortunate and grateful to have the parents he did. Victor admits that he had a bad temper. He also acknowledges that he had a thirst for learning. He didn't want to learn about the usual things, though, he wanted to learn about heaven and earth. His mother dies when he is young and he is left to be raised by the servants of the family. This is a precursor that leads to his break with reality in the future. He loved his parents dearly and they gave him unconditional love, but with the death of his mother, his lost wasn't just for the person but also for the love he always felt from her.

"My mother's tender caresses, and my father's smile of benevolent pleasure while regarding me, are my first recollections. I was their plaything and their idol and something better---their child, the innocent and helpless creature bestowed on them by Heaven." 

Victor’s childhood is portrayed as having many external advantages, and his family was described as “loving”. His parents are citizens of the Italian city of Genoa. His ancestors are described as politically important within the city and his parents as “most distinguished.” He is the eldest of three children. His mother dies when he is young and his father appears to have been busy with important offices in the city government, thus causing the child to be raised by servants and schoolmasters, a situation similar to that of Mary Shelley’s husband Percy (who may have been the model for Victor). Victor shows an early interest in and aptitude for science.

In many ways, the creature seems to have been like a baby when he first came to life. He says,

A strange multiplicity of sensations seized me, and I saw, felt, heard, and smelt at the same time; and it was, indeed, a long time before I learned to distinguish between the operations of my various senses.

He appears to be what Enlightenment philosopher, John Locke, called a tabula rasa, or blank slate. The creature knows nothing initially and can only begin to learn from the sensory information he receives. He describes his "fatigue" and the way he felt "tormented by hunger and thirst." He eats and sleeps a lot, and he cries a good deal; sounds a lot like a baby, doesn't it? Moreover, he says, "feeling pain invade me on all sides, I sat down and wept." Not knowing what else to do, he simply cries. He fixes his eyes upon the moon, almost like an infant would do with a large and bright object.

Soon, he "began also to observe, with greater accuracy, the forms that surrounded [him] . . . ." His senses become distinct rather than muddled, and his "mind received every day additional ideas." He learns quickly, even more quickly than a typical infant. After he encounters the DeLaceys, he begins to develop language and a knowledge of history and the world around him.

In Frankenstein, Volume II (chapter III), the Creature describes his early development. Initially, the Creature is very confused by his own senses and his "era of...being." The Creature felt severe pain when looking into light, so much he had to shut his eyes. It is "here" where the Creature first experienced darkness. This darkness troubled him so much that he had to reopen his eyes.

Upon realizing that he was mobile, the Creature found that there were "no obstacles which [he] could not either surmount or avoid." After walking, the Creature found that he was hungry and thirsty. He was able to drink and eat, soon falling to sleep. Once awaking, the Creature found he was cold. He searched for clothing, which he found, and covered himself.

Soon after, the Creature realized that he was alone--and he wept. While weeping, the moon rose to a point where the Creature could see it. The moon, wondrous to the Creature, was the only thing (at this point) which brought him pleasure.

As the first chapter of Victor's accout makes clear, Victor grew up as a child of a family with money and social status. This is in part due to the family dynasty that he emerges from, but it is also due to his father and the important character that he is in terms of the role that he played in politics. Note how this is observed in the following quote:

I am by birth a Genevese; and my family is one of the most distinguished of that republic. My ancestors had been for many years counsellors and syndics; and my father had filled several public situations with honour and reputation. He was respected by all who knew him for his integrity and indefatigable attention to public business. He passed his younger days perpetually occupied by the affairs of his country...

We can therefore infer that Victor grew up having every advantage that money could gain him. His parents are described as "loving" and caring individuals who did their best to bring up their children. However, the only negative about his childhood would have been after his mother died and his father, because of his work, had to leave Victor and his siblings to be brought up by schoolmasters and servants.

The details we are given about his childhood therefore help us to see that Victor is a character who knows privilege and wealth. We could also suggest that one of the reasons he is so solitary is that he has been left to grow up without much parental input from his father after the death of his mother. The death of his beloved mother might also be a reason why he seeks to learn the secret of life and death in order to prevent disease destroying the lives of other characters who are dear to him.