Harry Potter only lived with his parents for a very brief period of his early life. His parents, James and Lily Potter, were murdered by Voldemort, in an attempt to get to Harry, when he was a year and three months old. At this time, Voldemort believed that Harry was the fulfillment of a prophecy which would lead to the downfall of the Dark Lord. He killed Harry's parents because they tried to get in his way.
At this time, Harry was living in Godric's Hollow, where Lily and James lived. Godric's Hollow makes a significant appearance in the later books of the series, but Harry does not really remember much about it. After his parents' death, he was sent to live with his aunt and uncle, the Dursleys, in a provincial Muggle housing estate.