There are many different ways to understand Exodus 20:5. Some say that it means that people only get punished if they still hate God down to the third or fourth generation. Look closely at the wording:
...I, the LORD your God, am a jealous God, punishing the children for the sin of the parents to the third and fourth generation of those who hate me...
It doesn't say he will "punish the children of those who hate me" so some people believe that this means you have to keep hating God to be punished.
Another way of looking at this is to say that God allows bad things to happen here on Earth to those whose parents do bad. This is not surprising. If my parents had chosen to do drugs and commit crimes and such, my life would have been much harder through no fault of my own. Some people argue that God simply allows these natural consequences to happen. They argue that he does not promise to send such people to Hell or punish them in the afterlife. This is how they square the idea of punishment for the sins of the father (guilt by association) with the idea of a just and merciful God.