We might feel at least a little sympathy for the Dursleys, awful as they are to Harry, because they are driven by a deep horror of magic. This is understandable because Petunia Evans, now Mrs. Dursley, had to deal with the death of her younger sister Lily Potter—and she holds magic responsible.
Mr. and Mrs. Dursley have no understanding of the wizarding world. They equate it with danger and evil, not realizing that the evil in that world lies in Voldemort, not the entire wizarding universe. They simply can't see the good in wizarding: to them, all it has brought is death and destruction to Petunia's family.
It's not surprising that the Dursleys fears Harry's magic: it brings creatures, people, and actions they loathe, fear, and don't understand into the periphery of their lives. A total, rigid conformity to the Muggle world is the only way they feel safe.
Fear of the unknown and the "other" can be difficult to deal with, especially if we have evidence it has caused damage or destruction to people we know. It controls us by causing us to cling tightly to a narrow realm of existence, as the Dursleys do, and to fear change or the intrusion of anything new into our lives. It can drive us to want to keep out the danger, not knowing that the new and different can also enrich and enlarge our lives.
The main social function of referring to things as "magic" is to explain what we do not understand and, therefore, what we are likely to fear. There are any number of historical examples involving hysteria over witches and wizards started by people very much like the Dursleys. The Puritans of Salem in 1692 were very much the same type of suspicious zealots and were also motivated largely by fear of the unknown.
There is no doubt that the Dursleys are angry and spiteful in their attitude toward Harry—a continuation of their dislike of his parents—but anger is very often an expression of fear (which is why tolerance is so difficult when one is afraid). The Dursleys' elemental fear of the unknown and of the very real dangers in the magic world is compounded by their concern over their reputation in the neighborhood with other Muggles. Moreover, both parents and son are obviously cowardly by nature, meaning that fear is particularly difficult for them to withstand.
When Petunia Evans first discovered that her sister Lily had magical powers, her first reaction wasn't fear—it was jealousy. However, after marrying Vernon Dursley, she began to share his obsessive desire to be seen as absolutely ordinary, and as the perfect example of neighbors.
Fear drives a number of the Dursleys' actions towards Harry. Initially, it drove them to make him live in the cupboard under the stairs. Later, however, it was their fear of Hagrid that forced them to allow Harry to go to Hogwarts, and fear of Harry's growing magical powers that forced them to give him Dudley's second bedroom. With specific reference to Harry Potter and the Chamber of Secrets, it was fear, and not anger, that made them lock him in his bedroom after Dobby's unfortunate appearance at a dinner party.
As human beings, we are naturally afraid of the unknown, and this fear can confront us when faced with people different to us. Fear could make us want to avoid somebody or malign them, rather than show tolerance and understanding.