Macbeth orders the hit on MacDuff's family only after he visits the witches again. The witches call upon their spirit "masters" at Macbeth's request for more information. The first apparition, a floating head, warns Macbeth:
"Macbeth! Macbeth! Macbeth! Beware MacDuff
Beware the Thane of Fife. Dismiss me. Enough" (IV.i.79-80).
Macbeth's motivation to kill the MacDuff family comes from that warning from the apparition. Then in the same scene, Macbeth hears from Lennox that MacDuff has fled to England. MacDuff's leaving seems to reinforce Macbeth's fears from the witches' prediction--that MacDuff is a threat that must be dealt with swiftly. He immediately resolves to strike out at MacDuff and orders the attack on the castle to "give to the edge o’ the sword/ His wife, his babes, and all unfortunate souls/ That trace him in his line" (IV.i.168-170). Macbeth's hasty decision to murder MacDuff's family reveals his growing paranoia; he will take whatever action necessary to ensure the end of MacDuff's line. It also reveals a remarkable lack of hesitation on Macbeth's part; the man who agonized over murdering Duncan now easily orders the murder of an entire family. Blinded by his thirst for power, Macbeth's morals have completely slipped away.
Macbeth does not kill Macduff. Actually it is the other way around: Macduff eventually kills him. What mainly concerns Macbeth is that Macduff has deserted him and fled to England to join up with Duncan's elder son Malcolm, who is seeking help from the English king to invade Scotland and claim the throne which rightfully belongs to him. Unable to get at Macduff, Macbeth has his soldiers slaughter Macduff's wife and children. This is not necessarily for revenge but to teach all the other thanes that if they follow Macduff's example they will suffer similar consequences. Macduff has shown from the beginning that he does not consider Macbeth the rightful king and that he believes he was responsible for Duncan's murder. Macbeth wants people to think that Malcolm had his father murdeered and then fled the country to avoid punishment for patricide; but if Macduff joins Malcolm, he is showing that he considers this an outrageous lie. Macduff was the one who discovered Duncan's body, and Macbeth was with him at the time. Macduff remembers Macbeth's guilty behavior and knows intuitively that Macbeth was the real murderer.