Unlike his father, who trusted too readily, Malcolm does not trust Macduff. Here, Malcolm shows his political savvy. He knows he is likely to be sacrificed (an innocent lamb to appease an angry god), and is well aware of Macbeth's will to harm him. He thinks Macduff is a potential enemy, for he knows that Macduff once thought highly of Macbeth. He also thinks that Macduff could profit by helping Macbeth and--this is dramatic irony--thinks that Macduff has somehow remained unharmed (we know otherwise, but neither Macduff nor Malcolm know yet).
Unlike his father, Malcolm understands that things can appear fair (beautiful and honest) on the outside yet be foul (evil) within. He wants to know what lies within. The only way for him to be certain is to test Macduff's character. Had Maduff agreed to back Malcolm regardless of how corrupt Malcolm claimed to be, Macduff would have revealed a weakness in his own character and would also have revealed to Malcolm his willingness to support anyone he thought would win. If Macduff would follow a ruler as corrupt as Malcolm claims to be, he would also follow Macbeth.The only way Malcolm could tell for sure that Macduff was on the right side was to test him.