This is a great questions. Usually a nemesis is considered a "bad guy" but actually, Malcolm is acting as the moral agent as well as a nemesis in this play. First let us look at some definitions of nemesis.
Nemesis is defined in several way, generally we know it as some type of unbeatable opponent. However, another definition is a reference to Greek Myth showing Nemesis as the Greek goddess of retributive justice.
Even though modern movies show a nemesis as losing to hero, such as Joker to Batman, in earlier literature, the nemesis usually ended up victorious.
Malcolm, as we can tell from early in the play in Act I, is a young man. He is being named Prince of Cumberland, so he is likely only in his early teens. This makes him a bit young to take on the role of nemesis, or to adminster justice to Macbeth. He also does not fulfill the prophecies of the witches.
Macduff however, is roughly the same age and position as Macbeth (before he became King, of course). His entire home has been slaughtered, and most importantly, he was not born of woman. By convincing Malcolm to return to Scotland, he takes on a greater, if temporary role, as mentor to a future King.