Malcolm is reporting, to Duncan, what Malcolm has been told about the manner of the Thane of Cawdor at his execution.  How a warrior, even a traitor such as Cawdor, acted at the moment of his death was important to Duncan and his court.  Malcolm is reporting that Cawdor died manfully and admirably, even though he had betrayed his king. 

That very frankly he confess'd his treasons,
Implored your highness’ pardon,and set forth
A deep repentance. (I.iv, 6-8)

This means that Cawdor confessed to the things he had done to betray his king, and begged forgiveness.  Cawdor, who has "repentance" is, we are to believe, very sorry for his actions.  Malcolm goes on to say:

Nothing in his life
Became him like the leaving it; he died
As one that had been studied in his death,(10)
To throw away the dearest thing he owed
As ’twere a careless trifle. 

 Malcolm is now saying that nothing that Cawdor had done in his life before was as good as how he acted in his death.  Malcolm, a loyal subject of Duncan, does not approve of Cawdor's traitorous acts; but Malcolm is impressed that Cawdor is not afraid of death, nor begs for mercy in a cowardly way.  This is what Malcolm means when he says "Became him like the leaving of it".  Nothing was so good and honorable in Cawdor's life as was the manner with which he faced his death.

When Malcolm says that Cawdor was "studied" in his death -- it means that he appeared to have prepared himself, philosophically, for death.  None of the predictable human fear of death are shown by Cawdor.  Cawdor takes the fact of his execution manfully, without fear, or pleas for mercy, or attempts to escape.  He gives up his life willingly -- which causes Malcolm to say that Cawdor throws "away the dearest thing he owed (meaning "owned")" -- that is, his life "as 'twere a careless trifle".  Cawdor gives up his most important possession -- his life -- as if it were nothing important at all.  Cawdor's courage in the face of death, and repentance for the betrayal that got him to his excecution, impress Malcolm and Duncan greatly.