"Angry as I was, as we all were, I was tempted to laugh whenever he opened his mouth. The transition from libertine to prig was so complete."
This statement is made in chapter 7 while Gatsby, Nick, Tom, Daisy and Jordan are in the hotel room. Tom has just found out about Daisy and Gatsby's affair and the argument is heating up.
A "libertine" is a person who doesn't have any moral constraints; it's someone who does whatever brings pleasure without regard to the rules of society or morality. A "prig" is the complete opposite; it's someone who adheres strictly to society's rules and is smug or self-righteous about it.
Tom has been unfaithful to his wife on multiple occasions and has had at least one affair with a married woman, Myrtle Wilson. He completely disregards the feelings of his own wife and of Myrtle's husband, George. It is in this way that he is a libertine.
However, he finds it outrageous that Daisy and Gatsby are having an affair. He shows himself to be a prig when he states, "Nowadays, people begin by sneering at family life and family institutions and next they'll throw everything overboard and have intermarriage between black and white."
Nick is fascinated and amused by the hypocrisy of Tom's statement. Someone who cheats on his own wife frequently (and even admits to it in front of everyone in this chapter) should not criticize others for disrespecting "family life," yet Tom does just that.