Explain Nick's fascination with Tom's transition "from libertine to prig."
First, concerning your question about The Great Gatsby, I suggest rethinking your use of the word "fascination." I think that's probably too strong. I don't think he's fascinated, I think he's just disgusted. Using fascinated here has a bit of a negative connotation, as if there's something odd about Nick's telling this part of the story. And Nick's disgust here is normal and appropriate.
Nick is revealing the fact that when Tom cheats on Daisy he is liberal-minded, but when Daisy cheats on him, he is conservative. In other words, it's okay for him to cheat on Daisy, but not for Daisy to cheat on him. Tom says:
...Nowadays people begin by sneering at family life and family institutions and next they'll throw everything overboard and have intermarriage between black and white.
The point of Nick's comment about the transformation from libertine to prig, of course, is that Tom doesn't raise these objections when he cheats on Daisy, only when he thinks Daisy is cheating on him.
With his comment, Tom is revealed to be hypocritical, as well as closed-minded and bigoted. Nick just lays this out for us.
I believe that the reason that Nick Carraway is so fascinated with this is because of how amazingly hypocritical Tom Buchanan is being.
Tom is absolutely outraged that Gatsby is "making love" to his wife (in the old sense of trying to woo her) because she is his wife and therefore belongs to him. But, at the same time, Tom is a libertine -- he is guilty of worse than what Gatsby has done. Tom has actually been cheating on his wife, sleeping with another man's wife.
Nick is amused by how hypocritical this is.