Nick shakes Tom's hand because Nick realizes there's no way of getting the truth from Tom, and even if there were, Tom is too shallow, unintelligent, and morally corrupt to comprehend his part in the tragedy. Nick could lecture Tom until the end of time - Tom was just ignorant and selfish.
"I couldn't forgive him or like him, but I saw that what he had done was, to him, entirely justified" (157).
"I shook hands with him; it felt silly not to, for I felt suddenly as though I were talking to a child" (158).
Nick just didn't want to waste any more time with this guy.