An interesting question. It assumes that the end needs justification, and I'm not sure that is the case. However, if I had to justify the ending in which Gurov distances himself emotionally from the woman, I'd point to subtle points of character in the paragraphs near the end. When the woman exposes her ignorance, it reminds Gurov of his daughter, and this starts a series of thoughts through which he mentally reduces her value to him, and/or judges her. That leads to the final judgment that she is pathetic. So, even if they have/continue an affair, it is painted as sad.