In 1984, why is the memory of the prostitute so frusturating for Winston?
For Winston, he is wrapped up with thinking about what he understands to be officially acceptable in the eyes of the Party:
Tacitly the Party was even inclined to encourage prostitution, as an outlet for instincts which could not be altogether suppressed.
So Winston understood that prostitution was acceptable under certain circumstances, but desire was never okay:
The sexual act, successfully performed, was rebellion. Desire was thoughtcrime. Even to have awakened Katharine, if he could have achieved it, would have been like a seduction, although she was his wife.
Ultimately, Winston's memory of the prostitution frustrated him because he could not desire this woman. There was no feature of her that attracted him in any way. Orwell goes the distance to paint the prostitute with gray hair, ugly make-up and wrinkles. However, the memory was important to record because it expressed humanity. I believe he had sex with her anyway because it felt like rebellion even though it was within the boundaries of the Party's expectations:
When I saw her in the light she was quite an old woman, fifty years old at least. But I went ahead and did it just the same.
He pressed his fingers against his eyelids again. He had written it down at last, but it made no difference. The therapy had not worked.
He thinks he can get over the idea that he had sex with this woman, but he can't because he did not experience the rebellion he wanted. He wanted to have desire, a very human emotion. But he couldn't because the prostitutes allowed by the Party were disgusting like this woman he describes.