The way that the party made Winston betray Julia (and vice versa) is cruel and effective way of humiliating the person by making them feel gulity.
Winston had betrayed information about Julia during the torture but as he had pointed out to O'Brien- he had not betrayed her. When he says this Winston means that he has not betrayed her in his heart. O'Brien had understood Winston and even agreed with him when he had said this.
However, O'Brien made him betray Julia by putting him in a machine with the thing he had feared most- and in that moment Winston wanted Julia to take his place, he had not only said, he meant it. That is how the party brakes him, by making him feel less then he is, by making him feel guilty.That is exactly what the party (Big Brother) wants- to humiliate the person until that person becomes less than a person and becomes a mindless creature. As Julia says later when she and Winston meet each other after the torture -it does not matter you're sorry afterward in that moment you meant it.
Winston's betrayal of Julia is also his faliure to overcome the Big Brother regime because he believes them- he belives he is not an individual.