The answer to this is not a straightforward one. Initially, Winston does not know that O'Brien is really working on behalf of the Party. This is shown by the fact that he visits his apartment and receives from him a copy of Goldstein's book. In addition, in part 3, chapter 1, when Winston is imprisoned in the Ministry of Love, he wonders if O'Brien has also been caught and arrested. When O'Brien finally visits Winston in his cell, Winston says:

"They’ve got you too!" he cried.

This shows that Winston has no idea that O'Brien works for the Party. However, in the next paragraph, Winston admits that deep down, he always knew that O'Brien was working for the Party:

Yes, he saw now, he had always known it.

This suggests that instead of realizing that O'Brien really was a member of the Inner Party, working on behalf of Big Brother, Winston allowed himself to be lulled into a false sense of security. Why? Because Winston was desperate to rebel. He so badly wants to bring down Big Brother that he lets it cloud his judgment. He chooses to believe in O'Brien, the Brotherhood, and Goldstein's book instead of realizing that he could never overcome a force as strong as the Party.