The Thought Police is part of it, but there is so much more. First, all members of the Party wear a uniform (conformity at its most basic). They are not permitted to have personal relationships--marriages are not approved if "love" is suspected, children are trained to report on their parents, and sex is not encouraged--thus no close, loving, trust-based relationships exist in the Party. The frustration that builds in the Party is taken care of in the Two Minutes' Hate, where everyone gets to scream and yell and throw things to release tension. Deep and meaningful friendships do not exist, either. No one knows where anyone else lives (they see people at work, and they go home. There is no socialization). The telescreens have a lot to do with it, as well. Having a screen in your home which can see and hear you at your every living moment has to be enough to drive a person crazy. These people were literally scared to death of even talking in their sleep and revealing their true feelings which many of them did, and the Thought Police were on them like honey draws flies.
For people who were born and grew up in this society, it might not be so bad (this is all they ever knew). However, for people like Winston who have memories of the time before the Party, it is a difficult and nearly impossible adjustment. The rules were harsh to encourage assimilation and they played on the fear of the unknown-- Room 101 and the Ministry of Love-- to make the people conform.
Hope this helps!