How is Gilead a place that is almost as horrible for men as it is for women?
From a purely sexual standpoint, being forced to copulate in front of others in order to impregnate the handmaids is humiliating and degrading. Of course, for the handmaids it's rape; but the men are well aware of this, and while some may take pleasure in exerting their power and influence (as only men of high social standing are allowed to father children in this way), it is very likely a form of humiliation that they can't attract these younger women (the handmaids) on their own, or have relations with them that aren't characterized by force and manipulation. In this way, men's sexual potency is called into question. Also, as we see with Offred, her assigned "lover" can't seem to father children any longer, so his wife secretly arranges for Nick the driver to have sex with Offred so that she'll become pregnant.
Well, I have to say, I don't think it is. The socially condoned rape and forced childbearing seems worse than what the men have to suffer.
However, that said, the men suffer too. They suffer directly, by being observed at all times. They suffer by not being allowed to love freely. They suffer by the strict gender and social roles placed upon them, and because so few of them can have the true power positions. Of course, they suffer in an economic sense too, and an abstract one: there is very little freedom, and they are part of an unethical regime.