I think you need to ask a more specific question, but I'll start. It's hard to generalize about all the men and all the women. The men we meet are all officials in the town or, in the case of Chillingworth, someone with a vested interest in Hester's case. They all seem to be cold, unfeeling, more interested in getting out the "truth" than they are in the state of the fallen. Part of this is the direct result of what they believe (remember that Hawthorne notes that religion and politics are virtually the same thing in their society.) It would have been interesting to see what the men in the town thought of the proceedings, but there isn't enough evidence to know.
On the other hand, the women who are portrayed in the story are NOT leaders in the town. Except for the young woman who seems to have sympathy for Hester, they too are cold, although there are hints that they are more accepting of her sooner than the town's officials (men) are.
Of course, the most individual woman in the story, Hester, is clearly the strongest character in the book. While feeling no personal guilt, she clear accepts the civil punishment for what she has done, and remains in the community to raise her daughter.
Hope this helps.