There are two ways to think about this question and each of these ways results in a different answer. On the one hand, we can ask whether a person’s sex has any bearing on how the law sees a given act. On the other, we can ask whether a person’s sex affects how society sees a given act.
A given act is much more likely to be considered sexual harassment by society if the offender is male and the victim is female. In our society, we see males as the aggressive sex, particularly in sexual matters. We feel that men should be too tough to be harmed by sexual harassment. We also feel that men should and will just be happy if a woman finds them attractive enough to harass. We do not feel that such actions are as serious as when a man does them to a woman.
However, this is just a societal attitude. It is not an attitude that is shared by the law. In the eyes of the law, the sex of the person doing an action makes no difference. Legally speaking, a group of women who talk about men in a sexual way when there is a man in the office are just as guilty of sexual harassment as a group of men who talk about women. Both groups are creating a hostile work environment.
So, both sexes can legally be guilty of harassment for identical behaviors, but society is much less likely to see acts by men as harassment.