Let us start off by remembering that Farfrae and Henchard are contrasted through their different kind of masculinities. Henchard is a big, strong and tall male who is aggressive at times and very open with his feelings. Farfrae, being smaller and much more slender and depending on hsi brains rather than his strength for success, is thus set up as a foil for Henchard's type of masculinity.
Because of Henchard's more traditional conception of gender, he naturally assumes that women are the "weaker sex" and because of this feels superior to women and looks down on them as needing to be protected by men. Consider how this view is presented in Chapter 35:
His old feeling of supercilious pity for womankind in general was intensified by this suppliant appearing here as the double of the first.
Henchard looks down on women, and his brute strength and power has the impact of often frightening other women away. Note the way that Henchard's rage and passion causes him to wear a very "thin" mask that covers up his true personality. When he is overwhelmed by emotion, this personality easily emerges, and we are told that "There was a temper under the thin bland surface." It is this temper, the outpouring of his masculinity, that causes him to lose his wife at the beginning of his novel and to send Elizabeth-Jane away. Henchard's masculinity is therefore shown to be a force that drives others from him if it does not compel them to obey his strength.