In Act 2 of George Bernard Shaw's play Pygmalion, Higgins and Col. Pickering discuss Higgins's true character with women. Higgins had declared that he has a huge problem with women because it seems as if males and females ultimately want to lead their own lives independently, but are raised to believe that a married life will be just as independently experienced. Higgins greatly disagrees with the notion of marriage, which is why he prefers to live alone. For this reason, being a bachelor may have awarded him the reputation of a ladies' man, which he clearly is not.
When Col. Pickering asks Higgins
Are u a man of good character where women are concerned?
He specifically asks this because Eliza will be living under his roof. At that time she was being bathed by Mrs. Pearce, and it is clear that Higgins will have to curb the tendencies of a man who lives alone in order to be presentable and decent in front of the new in-house guest.
To this, Higgins responds:
Have you ever met a man of good character where women are concerned?
To this, Pickering answers back that he has, indeed, met men of good character. This is when Higgins declares his idea that a man and a woman could never be able to understand each other's goals as independent persons if they bind their lives together in marriage.
He declares, quite dramatically
Well, I haven't. I find that the moment I let a woman make friends with me, she becomes jealous, exacting, suspicious, and a damned nuisance. I find that the moment I let myself make friends with a woman, I become selfish and tyrannical. Women upset everything. When you let them into your life, you find that the woman is driving at one thing and you're driving at another.
Basically what he is referring to is the notorious desperation of Victorian women to find a suitable husband, especially since women will acquire status when they "conquer" the heart of an eternal bachelor, and especially still when this bachelor is intelligent, educated, and well-to-do.
In all, what Higgins detests the idea of making friends with women because Victorian women do not know how to keep a friendship with males under friendship terms. They immediately begin to claim ownership of the male, and create expectations of them. This is why, as far as Higgins is concerned, he is better off alone.