Both males and females in the planter class were affected by slavery; often with negative results for slaves. The planter himself was largely responsible for managing the day to day affairs of the plantation, including supervision of field hands; but quite often had sexual relations--not always consensual--with female slaves. On many plantations, the master of the plantation was the father of one or more children by a female slave. Younger white men also often forced themselves on female slaves as the slave had no right to resist. In a strange but cynical way, this became something of a rite of passage to manhood.
The plantation mistress was responsible for running the household, but also was responsible for the medical care of the slaves. If a pregnant slave went into labor, she was responsible for seeing that the child was safely delivered. She was not to question her husband about business or his relationship with the slaves, as she was to remain subordinate to him. Most plantation mistresses knew if their husbands were engaging in sex with female slaves, but were forced to remain silent about it. Quite often, since she could not vent her frustration on her husband, she did so against the slave, often a house servant. There are many instances of cruel beatings, burnings, etc. inflicted by the mistress on the household servant, normally because of some slight misfeasance; but the real reason was frustration over her husbands conduct. An excellent resource describing the experiences of slaves, household and otherwise is a compilation of memoirs by Ira Berlin, et. al. entitled Remembering Slavery.