In Chapter 1 of his autobiography, Frederick Douglass says that many people believed that his owner was also his father. He says that this does not matter -- he would still be a slave.
He goes on to say that slaves who were the children of the master had a harder life. He says that they were likely to get picked on by the master's wife because she would be mad at them as evidence that her husband was sleeping with the slaves.
He also says that such slaves often get sold because it is too hard for the master to deal with having them around.
Fredrick Douglas was the son of his white master. In his book he writes that he had endured greater hardships because of this. One of the first reasons that he gives is that the mistress/wife of the master does not enjoy seeing her husband's off spring. The child serves as a reminder of her husband’s unfaithfulness and is like a thorn in her side. As a result the mistress is always finding fault a something they did. In addition, she is pleased when it results in the child being whipped.
The master is also pushed to sell his mulatto children away because their presence irritates the mistress. The children are then sold to flesh mongers who will whip and abuse them.